Let’s look at some benchmarks – the issue was that this job was taking 30+ seconds which would have timed out the HTTP server and it’s not yet a background job. It unzips a file to a temporary directory (not in EFS), does some validation, then copies the contents to EFS. The zip in question was 3000 files, around ~150MB extracted.
# find /big_dir | wc -l
3098
# time cp -R /big_dir /efs_dir
real 0m37.957s
user 0m0.052s
sys 0m0.960s
Ouch, that’s not good. We could try rsync, maybe that will help:
rsync -r /big_dir /efs_dir
real 1m10.210s
user 0m0.931s
sys 0m1.744s
Even longer! Why is this? It’s because:
Metadata I/O occurs if your application performs metadata-intensive, operations such as, "ls," "rm," "mkdir," "rmdir," "lookup," "getattr," or "setattr", and so on. Any operation that requires the system to fetch for the address of a specific block is considered to be a metadata-intensive workload.
rsync is also checking the destination file to see if it needs to sync it, which causes a bottleneck. So plain rsync and cp aren’t an option.
The issue is that Elastic File System is not built for serial operations. That is, copying a file, waiting, and copying the next one. EFS must replicate all the files to multiple locations so there is a delay while it does so. There is also some overhead from NFS, as each filesystem operation is a network call. What EFS is designed for is actually parallel operations. But rsync or cp can’t run in parallel, so you’ll need to manually batch up your files or use this tool that was referenced in the document above called fpsync (Filesystem partitioner sync).
What fpsync can do is split a directory of files up into chunks, and then send those contents in parallel via rsync. This is also possible with GNU Parallel but you’d have to write your own script. fpsync was available on CentOS, and probably many other distributions. Let’s run it out of the box:
fpsync /big_dir /efs_dir
real 0m59.790s
user 0m1.975s
sys 0m3.925s
Not much of an improvement…but why? Because fpsync doesn’t run in parallel by default, and you have to tweak it a bit. Let’s process 100 files at a time using 10 concurrent runners:
# time fpsync -f 100 -n 10 -v /big_dir /efs_dir
1662569967 Info: Run ID: 1662569967-43986
1662569967 ===> Analyzing filesystem...
1662569968 <=== Fpart crawling finished
1662569980 <=== Parts done: 29/29 (100%), remaining: 0
1662569980 <=== Time elapsed: 13s, remaining: ~0s (~0s/job)
1662569980 <=== Fpsync completed without error in 13s.
real 0m13.467s
user 0m2.086s
sys 0m4.400s
Much better! But let’s try more concurrent runners. Since we had 3000 files, there would have been a queue in our last command (100*10 = 1000). So let’s run 50 batches of 50 files each:
# fpsync -f 50 -n 50 -v /big_dir /efs_dir
1662570120 Info: Run ID: 1662570120-51913
1662570120 ===> Analyzing filesystem...
1662570122 <=== Fpart crawling finished
1662570129 <=== Parts done: 58/58 (100%), remaining: 0
1662570129 <=== Time elapsed: 9s, remaining: ~0s (~0s/job)
1662570129 <=== Fpsync completed without error in 9s.
real 0m8.903s
user 0m2.093s
sys 0m4.868s
So, the more concurrent copy operations we can run, the better.
On a regular disk this wouldn’t have any effect since the filesystem operations are negligible and your only bottleneck is the disk speed. It might even slow it down. There may be some other options inside of fpsync that would speed it up even more. What about rsync --inplace? This would eliminate a step that rsync would usually take, which is to create a new file, then rename it.
# time fpsync -o "--inplace" -f 100 -n 50 -v /big_dir /efs_dir
1662584365 Info: Run ID: 1662584365-126224
1662584365 ===> Analyzing filesystem...
1662584367 <=== Fpart crawling finished
1662584371 <=== Parts done: 29/29 (100%), remaining: 0
1662584371 <=== Time elapsed: 6s, remaining: ~0s (~0s/job)
1662584371 <=== Fpsync completed without error in 6s.
real 0m5.872s
user 0m1.634s
sys 0m3.438s
Running batches of 100 brought it down to under 6s. After that it started to get slower. Also running a huge number of rsyncs and small batches got slower. This is likely due to the system itself – after all, it’s running 250+ instance of rsync.
Enhancing modules with Rules-based conditions was very easy in D7. Using hook_default_rules_configuration we could dynamically generate a bunch of rules called mymodule_rule_[some_key], use rules_ui()->config_menu() to add the menu items for the Rules admin UI, then invoke the generated components to evaluate conditions. Every entity or option would have its own Rules component that we can edit and add arbitrary conditions. Some examples of this in D7 were:
Payment methods (Ubercart/Commerce)
Coupons
Tax rules
Block visibility
User access or eligibility
And anything where you could not possibly know of the conditions that would be needed. Some of the above were changed to use Core conditions in D8, but that didn’t cut it for our use case since I could not possibly write a new condition for every requirement that came up. Real life examples of these are:
A user can only claim a certain kind of course credit when the credit code on the course contains specific characters and the user is from Florida.
The user can only use the payment method when there is a valid role attached to the user and specific products are in the cart.
A user is not eligible to receive a certain type of credit when they are eligible to receive a certain type of credit.
A quiz taker can only see correct answers once two weeks have passed and the user exhausted two attempts.
These aren’t out of the ordinary and we would be writing custom PHP if/else trees every day. For a SaaS-like product this is not ideal.
It’s a little trickier to add arbitrary conditions to entities but well worth it in the end. Rules provides a test module that you can look at: rules_test_ui_embed. This example illustrates using 1 rule component embedded into a page. But we need to build Rules into all instances of a configuration entity.
There is documentation for extending Rules with new conditions and actions, but it is pretty lacking around integration. There is some embedded developer documentation so let’s take a look.
If we look at rules_test_ui_embed we see that there is some sort of plugin file – rules_test_ui_embed.rules_ui.yml. That must define something!
The above defines a Rules UI plugin which will create routes on rules_test_ui_embed.settings .The configuration for the Rules component will be saved to rules_test_ui_embed.settings under the conditions key. But that doesn’t work for us, we need to have multiple components on multiple entities.
There’s another parameter in RulesUiConfigHandler we can use to allow wildcard editing of components: config_parameter
It appears that config_parameter and config_key can be used to dynamically set which configuration object and key will be updated. With a little trial and error I applied it to Quiz feedback types. Feedback types hold sets of review options that display feedback to quiz takers after they answer a question or finish an entire quiz. They can also be used for post-review feedback, in the case of revisiting the quiz after 2 weeks. Only seeing correct answers after 3 attempts, only seeing instructor feedback once given a role, etc…
Let’s assume that we already have a QuizFeedbackType entity to allow creation of custom feedback “times”, and all the edit forms are already set up. We want to add conditions to each feedback type so that we can conditionally display their items. In Quiz we have two built in: “Question” and “End”.
Define route and *.rules_ui.yml
This will indicate that we want Rules UI functionality appended to a route that we will also create. It will also tell Rules that we want the component to be saved onto the object loaded from the quiz_feedback_type parameter. Note how _rules_ui option on the route matches the plugin name defined in quiz_rules.ui.yml:
This is a normal form that extends ConfigFormBase, but is provided with a Rules UI handler from the plugin definition that matches the route above. Most of this code is copied from rules_test_ui_embed:
QuizFeedbackConditionsForm.php
[...]
class QuizFeedbackConditionsForm extends ConfigFormBase {
/**
* The RulesUI handler of the currently active UI.
*
* @var RulesUiConfigHandler
*/
protected $rulesUiHandler;
/**
* {@inheritdoc}
*/
protected function getEditableConfigNames() {
return [];
}
/**
* {@inheritdoc}
*/
public function getFormId() {
return 'quiz_feedback_conditions';
}
/**
* {@inheritdoc}
*/
public function buildForm(array $form, FormStateInterface $form_state, RulesUiConfigHandler $rules_ui_handler = NULL) {
$form = parent::buildForm($form, $form_state);
$this->rulesUiHandler = $rules_ui_handler;
$form['conditions'] = $this->rulesUiHandler->getForm()
->buildForm([], $form_state);
$form['actions']['cancel'] = [
'#type' => 'submit',
'#limit_validation_errors' => [['locked']],
'#value' => $this->t('Cancel'),
'#submit' => ['::cancel'],
];
return $form;
}
/**
* {@inheritdoc}
*/
public function validateForm(array &$form, FormStateInterface $form_state) {
parent::validateForm($form, $form_state);
$this->rulesUiHandler->getForm()
->validateForm($form['conditions'], $form_state);
}
/**
* {@inheritdoc}
*/
public function submitForm(array &$form, FormStateInterface $form_state) {
$this->rulesUiHandler->getForm()
->submitForm($form['conditions'], $form_state);
// Save the configuration that submitForm() updated (the config entity).
$config = $this->rulesUiHandler->getConfig();
$config->save();
// Also remove the temporarily stored component, it has been persisted now.
$this->rulesUiHandler->clearTemporaryStorage();
parent::submitForm($form, $form_state);
}
/**
* Form submission handler for the 'cancel' action.
*/
public function cancel(array $form, FormStateInterface $form_state) {
$this->rulesUiHandler->clearTemporaryStorage();
$this->messenger()->addMessage($this->t('Canceled.'));
$form_state->setRedirectUrl($this->rulesUiHandler->getBaseRouteUrl());
}
}
In buildForm we take in the Rules UI handler and use it to generate the condition form. In submitForm, the Rules UI handler will notify our Rules component “provider” that there is a component that has to be saved.
Implement RulesUiComponentProviderInterface
The rulesUiHandler from above requires the entity type to handle getting the Rules component and saving it onto itself since we are not specifying a static config_name or config_key We add the component property to config_export, then we implement RulesUiComponentProviderInterface and implement the 2 methods:
/**
* @ConfigEntityType(
[...]
* config_export = {
* "id",
* "label",
* "component"
* },
*/
class QuizFeedbackType extends ConfigEntityBase implements RulesUiComponentProviderInterface {
/**
* {@inheritdoc}
*/
public function getComponent() {
if (empty($this->component)) {
// Provide a default for now.
$this->component = [
'expression' => ['id' => 'rules_and'],
'context_definitions' => [
'quiz_result_answer' => [
'type' => 'entity:quiz_result',
'label' => 'Quiz result',
'description' => 'Quiz result to evaluate feedback',
],
],
];
}
if (!isset($this->componentObject)) {
$this->componentObject = RulesComponent::createFromConfiguration($this->component);
}
return $this->componentObject;
}
/**
* {@inheritdoc}
*/
public function updateFromComponent(RulesComponent $component) {
$this->component = $component->getConfiguration();
$this->componentObject = $component;
return $this;
}
}
In getComponent() we check to see if the entity already has conditions, and return a RulesComponent. If it does not, we provide a default that intakes a QuizResult entity to evaluate.
In updateFromComponent(), we get the RulesComponent and store it on the entity.
And there it goes, components being attached to entities.
Now that the components are stored on an entity that implements RulesUiComponentProviderInterface, we can invoke the rule in our code to validate the conditions:
/* @var $component RulesComponent */
$component = QuizFeedbackType::load('question')->getComponent();
// Answer 31 has not been answered.
$not_finished = QuizResult::load(31);
$component->setContextValue('quiz_result', $not_finished);
$x = $component->getExpression()->executeWithState($component->getState());
var_dump($x); // $x = FALSE
// Answer 32 has been answered, show feedback.
$finished = QuizResult::load(32);
$component->setContextValue('quiz_result', $finished);
$y = $component->getExpression()->executeWithState($component->getState());
var_dump($y); // $y = TRUE
If you already have Drush aliases set up for your live machines, deploy those aliases to your CI server, and use them to launch your nodes (instead of copying the SSH info).
Create a new “Dumb slave” but instead of “SSH slave”, select “Launch slave via execution”
Alias files are useful. If you have many sites on a remote server, instead of manually adding them to your alias file every time, write some code to automatically generate them.
[php title=”~/.drush/dynamic.aliases.drushrc.php”]
$domains = explode(“\n”, trim(shell_exec(“ssh user@myserver.example.com ‘ls /some/path'”)));
// now $domains is a list of directories on your remote server
// set up an alias for each one
foreach ($domains as $domain) {
$aliases[$domain] = array(
‘root’ => “/path/to/$domain/public_html”,
‘remote-host’ => ‘some-server.com’,
// if necessary, URI other than default
‘uri’ => $domain,
‘remote-user’ => ‘user_if_necessary’,
);
}
[/php]
Now whenever a site is configured on your target server, you will automatically have the alias!
Our team uses Drush frequently during the entire development workflow for doing things like grabbing database dumps of sites and running commands – drush make, registry rebuild, custom company-specific ones, etc. – and in the past everyone would have to manually download or copy them to their .drush.
Now, we version the .drush directory, so when a new developer onboards, they can just checkout the .drush directory from version control.
The bottleneck is that drush sql-sync works with temporary files – meaning it has to:
Connect to the remote machine
Perform a sql-dump to a file on the remote machine and compress it
Transfer that file to your machine
Restores the dump to database
The problem with this is that each step is executed consecutively. It would be better if all these steps were performed concurrently. Drush defaults to this method because it is compatible with most systems. If you’re a power user though, you may want a find a faster solution.
What we’d like to do is
Connect to the remote machine
Perform these steps at the same time
Read the file remotely
Compress on the fly
Stream it to your local machine
Uncompress on the fly
Pipe sql to database
I wrote this little script that accomplishes just that and a little extra for dumping locally. The key is piping data instead of saving it temporarily. Note that this only works on Linux/Mac.
#!/bin/bash -x
drush -y sql-drop # this doesnt have an alias for a reason. only work locally
drush $1 sql-dump --gzip | gzip -cd | drush sqlc
drush -y updb
# Set last update date to now to prevent checking for updates for a bit
drush -y vset update_last_check `date +%s`
# Setting file paths (use default)
drush -y vdel file_directory_path
drush -y vdel file_public_path
# Clear cache
drush cc drush
Put this script somewhere (maybe ~/bin) and chmod a+x it.
From within your site directory, run `fastdump @someAlias`
This will
Delete all the local tables (to ensure tables that don’t exist in your source are gone)
Restore the database from an alias
Run updates
But quickly! The next step for this would be making it into a Drush command instead of a shell script.
We have a shared alias file that represents every site that we work with. For example
@abcstage
@abctest
@abclive
are all valid aliases. Developers would have access to stage and test, while live only works for privileged users.
But, we still want to make sure that no funny business goes on.
Create a file, ~/.drush/policy.drush.inc
function drush_policy_sql_sync_validate($source = NULL, $destination = NULL) {
if (strpos($destination, 'live') !== FALSE) {
return drush_set_error(dt('Per ~/.drush/policy.drush.inc, you may never overwrite the production database.'));
}
if (strpos($source, 'stage') !== FALSE && strpos($destination, 'test') !== FALSE) {
return drush_set_error(dt('Dumping from stage to test is a terrible idea.'));
}
if (strpos($source, 'stage') !== FALSE && strpos($destination, 'live') !== FALSE) {
return drush_set_error(dt('Dumping from stage to live is even worse.'));
}
}
This will ensure that nobody can accidentally sql-sync to a live site. You can adjust the criteria as need be.