Scheduling jobs with Yii

Sometimes in your app you'll have a long running, or computationally expensive job that really shouldn't be handled by your web server while your user waits, but should be processed in the background. You could also want to wait to execute a job until a later time, like send then user a report at the end of the day. The solution to both of these scenarios is to have a job queue and a different, non-web server processing your job queue.

Heroku Scheduler

Heroku offers a free add-on called "Heroku Scheduler" that can run a job every 10 minutes, every hour, or once a day. We are going to use this scheduler to call our Yii Console command.

The Queue

I'm going to use a MySQL table as a queue, even though some people think that's a bad design pattern. For what I'm doing, I won't run into any of the issues mentioned in that article. You could use any of the *MQ add-ons if you like.

The Queue Table Migration

The migration for the queue table is pretty simple. We have a timestamp that we want to execute the job after, when the job was actually executed, whether or not it succeeded, the action we want to perform, the parameters, and an optional execution result. We'll talk about how these field are used when we get to the console command. (You can find the code for this and the other code in this post at https://github.com/aarondfrancis/yii-CronCommand.)

$this->createTable('tbl_cron_jobs', array(
    'id' => 'pk',
    'execute_after' => 'timestamp',
    'executed_at' => 'timestamp NULL',
    'succeeded' => 'boolean',
    'action' => 'string NOT NULL',
    'parameters' => 'text',
    'execution_result' => 'text'
));

Code highlighting powered by torchlight.dev (A service I created!)

The CronJob Model

The CronJob model is the standard Gii generated model, except for the following two methods.

public function beforeValidate(){
    if(gettype($this->parameters) !== "string"){
        $this->parameters = serialize($this->parameters);
    }
    return parent::beforeValidate();
}
 
public function afterFind(){
    $this->parameters = unserialize($this->parameters);
    return parent::afterFind();
}

The BeforeValidate method serializes any parameters and the AfterFind method unserializes them. Say, for example, that we want to send a welcome email two hours after the user signs up, we could create a new CronJob and pass an array with the user's id. The model will serialize it, save it to the DB, and when we get it back it will be unserialized and ready to go.

The Yii Console Command

Now that we have a model to work off of, we can look at the actual CronCommand that we'll be running. (Again, you can get the code at https://github.com/aarondfrancis/yii-CronCommand.) I'll only cover the relevant parts here.

The entry point for our script in this example is going to be the Index action. This is what we'll call from the Heroku Scheduler, and then we'll let the index action determine what needs to be processed. (This is only one way of doing it. I like to do it this way because I only want to deal with the logic in one place. If I add a new command I need to process, all I have to do is add it to the CronCommand and the next time Heroku calls my index action, it can process the new command. The alternative would be to create an action{New} for every new action and then add a schedule on Heroku to call that new action. I prefer having one entry point: actionIndex.)

The first thing we do is get a list of jobs that need to be processed. We select any jobs where Now is greater than the execute_after time (meaning it should be executed) and it hasn't already been executed. We sort by ID, ascending, so that the oldest jobs will come first. (If we had a really active queue, we may never get to the oldest jobs if we don't select the oldest ones first.)

$jobs = CronJob::model()->findAll('execute_after <:now AND executed_at IS NULL ORDER BY id ASC', array(':now'=>$now));

We then start to loop through all the jobs and process them. The first thing we need to do is make sure that we have a method to handle it. If we do, we call that method.

if(method_exists($this, $job->action)) {
    $result = $this->{$job->action}($job->parameters);
}

So if you save a CronJob model with "testJob" as the action, the CronCommand is going to call $this->testJob($job->parameters) when it processes that particular job. If the result is FALSE, we happily skip it and process it again next time. Otherwise, you should return an array with succeeded as a boolean and an execution_result if you like.

Making It Run

Now that you have your Cron command set up, you can test it by running ./yiic from your Terminal in the protected folder. You should see "cron" as an option. Now try running ./yiic cron. If you have any jobs stored in your table already, you should see them processing.

One last thing we need to do to get it running on Heroku is set up a bash script to call it. It's a super simple two-liner:

export LD_LIBRARY_PATH=/app/php/ext
bin/php www/protected/yiic.php cron

The first line sets a path required for PHP to run properly from the command line (thanks to Norbert Kéri for pointing that out) and the second line calls our yiic.php with the cron command. Save this as "heroku.sh" or whatever you want to call it and then add it to your Heroku scheduler by entering the following command:

www/protected/heroku.sh

Code highlighting powered by torchlight.dev (A service I created!)

Now your actionIndex will be run every 10 minutes, grabbing items off the queue, processing them, and saving the results back.

Problems

Let me know if you have any issues, I'll do my best to help.