Reservable models in Laravel

As I'm in the process of rebuilding my personal site, I'm orchestrating a lot of 3rd party services. Pulling videos down from YouTube. Transcribing audio. Asking OpenAI for summaries of the transcripts, etc.

I rely on Eloquent's scopes a lot to figure out what needs to be done. For example, here are several scopes from my Video model.

public function scopeReadyForDownload(Builder $query): void
{
    $query->whereNull('video_path');
}
 
public function scopeReadyForAudio(Builder $query): void
{
    $query->whereNotNull('video_path')->whereNull('audio_path');
}
 
public function scopeReadyForTranscript(Builder $query): void
{
    $query->whereNotNull('audio_path')->whereDoesntHave('transcripts');
}
 
public function scopeReadyForThumbnails(Builder $query): void
{
    $query->whereNotNull('video_path')->whereNull('thumbnails');
}

Code highlighting powered by torchlight.dev (A service I created!)

For each of these different tasks, I have a command that gets the models and does the work. Here's the Download command:

class Download extends Command
{
    protected $signature = 'youtube:download';
 
    protected $description = 'Download YouTube videos and put them on S3.';
 
    public function handle(): void
    {
        Video::readyForDownload()->each(function (Video $video) { 
            Log::info('Downloading ' . $video->title);
 
            // Do something interesting
        });
    }
}

But! There are two problems with this approach.

The unreserved problems

The first problem is that depending on how long the command takes to complete, how often the command is run, and a myriad of other real-world considerations, we could end up processing a model through this workflow more than once.

We could add withoutOverlapping on the schedule ( and I recommend you do!) but someone could still run the command manually and you end up with overlapping processing.

The second problem has less to do with duplicate processing, and more to do with stampeding on failures. Lets say we're running this command every minute (which is totally reasonable) and one particular video keeps failing. We could potentially try to download a video from YouTube 60 times in a single hour. YouTube won't like that very much. So we need to add a sort of gate, or buffer, to ensure we're not constantly hammering external resources on repeated failures.

Reserving models

Instead of relying on convention or somehow enforcing team rules, we're going to solve it once and for all with cache locks.

Laravel has a robust atomic locking mechanism that can be backed by a number of drivers, including the database driver.

(I have a full video on Laravel's atomic locks if you're interested: Laravel solved race conditions.)

Using atomic locks, we're absolutely guaranteed that only one process can hold the lock. But instead of interacting with the lock directly, we're going to access it through the model, since we're specifically interested in reserving models.

We're going to create a trait called Reservable that we can add to models.

trait Reservable
{
    public function reserve(mixed $key, string|int|Carbon $duration = 60): bool
    {
        //
    }
 
    public function release(mixed $key): void
    {
        //
    }
}

These two methods will give us a pretty nice experience when it comes time to reserve a model for a particular purpose.

$reserved = $video->reserve('download', '+1 hour');

I like that! I don't have to worry about some of the more nitty-gritty details of the cache system. I just try to reserve a model for a specific purpose and get told yes or no.

Let's flesh out the trait a bit. We're going to put most of the logic in a method called reservation, which gives you back an instance of Illuminate\Contracts\Cache\Lock.

trait Reservable
{
    public function reserve(mixed $key, string|int|Carbon $duration = 60): bool
    {
        //
    }
 
    public function release(mixed $key): void
    {
        //
    }
 
    public function reservation(mixed $key, string|int|Carbon $duration = 60): Lock
    {
        // Convert e.g. +6 hours to a Carbon instance
        if (is_string($duration)) {
            $duration = Carbon::make($duration);
        }
 
        // Convert Carbon to seconds from now
        if ($duration instanceof Carbon) {
            $duration = max(0, $duration->diffInSeconds(now()));
        }
 
        // Convert enums to strings
        if ($key instanceof UnitEnum) {
            $key = $key->name;
        }
 
        // Convert objects to strings
        if (is_object($key)) {
            $key = get_class($key);
        }
 
        // Use the most stable methods of representing a model.
        return Cache::lock(
            "{$this->getMorphClass()}:{$this->getKey()}:{$key}", $duration
        );
    }
}

We provide a pretty flexible interface for developers to use. You can pass almost anything as the $key and almost anything as the $duration. We'll come back to the reason for this.

Let's finish our implementation.

trait Reservable
{
    public function reserve(mixed $key, string|int|Carbon $duration = 60): bool
    {
        return $this->reservation($key, $duration)->get(); 
    }
 
    public function release(mixed $key): void
    {
        $this->reservation($key)->forceRelease();  
    }
 
    public function reservation(mixed $key, string|int|Carbon $duration = 60): Lock
    { ...
        if (is_string($duration)) {
            $duration = Carbon::make($duration);
        }
 
        if ($duration instanceof Carbon) {
            $duration = max(0, $duration->diffInSeconds(now()));
        }
 
        if ($key instanceof UnitEnum) {
            $key = $key->name;
        }
 
        if (is_object($key)) {
            $key = get_class($key);
        }
 
        return Cache::lock(
            "{$this->getMorphClass()}:{$this->getKey()}:{$key}", $duration
        ); 
    }
}

Code highlighting powered by torchlight.dev (A service I created!)

Those implementations become quite simple. In the reserve method we create the lock and try to get it, which returns a success/failure boolean.

In the release method, we create the lock and forceRelease it. We have to use forceRelease because we're not the owner. (We didn't pass an $owner string, so it's just set to a random value.) Typically you don't want to release someone else's lock, so I wouldn't use this method unless you know exactly why you're doing it. Here it makes sense because our locks are very scoped down.

Using the Reservable trait

Now that the Reservable trait is written, we can easily add it to our Download command.

class Download extends Command
{
    protected $signature = 'youtube:download';
 
    protected $description = 'Download YouTube videos and put them on S3.';
 
    public function handle(): void
    {
        $video = Video::query()
            ->readyForDownload()
            ->get()
            // Get the first one that can be reserved for this command, for 6 hours. 
            ->first(fn(Video $video) => $video->reserve($this, '+6 hours'));
 
        if (!$video) {
            return;
        }
 
        Log::info('Downloading ' . $video->title);
 
        // Do something interesting
    }
}

Because we allowed almost anything to be passed in for the reservation key, we're going to pass the command itself. The resulting cache key looks something like this:

video:249:App\Console\Commands\YouTube\Download

We don't need to release the lock at all, we can just let it expire. If the command completes successfully, the lock is irrelevant. We don't need to download it again, and our scope will exclude it from consideration. If the command fails, we want it to be locked for 6 hours, to prevent those continuous errors.

Macros

You could take this one step further and add a macro to the Eloquent Collection to make this line a bit cleaner

->first(fn(Video $video) => $video->reserve($this, '+6 hours'))

Collection::macro('firstReserved', function (mixed $key, string|int|Carbon $duration = 60) {
    return $this->first(fn($item) => $item->reserve($key, $duration));
});

Which gives us this:

class Download extends Command
{
    public function handle(): void
    {
        $video = Video::readyForDownload()->get()->firstReserved($this, '+6 hours'); 
 
        if (!$video) {
            return;
        }
 
        Log::info('Downloading ' . $video->title);
 
        // Do something interesting
    }
}

Heck, we could even macro it on the Builder if we wanted!

Builder::macro('firstReserved', function (mixed $key, string|int|Carbon $duration = 60) {
    return $this->get()->first(fn($item) => $item->reserve($key, $duration));
});

This allows us to drop that one pesky get in the middle there. Who needs it.

class Download extends Command
{
    public function handle(): void
    {
        $video = Video::readyForDownload()->firstReserved($this, '+6 hours'); 
 
        if (!$video) {
            return;
        }
 
        Log::info('Downloading ' . $video->title);
 
        // Do something interesting
    }
}

I'd be curious to hear your thoughts on this! I've just started using it and I think it's a nice experience, but do you have ideas to improve it? Let me know!