Yii and the asset pipeline: part 2

Note: I don't recommend this anymore. I recommend using a Javascript build tool like Gulp, or Laravel Elixir.

A while back, I wrote a post called Yii, Heroku, and the Asset Pipeline where I tried to come up with a reasonable way to manage publishing assets with Yii on Heroku. (Note, this applies not only to Heroku, but to any host that runs on top of Amazon EC2, or Amazon EC2 itself.) What I came up with in that first post worked decently well. I had a system that would allow me to work locally with files that were always up to date and then change a version number and publish them all for my staging and production environments. There were still problems though, things that totally bugged me.

The Problems

Let me quickly rehash the problems with the basic Yii pipeline, as far as I'm concerned, and then I'll talk about the problems with the system I was using up until yesterday.

Problem's with Yii's Pipeline

The problem with the Yii asset pipeine is not really Yii's fault, but stems from the fact that Yii wasn't built with an ephemeral filesystem host in mind. The problem we run into is that an ephemeral filesystem will occasionally blow away your assets, because the filesystem can't be trusted to persist. This pushes us to an external static asset store, like Amazon S3. This, it turns out, is a good thing. Cookieless domains reduce overhead and decrease page load times. My first system fixed those problems, but it had problems of its own.

Problems with My (First) Solution

My fist solution was far from perfect. What I hated about it most was that it left me with a lot of manual processes (which means a lot of potential mistakes) and a subpar experience for my users.

When it came time to publish, I had to manually combine and minify all my CSS and JS. More than one time I forgot to do that. More than forgetting to do it a couple times, the painful part was having to do it every time I made a small CSS tweak. I looked at a few minifiers that minify on the fly, but I was never really that comfortable with the way they worked. I wanted everything to be done locally and not have that load placed on the server to do the minifying.

The other problem with my system was mass versioning. Any time I tweaked a CSS file or added a bit of JS code, every asset got invalidated when I incremented my version number. Way less than ideal for my end users who have their caches all invalidated when the assets go up a version. That's what I set out to fix with the Mantis Asset Manager.

The Solution: The Mantis Manager - Overview

If you're wondering where the Mantis Manager name comes from, it's from the SaaS app I'm building for timecard management for small businesses: Mantis. I built the Manager specifically for Mantis, to hopefully solve my asset issue once and for all. (And I think I have!)

You can get all the code and a pretty technical explanation of the Mantis Manager over at the github repo: https://github.com/aarondfrancis/mantis-manager. I'll give an overview here.

The Good Stuff

Let me just first say what the Mantis Manager gets you, and then I'll tell you more about it. The Mantis Manager combines, minifies, and publishes assets based on the SHA of their contents, publishing only files that have changed. It gives you a way to work locally and remotely without muddying the waters between the two. Here's the money shot:

Look how awesome that is. Mantis is happily and automatically looping through my assets folder checking files, ignoring stuff, minifying, updating, and publishing. This is the result, let's talk about the process.

The Two Types of Assets

In my mind, Yii has two types of assets. The first type is our assets. The assets we use to build our sites, the CSS we write, the images we need, and the Javascript we code. This means that the second type is, obviously, not our assets. These are the Yii JS files, component CSS/JS files, or extension files. These are assets that our sites use, but we don't write or maintain. We are going to treat these two sets of assets differently, because they are indeed quite different.

With our assets, they are likely to change fairly often as we build. When we build a new feature or fix a bug, it's likely that some of our assets are going to change. This is totally different with other people's assets, they are not likely to change. How often do the Yii JS files change? Hardly ever. Only when we update the Yii framework, I would suspect. So almost never. We'll treat these assets differently then. For our assets, we'll use the Mantis Manger, for other people's assets, we'll rely on the traditional publish() call. We'll need the two systems to work happily together.

The Solution: Details

SHAs

Let's take a look at how I managed to solve the problem. The first thing I wanted to do was to publish only changed assets. I looked at using the modified time of the file, but I found that to be less reliable than I'd like. There are times when the modified time could change, but the contents haven't changed at all. I settled on using PHP's sha1_file() function that digests the file and spits out a 20 character string. This seemed to be the best method of ensuring that a file has changed.

Before I can go all SHA crazy on the files, I need a list of files. I like to store my files in "protected/assets" because it isn't web accessible, which means I get to explicitly determine what gets published and what doesn't. If you look at the code on GitHub, you can see that this is a configurable option.

Now that Mantis knows where to look, it'll loop through every file in that folder and do it's thing. You can see from the photo above that some files are ignored; you can set ignore patterns in the config. As it loops through, it calculates the SHA of each file and stores the list in an array. When you run the command a second time, you'll see that it compares the SHAs to see if anything has changed. If nothing has changed, it moves on to the next asset.

Minifying and Combining

If you'll recall, something else I wanted was automatic minifying and combining of files so I only had to serve one file of each type rather than several. Mantis takes care of this by leveraging a few open source libraries: minify for CSS and JShrink for javascript. Mantis will create and published a minified version of your file, leaving your original (development) version of the file untouched and totally readable. To combine files, you simply feed Mantis an array of the files you'd like to combine and where you'd like this new file to be created, and the Mantis Manager will take care of the rest.

Busting Cache

It's a best practice to have a CDN sit in front of your asset server so that your assets are as close to your end users as possible, cached and ready to go. You could use Cloudfront to sit in front of your S3 bucket. With that comes a problem though, it could take a few hours to have asset updates pushed through to every node of the CDN. The best way to do this is to just change the URL of the requested asset altogether. The MM does this by sticking a version in the front of the path, your image could end up having assets/2/d4k8d33/image.png as its path, for example. That 2 represents the version it's currently on. That way the freshest possible version of each asset is being served, but we're not serving new versions of assets that haven't changed.

Referencing Assets in CSS

Having explained that, you may be thinking: "How can I reference my images from my CSS if all the assets' paths are constantly changing?". Good question. You can't just say ../images/image.png in your CSS, because you have no idea where that image is actually going to end up. Definitely not in the folder you think it's going to end up in, that's for sure. In your CSS, you can simply include this template @{{ asset("/images/image.png") }} and the MM will take care of the rest. Part of the processing loop is to look through the CSS files and replace references to assets with their current versions.

Referencing Assets from Yii

I've created a modified version of the standard CController that all my controllers inherit from that has an asset() method that you can call to get the appropriate reference to an asset. When you are in a view, you simply call $this->asset("/original/path/to/image.png") and Mantis will return the appropriate URL for the latest version of that asset.

Local vs Remote

You aren't going to want to be publishing to S3 for every little development change you make, so there is an option on the mantis command to specify local or remote (local is default).

> ./yiic mantis --type=local
> ./yiic mantis --type=remote

Local and remote keep their own separate caches with the SHAs, so you can publish 100 times locally and then run publish once to remote and all the files will be compared to the last time you published to remote. The local publish makes use of the standard Yii CAssetManager, while the remote publish relies on a heavily modified version of the S3AssetManager.

$post->end();

The Mantis Manager took me a couple days to write, I don't anticipate it should take you that long to implement, but I will warn you: This is not a drop-in replacement for any AssetManager class you are already using, it is much more integrated than that. There are likely things I haven't covered here that may be important, feel free to reach out or raise an issue over on the GitHub repo. I'm happy to do what I can.

If nothing else, I hope you can at least use pieces of my code to implement your own smart asset manager.

The only thing I'd still like to do is integrate this into a pre-commit git hook, so that before I push code to my staging/production sites, all my assets are ensured to be up to date.

I'll save that for another day though!