Just F***ing Ship It has become the mantra for the modern age. In software you can refine that to “Contiuously F***ing Ship It” because of the trend toward Continuous Integration and SaaS software development. Given that, I believe a highly important and routinely ignored part of any healthy software project is a decent build system. The great thing about modern software development is there are many different options available to us. You aren’t just tied to make and other ancient tools. (Don’t get me wrong, make is a great tool, but so was CVS back in the day, times have moved on and so have project sizes and complexity.)
The build system I updated recently was a Grunt system for a web front-end. Previously it had been been a bit of an odd structure:
Other problems with the build system included:
- No easy way to build it locally (it used hard-coded values for a build server).
- If you did manage to build it locally you would inevitably have edited one or more of the additional package.json files. Meaning you ran the risk of accidentally checking those back in.
First let me explain the structure. Each part of the build system was a whole self contained Grunt build system. In fact there was a fair bit of copy/pasta going on here. So it volatilized the DRY principal to an extreme extent. Given that they were individual build systems it would mean you’d have to do an npm install in several different directories and some of the plugins/dependency would have to be downloaded in multiple places and the package.jsons had gone out of sync, meaning some of the build system depended on new versions of certain plugins. Clearly a bit of a disaster.
It was time to clean things up.
Grunting in unision
My task was to merge all of the different grunt files and package.jsons into one place. I started with the package.jsons, which seemed like an easy enough job. I reasoned that there would be no adverse affects to getting the build system to converge on the latest version of each of the plugin. I was correct and was able to get a workable package.json fairly quickly.
I knew looking at the different grunt files that merging those into a single file right away would be a bad idea. So I researched a few plugins which would help me split the contents up a bit. I found a couple of great articles on managing grunt build system and the two plugins that stood out from that research were:
Splitting into multiple files using load-grunt-config
Load-grunt-config is a great plugin. It is a nice little piece of work created by [Anatoliy Syernyi][asyernyigithub] which handles the loading of plugins configurations and tasks for you. This allowed me to create a whole new file structure that looks something like this:
Let me explain the structure. Now there was a single entry point for the build system, one single Gruntfile. One single package.json. Now one npm install was required to set everything up. All the different plugins the build uses belong there, as for the individual tasks, they would be defined in a file of there own under tasks. This looked like over kill at first, but eventually you have very specific handler routines and configuration for some tasks which it would make sense to encapsulate in a single location. I also include a special utils area where shared functionality would live (helpers.js) and derived project paths (globalSettings.json).
The major and important element of this restructure was the devSettings.json.sample file. This is a special configuration file that would sit on each developers machine and when renamed to devSettings.json would allow them to have their own local file paths/and overrides for the various build settings. The file devSettings.json was made to be ignored by the source control system meaning there was no chance of it accidentally getting checked back in.
I got this to work by using the postProcess handler method that load-grunt-config exposes which passes you the an object instance of the package.json file content. Here you can use a plugin like deep-extend to merge those package.json settings with your own personal overrides, which is what I did.
One of the upsides of using load-grunt-config was it allowed me to use a plugin called jit-grunt. This plugin reports to be able to have a build system where:
“Load time of Grunt does not slow down even if there are many plugins.”
This sounds great and I used it right out of the box. I don’t think the amount of plugins that are there currently are causing that much of an issue, but I know I want developers to use the build system as a part of their daily work flow and that the run speed was going to be an issue so I decided to be proactive. Anyway, it was only 3 lines of code to active it.
Plugins that don’t play nice
One of the downsides of load-grunt-config is it depends on somewhat of a defacto naming pattern for plugin tasks and their npm names. This can fall down quite easily, I encounter this with two plugin in this build system:
In the end I had to sully the main grunt file with two grunt.loadNpmTasks which annoyed me greatly.
Mo' Plugins Mo' Problems
One catch with using jit-grunt is because it does not load tasks into the namespaces until they are explicitly called; then running grunt –help and grunt –version –verbose no longer worked properly. As a result I am currently unable to do any kind of bash autocompletion of the tasks for my newly cleaned up build system :(. There might be a way to get it back, I’ve put in a github ticket against grunt-cli to see what those guys think of my issue.
The previous build system was lacking on the logging front. If something went wrong it would typically get swallowed, which is a pain because being able to understand what went wrong and how long it might take to fix it is important. Throughout this restructure and rewrite process I took the trouble of using grunt.log, grunt.fail.warn and grunt.fail.fatal where appropriate.
I also made sure to include an extra command line flag, –debug, so our email reports wouldn’t be drowned in debug details.
Additional cool stuff
- Cool and advanced task/time tracking. I haven’t gotten around to include the details provide here in the reported output, but it is handy in debug mode.
- Get your files to the server all in one simple grunt step: ‘'grunt buildAndFtp’'
Using this set up I was able to create a build system was:
- Easily enhanceable
- Easy to setup (single npm install in single folder)
- Locally override-able using a local developer settings file.
- Without the risk of accidental check-ins