Fabric, fabric, everywhere...
Using Fabric to automate pretty much everything
If you are a python developer and you don't already use Fabric, then you should. I can't imagine working without it.
When we started out on the YunoJuno journey we built a prototype, and deployed it. It worked. We deployed it to Heroku, which made things easy for us, as a deployment is really just a git push
. Well, it's a git push
, followed by a Django syncdb
as we updated the underlying models. And then we started growing up, and integrated South migrations, so then we had to add migrate
to the list of actions. Followed up quickly by a collectstatic
, and finally a little bit of load_data
. All of this we still did by hand - it was pretty easy, and not much went wrong.
And then we started to expand out our environments - we had a live environment, of course, but also 'dev' and 'uat' environments. And then someone wanted to push a new branch to a new environment, so we span up 'dev2', at which point someone else suggested a dedicated 'demo' environment. By this time we also had to configure settings for SMTP services, Postgres databases, Redis and Memcache. Some environments had DEBUG=True
, others did not. Some use async queue, others do not. The matrix of settings, services, branches, and data is very hard to even visualise, let alone manage.
All of this would have become a full time job to manage, if it hadn't been for Fabric and the magic of automation. If you are a Python developer, Fabric is such a natural extension that it's almost easier to write a Fabric task to do something than it is to work it out on the command line. And the extra ten minutes spent documenting the task, and working out the relevant task parameters and default options gives you just enough time to work out in your own head what you are really trying to achieve.
Initially our Fabric script was used to deploy code to all of our environments, except live, which, for some mad, sub-conscious, reason I thought was better to do by hand. In retrospect this is completely wrong - if you need something to be bulletproof and repeatable, then that's the one thing you should automate. So now everything is automated, and just so we don't do something we regret, deployments have a human 'turn the key' moment in them.
These are our current Heroku automation tasks:
# Compare values in conf files with current Heroku app settings.
$ fab heroku_check_vars
# Runs collectstatic on live, uat or dev environments.
$ heroku_collectstatic
# Deploys django app to target environment.
$ heroku_deploy_app
# Restores a copy of live to uat or dev, and anonymises the data.
$ heroku_migrate_live_data
# Configures an Heroku Postgres instance as the default database.
$ heroku_promote_postgres
# Provision a new Heroku app ready for deployment (install add-ons
$ heroku_provision_app ).
# Configure Heroku app with an email service.
$ heroku_provision_email
# Set environment variables on Heroku.
$ heroku_set_vars
# Display listing of all valid Heroku environments.
$ heroku_list_environments
It's not just our remote deployments that we now automate however, we also automate our interactions with our Vagrant VMs, and our local development environment:
# Runs django_admin commands on the VM.
$ fab vm_django_admin
# Runs the django collectstatic command on the VM.
$ fab vm_collectstatic
# Runs the `fab dump_fixtures` command on the VM.
$ fab vm_dumpfixtures
# Kills any running django devserver processes on the VM.
$ fab vm_kill_devserver
# Kills any running Gunicorn processes on the VM.
$ fab vm_kill_gunicorn
# Runs the `fab load_fixtures` command on the VM.
$ fab vm_loadfixtures
# Runs the django migrate command on the VM.
$ fab vm_migrate
# Reloads the Nginx conf file from local repo.
$ fab vm_nginx_reload
# Runs pip install on the requirements.txt file.
$ fab vm_pip_install
# Import the dev data into a local databse. (Not another Heroku env.)
$ fab vm_restore_live_data
# Runs the Django dev server on Vagrant VM.
$ fab vm_run_devserver
# Runs the `foreman run` command on the VM.
$ fab vm_run_foreman
# Runs the app through Gunicorn on Vagrant VM.
$ fab vm_run_gunicorn
# Runs the actual Procfile through Foreman on the VM.
$ fab vm_run_procfile
# Runs the fab run_tests command on the VM.
$ fab vm_run_tests
# Rebuilds the Solr schema, then drops and rebuilds the index.
$ fab vm_solr_update_schema
# Runs the django syncdb command on the VM.
$ fab vm_syncdb
Frankly, if it moves, automate it.
Making Freelance Work