We publish our changelog on the site, which we extract from Heroku's release API, and push to GitHub's Gist API. Here's how we do it.

I've written a couple of posts already about our deployment process, and its creeping automation. The final step has always been updating the changelog, which we publish on the site.

Why do it all?

Making the changelog public serves no practical purpose, and there is always the danger that we post something confidential by mistake, so why do it? First off, it's easy to do, so why not - we may not be open source, but we can at least be transparent; I also like to think that doing this demonstrates our ongoing commitment to the platform (it's impossible to accuse us of not fixing things if we show everyone what we're fixing). And besides, everything that we push live is, well, live - so we're not giving anything away.

Publishing the changelog also forces us to take a little more care in our commit messages, which is a pretty good discipline, although it can cause some embarrassment. I tweeted yesterday:

That feeling when you see a commit labelled "WIP - do not push" heading off to a remote repo, http://t.co/J1OL7MN65Y

Only to see my error appear on the site later in the day:

Anyway - that will teach me to rebase locally next time! And we do at least now have some stats on our backoffice homepage :-).

I've finally gotten around to automating this step, and I thought I'd post some of the code involved, as it's not specific to us, and it shows some of the information available from the Heroku API, and how to push content to a Gist through the GitHub API. (The changelog is hosted on GitHub as a Gist, which just happens to be the easiest way to embed snippets like this.)

The basic process is this:

1. Fetch the complete list of releases from the Heroku API
2. Calculate the set of (non-merge) commits between each release
3. Push the list of changes to a Gist

1. Fetch release info from Heroku

>>> from os import environ  # this is where we keep our secrets
>>> HEROKU_APP_NAME = environ['HEROKU_APP_NAME']  
>>> HEROKU_API_KEY = environ['HEROKU_APP_KEY']
>>> HEROKU_API_URL = 'https://api.heroku.com/apps/%s/releases'

# return JSON containing all your application releases
>>> import requests
>>> auth = requests.auth.HTTPBasicAuth('', HEROKU_API_KEY)
>>> releases = requests.get(HEROKU_API_URL % HEROKU_APP_NAME, auth=auth).json()

This will return a lot of information - including all of your environment settings and addons, for each release. We only need a few fields, so next we parse them out:

>>> commits = [(r['name'], r['commit'], r['created_at']) for r in releases]

We don't want to publish every release, as that would get out of hand, and if we want the 'latest X releases' we need order them by name (which is the text value that Heroku sets for a release, e.g. 'v123' where '123' is the version number), in reverse order:

>>> commits = sorted(commits, key=lambda x: x[0], reverse=True)

We now have a list of 3-tuples, in reverse chronological order (latest release first), containing the 'name', commit hash and date for each.

>>> commits
[
    (u'v158', u'e950aac', u'2013/09/26 03:56:19 -0700'),
    (u'v157', u'37e67b8', u'2013/09/25 11:20:56 -0700')
]
>>>

At this point we have everything we need to move on to git. We are using the local git repo to extract our diff. We need to iterate through the releases, extracting the current and previous commit hashes for each. From there we can call our git_log function to retrieve the individual commits that exist between the two hashes:

>>> changelog = []  # this is what we'll use to build the Gist contents
>>> history = 10  # the number of releases we want to print
>>> for i in range(history):
...     commit = commits[i]
...     previous = commits[i+1]
...     # print out a header line for each release
...     changelog.append("\n**%s, deployed on %s**\n" % (commit[0], commit[2]...     # this function gets the git log info, and is described below
...     log = git_log(previous[1], commit[1])
...     # print append each individual commit description
...     for line in log:
...         changelog.append('* %s' % line[1])
...
>>> # remember this - we'll need it later
>>> printable_output = "\n".join(changelog).strip()
>>> print printable_output
** v158, deployed on 2013/09/26 03:56:19 -0700 **

* Fix for failing tests
* Refactoring of inline css

** v157, deployed on 2013/09/25 11:20:56 -0700 **

We now have a list that contains our changelog, nicely formatted, and ready for pushing to GitHub. Before describing that step, I will just mention our git_log function, which extracts the commit history from git itself.

2. Extracting commit history from git

The specific git command to run (on the command line) is trivial. Using the commit examples above, for release 'v158' this would be:

$ git log --oneline --no-merges 37e67b8..e950aac"

The only subtlety that we have is that we have separate code and config repositories, and the deployment script is in the config repo. Which means that running the above command from the config directory (where the Fabric file sits) won't work. The solution is the use of the --git-dir option, which must be the first option on the line. If we assume a directory structure like this:

my_project$ tree
.
├── config
│   ├── fabfile.py
│   ├── [other config]
│   └── [etc]
└── src
    ├── .git
    ├── readme.md
    └── requirements.txt

, the full command becomes:

~/my_project/config$ git --git-dir=../src/.git log --oneline --no-merges 37e67b8..e950aac
81a5ea8 Fix for failing tests
62d49e9 Refactoring of inline css styles
~/my_project/config$

We run this command using the Fabric local function, so that we can capture the output, which we parse into a 2-tuple for ease of use:

>>> GIT_DIR = '../src/.git'
>>> from fabric.api import local
>>> def git_log(commit1, commit2):
... "Return list of changes between two commits, as a (hash, comment) 2-tuple."
...     diff = local("git --git-dir=%s log --oneline --no-merges %s..%s"
...                  % (GIT_DIR, commit1, commit2), capture=True)
...     return [(d[:7], d[7:]) for d in diff.decode('utf8').split('\n')]
...
>>>

If we run this function on the commit hashes from above we get:

>>> git_log('37e67b8', 'e950aac')
[
    ('81a5ea8', 'Fix for failing tests.'),
    ('62d49e9', 'Refactoring of inline css')
]
>>>

3. Push to GitHub

First off, in order to be able to push to GitHub via the API, we need to get some auth in place. Generating test authentication keys is a well-known nightmare, but fortunately GitHub have made it easy to get tokens for command line access - Personal Access Tokens - https://github.com/settings/applications. Once you have a token, it's a very simple API 'patch' call:

>>> from os import environ
>>> GIST_ID = environ['GIST_ID']
>>> # Personal Access Token from Github - https://github.com/settings/applications
>>> AUTH_TOKEN = environ['GITHUB_AUTH_TOKEN']
>>> AUTH_USER = environ['GITHUB_AUTH_USER']
>>> auth = requests.auth.HTTPBasicAuth(AUTH_USER, AUTH_TOKEN)
>>> url = 'https://api.github.com/gists/%s' % GIST_ID
>>> headers = {'content-type': 'application/json'}
>>> today = datetime.date.today().strftime('%A, %d-%b-%Y')
>>> data = json.dumps({
...     'description': 'Last %i deployments to yunojuno.com, automatically generated on %s' % (history, today),
...     'files': {
...         'changelog.md': {
...             'content': printable_output
...         }
...     }
... })
...
>>> patch = requests.patch(url, auth=auth, headers=headers, data=data)
<Response 200>
>>>

And that, my friends, is that.