All of the packages listed below provide some value to our platform -
and the majority started out life as internal functions / modules that
we wrote as part of our daily work. At some point in time they became
sufficiently generic / non-specific that they became candidates for
releasing publicly. The reason for doing so is principally because it
forces the developer to think about the problem they are solving in a
more rounded way. The solutions that come out of treating a problem as a
piece of drop-in functionality are nearly always cleaner and clearer
than the equivalent code placed in the middle of a 1,000LOC module.

An additional benefit is that forcing the developer to acknowledge that
the code is going to be public subconsciously makes them try a little
bit harder - comments are clearer, code is tidier, tests more complete.
An old trick when reviewing code with junior developers was to ask how
they'd feel if it was presented as example code in 128pt font to an
audience of their peers at a conference. Writing code 'in public' has
the same effect.

A final benefit is just that it's easier to read / reason code that is
isolated from the rest of the behemoth that you are building elsewhere.
Most of the packages are small and simple - a small suite of tests has a
large degree of coverage, and they all come with a README that explains
how it works, and what problem it solves.

That's enough preamble - this is the list we'll go through in due course:

django-appmail           # Django app for managing localised email templates.
django-charid-field      # Provides a char-based, prefixable ID field for your Django models. Supports cuid, ksuid, ulid, et al.
django-command-log       # Django management command auditing app
django-countries         # Provides a country field for Django models.
django-csv-downloads     # Django app for enabling and tracking CSV downloads
django-frozen-field      # Django model field used to store snapshot of data.
django-geoip2-extras     # Additional functionality using the GeoIP2 database.
django-magic-link        # Django app for managing tokenised 'magic link' logins.
django-nps               # Django app supporting Net Promoter Score (NPS) surveys.
django-onfido            # Django app for integration with Onfido.
django-perimeter         # Site-wide perimeter access control for Django projects.
django-project-checks    # Django management commands used to output useful project information.
django-request-logger-2  # Django model for storing HttpRequest information.
django-request-profiler  # A simple Django project profiler for timing HTTP requests.
django-request-token     # JWT-backed Django app for managing querystring tokens.
django-s3-upload         # Integrates direct client-side uploading to s3 with Django.
django-side-effects      # Django app for managing external side effects.
django-stripe-lite       # A library to aid Django integration with Stripe.
django-user-visit        # Django app used to track user visits.
django-utm-tracker       # Django app for extracting and storing UTM tracking values.
django-visitor-pass      # Django app for managing temporary session-based users.
django-zapier-triggers   # Simple Django app for managing Zapier triggers.
elasticsearch-django     # Elasticsearch Django app.
python-env-utils         # Utility functions to make it easier to work with os.environ

Here's the story: After a year in beta, @Glitch opens up today, ready for you to build and launch real, live apps. And the app that runs http://glitch.com itself is now open source, so you can directly help shape Glitch's features and future.
@anildash

On this, the day after @anildash's announcement, I thought I'd share how we've been using Glitch to automate Github, and personalise our own team workflow.

tl;dr Glitch is a great webhook platform - it's like IFTTT for developers, and when combined with Github's API can automate almost any git workflow you can think of.

We use gitflow (with some modifications) as our git workflow in combination with Github pull requests, and Jira for tickets. Whilst this combination works well enough, without any automation there are a number of steps that require manual intervention, and are prone to errors:

1. Incorrect merges - with gitflow a hotfix should be branched from and merged into master (as well as dev, more on that later) - it's easy to forget;

2. Pull request titles - by convention we prefix PR titles with the Jira ticket number - YJP-123: Implement feature foo;

3. Labels - we like to label hotfix PRs;

4. Reviews - we insist on two approvals per PR - something that GH does not natively support.

Using Glitch as a webhook handler, we have now automated all of the above. We have a series of Github status checks (like the CI build checks you see) that enforce all of the rules, flag any errors, and apply labels. In addition, it will automatically open and maintain a Release PR, so that we can always see what is ready to deploy. I built the initial handler myself, starting from a JS knowledge of (effectively) zero - and it was a great developer experience. (Pls don't judge me on the code - as I said, I don't really know anything about JS.)

This is the handler that enforces two reviews:

// Check the number of approvals on the PR, and respond with an appropriate
// status ('pending' if not enough, 'success' if there are) and message.
app.post('/approval_check', (req, res) => {

  let status;
  let pr = req.body.pull_request;

  if (utils.isRelease(pr)){
    status = "No approvals required for a release";
  }
  else {
    github.countApprovals(pr).then(approval_count => {
      if (approval_count === 0){
        status = github.sendStatusUpdate(pr, "approvals", "pending", "Requires at least two approvals");
      }
      else if (approval_count === 1){
        status = github.sendStatusUpdate(pr, "approvals", "pending", "Requires one more approval");
      }
      else {
        status = github.sendStatusUpdate(pr, "approvals", "success", approval_count + " approvals received");
      }
    });
  }
  res.send(status);
});

The Github call to update the status is:

// send the PR status check update
sendStatusUpdate = async function(pull_request, context, status, description){
  let retval = status + " on '" + pull_request.title + "': " + description;
  console.log(retval);
  // https://octokit.github.io/rest.js/#api-Repos-createStatus
  let response = await octokit.repos.createStatus({
    owner: pull_request.head.repo.owner.login,
    repo: pull_request.head.repo.name,
    sha: pull_request.head.sha,
    state: status,
    description: description,
    context: "Junobot (" + context + ")"
  });
  return retval;
}

And the output looks like this:

In addition, we can now automatically create a Release PR (dev to master), so that we always have an open release ready to merge / deploy - that details the list of all the constituent Jira tickets, and also has the labels from all the merged branches applied, so we can see if a release requires database migrations, feature flags, etc.

// If this is a merge event, then update the release PR.
if (req.body.action === "closed" && pr.merged_at){
  github.getRelease(pr.head.repo).then(release => {
    // NB this will add all labels, so if you still
    // have WIP or DNM, these will be added to the release!
    github.addLabels(release, github.getLabels(pr));
  });
}

All of this is created automatically on our behalf:

Emboldened by the relative success of the above, I decided to have a go at automating the merge itself - we like to squash feature and hotfix branch merges, so that we have a clean dev, and devs can easily forget this. In addition, with gitflow it is necessary to merge a hotfix into master and dev at the same time. This is not something that Github supports, and is the one step in the process that devs still complete locally, rather than through the GH interface. Automating this completes the process.

// Merge a numbered pull request, using a specific method
// ('merge', 'squash'), and then merge back into dev if
// this is a hotfix
const mergePullRequest = async (pull_request, merge_method) => {
  let repo = pull_request.base.repo;
  return octokit.pullRequests.merge({
    owner: repo.owner.login,
    repo: repo.name,
    number: pull_request.number,
    merge_method: merge_method,
  }).then(resp => { 
    // NB assuming that it merged OK     
    return resp.data.sha;
  }).then(sha => { 
    if (utils.isHotfix(pull_request)){
      mergeBranch(repo, "dev", sha, "Merge hotfix: " + pull_request.title);
    }
    return "OK";
  }).catch(error => {
    return "Error";
  });
};

This update to Junobot has also involved a sinister change in personality:

We now have no option but to comply. I for one welcome our robot overlords.

The merging webhook is available for remixing here: https://glitch.com/edit/#!/bog-rifle

Background

AWS Lambda is Amazon's 'serverless' or Function-as-a-Service, product. They were first out of the gate, but Google has Cloud Functions, and Microsoft has Azure Functions, so everyone's at it - they are officially a thing. The concept is pretty simple - written a small single-purpose function (in the language of your choice - they all support a broad range of environments) - deploy it to the relevant platform, and configure it to fire on a specified event.

We chose AWS because it suited out purpose best - we had a requirement to push a data extract from Amazon Redshift (our data warehouse) to a remote SFTP server, and the recommended method for extracting Redshift data is to use the UNLOAD command which will publish an extract file to S3. The bit we had to write was how to get the file from S3 to the SFTP server, and to do this on receipt of the extract. It's the perfect Lambda scenario. All we had to do was write the function, and deploy it.

Packaging Python3 Functions

1. Write a python function

At its simplest, a Lambda function is literally just that - a function. We chose Python as our runtime environment, so all we had to do was write a function with the correct signature:

def my_lambda_function(event, context):
    print("Hello, world")

If the function were this simple, you could write it directly into the Lambda console, however most functions will rely on some external dependencies, and if that's the case, you'll need to create a function package.

2. Add dependencies

To package up a function that has external dependencies, you need to create a zip archive that contains the dependencies and your function module. The Amazon docs are pretty good on how to do this (http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html) - and if you follow them through you'll have a package.zip to upload in no time. The following Makefile will do this for you:

package:
  # install dependencies into package directory
  pip install -r requirements.txt -t package
  # copy in your lambda .py source file(s)
  cp my_script.py package/
  # zip up the entire directory into package.zip
  cd package; zip -r ../package.zip .
  # always clean up after yourself
  rm -rf package/

This works very well so long as your dependencies have no C extensions or platform-specific installs - if you do (e.g. image processing, crypto, etc.), then you'll find the function will run fine locally, but fail when uploaded to AWS, because AWS isn't running macOS (/Ubuntu/Windows/...) - it's running amazonlinux.

3. Adding platform-specific dependencies

In order to create a package that contains the right packages for amazonlinux, you need to run the pip install from within an amazonlinux environment. Fortunately Amazon make available a range of Docker containers to make this pretty easy. Unfortunately, the default image does not come in a Python3 variant, so you'll need to install that (as well as zip).

4. Add Python3 support

Although the docs will tell you that there isn't any formal support for Python3 yet, it turns out that there is a distribution available, and installing it is pretty simple. This Dockerfile will install python3 and zip (so we can zip up the package):

FROM amazonlinux:latest
# Install python3.6 and zip
RUN yum install -y python36 zip 

# This is the mount location for the Lambda function directory
VOLUME ["/lambda"]
WORKDIR "/lambda"

# Default entrypoint / command is to package the function
ENTRYPOINT ["make"]
CMD ["package"]

Using this Dockerfile you can build an image that can be used to package up your application for deployment to Lambda. You need to run make command from within the container, thus:

cd path/to/project
# build the new image
docker build -t lambda-packager .
# run a single-use container off the image to create the package
docker run --volume $(pwd):/lambda lambda-packager

This should produce a single package.zip archive that contains your script together with all the dependencies required, built for the amazonlinux environment. Upload this to AWS, and test it out.