Clojars Server Config

This repo contains the Ansible config for building the AMI for the Clojars server, the terraform for managing the Clojars infrastructure on AWS, and scripts to deploy a Clojars release.

System Diagram

Setup

AWS Credentials

You will also need a AWS access key, exported as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. These vars need to be set to run terraform, build an AMI, or deploy. You will also need to set CLOJARS_SSH_KEY_FILE to the path to the private key used by the server if you want to deploy or ssh in to the server.

clojars-env script

One way to have all those vars set is to create a wrapper script that sets them (called clojars-env in this example):

#!/bin/bash

export AWS_ACCESS_KEY_ID=ASDFASDFASDF
export AWS_SECRET_ACCESS_KEY=3ASD3434AA
export AWS_REGION=us-east-2
export CLOJARS_SSH_KEY_FILE=~/.ssh/clojars-server.pem
export DNSIMPLE_TOKEN=dnsimple-api-v2-token
export DNSIMPLE_ACCOUNT=12345

exec $@

Then execute commands with:

clojars-env terraform apply

direnv

Alternatively, you can use direnv to set the environment variables.

Create a .envrc file at the root of the repo:

export AWS_ACCESS_KEY_ID=ASDFASDFASDF
export AWS_SECRET_ACCESS_KEY=3ASD3434AA
export AWS_REGION=us-east-2
export CLOJARS_SSH_KEY_FILE=~/.ssh/clojars-server.pem
export DNSIMPLE_TOKEN=dnsimple-api-v2-token
export DNSIMPLE_ACCOUNT=12345
PATH_add bin

DNSIMPLE_TOKEN is a v2 API token generated from https://dnsimple.com/user → "Access tokens". DNSIMPLE_ACCOUNT is the numeric account ID (visible in the DNSimple URL when signed in).

Install direnv and run direnv allow in the repo directory. Now, everytime you cd into the repo directory, the environment variables will be set.

Terraform

We have a wrapper around terraform (bin/terraform) that will download and install the correct version (cached in bin/.cache/).

Initialization

The terraform state is stored in S3 and uses a DynamoDB table to lock that state when it is being altered. On first run, you will need to initialize terraform with (this assumes you have set up direnv as above to use bin/terraform):

cd terraform
terraform init

Applying the configuration

cd terraform
terraform apply

DNS records (DNSimple + Route 53)

DNS for clojars.org and clojars.net is served by two authoritative providers in parallel: DNSimple and AWS Route 53.

Both zones are managed by Terraform in terraform/dns.tf, and each record is declared once via the ./modules/dns-record module — applying writes the same record to both providers in lockstep. Make DNS changes by editing dns.tf and running terraform apply — not in either provider's dashboard.

Packer

We have a wrapper around packer (bin/packer) that will download and install the correct version (cached in bin/.cache/).

Sensitive Configuration Data

We store sensitive configuration data in AWS SSM parameters.

the following parameters exist currently (ones marked with a 🔒 are encrypted):

• /clojars/production/ami_id
• /clojars/production/cdn_token 🔒
• /clojars/production/db_host
• /clojars/production/db_password 🔒
• /clojars/production/db_user 🔒
• /clojars/production/github_oauth_client_id 🔒
• /clojars/production/github_oauth_client_secret 🔒
• /clojars/production/gitlab_oauth_client_id 🔒
• /clojars/production/gitlab_oauth_client_secret 🔒
• /clojars/production/sentry_dsn 🔒
• /clojars/production/sentry_token 🔒
• /clojars/production/ses_password 🔒
• /clojars/production/ses_username 🔒
• /clojars/production/ssh_keys

You can retrieve the value of a parameter with:

aws ssm get-parameter --name <name> --query "Parameter.Value" --with-decryption

Listing running instances

There is a convenience script to list all EC2 instances:

scripts/list-instances.sh

Deployment

To deploy a new release of Clojars, you have a few options:

• You can build and upload a new release to S3, then deploy a new AMI that will pick up the release (see below)
• You can build and upload a new release to S3, then request that a running server switch to that release
• You can also switch back to an older release

To build and upload a new release, first tag the release (in clojars-web):

make tag-release

This will create and push a tag of the form <date>.<commit-count> (example: 2023-08-20.1982).

Then run:

scripts/upload-release.sh <version-tag>

This will check to see if an artifact for that tag already exists in the deployment bucket. If not, it will pull down the tag from GitHub, build an uberjar, then upload a zip containing that uberjar and the scripts/ dir from clojars-web to the deployment bucket.

It then writes a current-release.txt containing the tag to the deployment bucket.

To deploy a release to a running server, run:

scripts/deploy.sh <server-ip> <version-tag>

This will first call scripts/upload-release.sh, then ssh to the server and run the deploy-clojars script. This script will pull down the version specified by current-release.txt and deploy it. This script is the same script that runs when the Clojars AMI boots.

Building an AMI

We build a custom AMI using packer, and apply changes to the AMI with ansible. To run packer, call (this assumes you have set up direnv as above to use bin/packer):

scripts/build_ami.sh

This will take a few minutes, but will produce a new AMI. The ID of the new AMI will be written to the /clojars/production/ami_id SSM parameter, which is read by terraform/asg.tf.

Deploying a new AMI

1. Run terraform apply in terraform/. This will pick up and apply the new AMI ID from the /clojars/production/ami_id SSM parameter.
2. Changes to a launch configuration don't affect running instances, so we will have to force a new instance. You can do so by running scripts/cycle-instance.sh.

Ansible Guidelines

• Follow Ansible best practices
• Add an {{ ansible_managed }} comment in the header of all templates and files

Distributed under the MIT License. See the file COPYING.