Deploying an Elixir app to Digital Ocean with mix_deploy

By Jake Morrison in DevOps on Mon 13 January 2020

This is a gentle introduction to getting your Elixir / Phoenix app up and running on a server at Digital Ocean (affiliate link). It starts from zero, assuming minimal experience with servers.

It builds the app on the server, then uses Erlang releases to run the code under systemd. It uses the mix_deploy library to handle deployment tasks.

Digital Ocean's smallest $5/month plan runs Elixir great. This guide uses their managed databases service so you don't need to manage the database.

We will be using a boilerplate Phoenix project with PostgreSQL database. It assumes you are running macOS on your dev machine and Ubuntu 18.04 on the server.

These instructions are based on this working example application and the principles described in the blog post "Best practices for deploying Elixir apps".

This post includes basic instructions to prepare your existing Elixir/Phoenix application for deployment using mix_deploy. See preparing an existing project for deployment for more details.

If you have any questions, contact me on the Elixir Slack at jakemorrison or open an issue on GitHub.

Overall approach

  1. Create the server
  2. Configure ssh
  3. Configure the build / deploy user
  4. Check out code on the server from git and build a release
  5. Deploy the release

You can first get the template running, then prepare your own project for deployment.

NOTE: This guide works with Ubuntu 18.04, CentOS 7, Ubuntu 16.04, and Debian 9.4. If you are not sure which distro to use, choose Ubuntu 18.04. The approach here works for dedicated servers and cloud instances as well.

The actual work of building and deploying releases is handled by simple shell scripts which you run on the build server or from your dev machine via ssh, e.g.:

ssh -A deploy@web-server
cd build/mix-deploy-example
git pull

# Build release
bin/build

# Extract release to target directory on local machine, creating current symlink
bin/deploy-release

# Run database migrations
bin/deploy-migrate

# Restart the systemd unit
sudo bin/deploy-restart

Create the server

Go to Digital Ocean (affiliate link) and create a Droplet (virtual server).

  • Choose an image: Choose Ubuntu 18.04
  • Choose a plan: Standard is fine
  • Choose a size: The smallest, $5/month Droplet is fine
  • Choose a datacenter region: Select a data center near you
  • Add your SSH keys: Select the "New SSH Key" button, and paste the contents of your ~/.ssh/id_rsa.pub file. Create an ssh key, if you don't have one already. On Mac OS, you can copy your SSH key to clipboard by running cat ~/.ssh/id_rsa.pub | pbcopy.
  • Choose a hostname: The default name is fine, but awkward to remember and type. Use "web-server" or whatever you like

The defaults for everything else are fine. Click the "Create" button.

Configure ssh to talk to your server

Note the IP address of your new droplet in the Digital Ocean UI.

Configure ~/.ssh/config on your local dev machine so you can connect to the server.

# Change the IP address below to the actual IP address of your Droplet
Host web-server
  HostName 123.45.67.89

Create a deploy user on the web server

For security, we use two operating system user accounts, the deploy user to build and deploy the app, and the app user to run the app.

Connect to the web server as root:

ssh root@web-server

Create the deploy user:

useradd -m -s /bin/bash deploy

Configure sudo to allow the deploy user run commands as root without a password:

echo "deploy ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/10-app-deploy

There are more sophisticated ways to manage users. We normally manage user accounts with Ansible.

Configure ssh access to the deploy user.

Create the .ssh directory and set permissions:

mkdir -p ~deploy/.ssh
chown deploy:deploy ~deploy/.ssh
chmod 700 ~deploy/.ssh

Allow the ssh key you set for the droplet root user to log into the deploy user account:

cp ~root/.ssh/authorized_keys ~deploy/.ssh
chown deploy:deploy ~deploy/.ssh/authorized_keys
chmod 600 ~deploy/.ssh/authorized_keys

Exit the ssh session and connect again using the deploy user.

ssh -A deploy@web-server

If it doesn't work, check that the ssh key on your dev machine (.ssh/id_rsa.pub) is in the ~/.ssh/authorized_keys file for the deploy user and check the file permissions. Try with -vv or look at /var/log/auth.log on the server.

The -A flag on the ssh command gives the session on the server access to your local ssh keys without copying them to the server. If your local user can access a GitHub repo, then you can do it on the server.

Make sure that the deploy user can run commands with sudo:

sudo -s
exit

Check out the app source

As the deploy user on the build machine, create the build dir:

mkdir -p ~/build

Check out the app source:

cd ~/build
git clone https://github.com/cogini/mix-deploy-example # or your app repo
cd mix-deploy-example

Install build dependencies

In order to build the app, we need to install Erlang, Elixir and Node.js on the build server. Run the following script to install Erlang, Elixir and Node.js from OS packages:

sudo bin/build-install-deps-ubuntu

See the instructions on the Elixir website for more details on installing Elixir and dependencies.

We generally use ASDF to manage build tools. That allows us to precisely specify versions and install multiple versions at once.

Create the database

Most apps use a database. You can install the database on the same Droplet as you run your app. It works fine and is cheaper, but then you have to manage the db. This guide uses Digital Ocean's managed databases service.

In the Digital Ocean UI, select Create → Databases.

  • Choose a database engine: Select PostgreSQL 11
  • Choose a cluster configuration: $15/month is fine
  • Choose a datacenter: Use the same data center as your droplet
  • Choose a unique database cluster name: The default name is fine

While the database is being created, in the “Getting started” section of the page, click the bullet point that says “Secure this database cluster.” Under “Restrict inbound connections” select your droplet and click “Allow these inbound sources only.” This ensures that only your application server can connect to the database.

Create app databases and users

If you are creating a database per app, you can use the defaultdb database and doadmin user that the setup wizard created for you. However, it is better to create a separate database and database user for each app environment.

Configuration

There are four kinds of things that we may want to configure:

  1. Static data, e.g. file paths. This is the same for all machines.

  2. Settings specific to the environment, e.g. the hostname of the db server.

  3. Secrets such as db passwords, API keys or TLS certificates.

  4. Dynamic attributes such as the IP addresses of the server or other machines in the cluster.

In a simple deployment, you can put all the configuration in config files like config/prod.exs. It will then be compiled into the release package.

When building on a different machine, e.g. a CI/CD system, it's more secure to keep secrets separate from the release and load them at runtime. Similarly, we may want to build a release and run it in a test environment, then deploy it in production.

In these cases, we keep the config outside the release file and load it at runtime. We might read it from environment variables or external config files. Or we might read it from a system like AWS Systems Manager Parameter Store. See "Best practices for deploying Elixir apps" for more details.

Building

On your build machine, build the app by running the build script:

bin/build

In addition to the normal Phoenix build steps, this command sets up the deploy scripts by running the following mix_systemd and mix_deploy commands:

# NOTE: run by bin/build
mix systemd.init
mix systemd.generate

mix deploy.init
mix deploy.generate

The configuration is minimal. We just change the name of the OS user that the app runs under to app in config/prod.exs:

config :mix_systemd,
  # Run db migrations before starting the app
  # exec_start_pre: [
  #   [:deploy_dir, "/bin/deploy-migrate"]
  # ],
  app_user: "app",
  app_group: "app"

config :mix_deploy,
  # Generate runtime scripts from templates
  templates: [
    # Systemd wrappers
    "start",
    "stop",
    "restart",
    "enable",

    # System setup
    "create-users",
    "create-dirs",
    "set-perms",

    # Local deploy
    "init-local",
    "copy-files",
    "release",
    "rollback",

    # DB migrations
    "migrate"
  ],
  app_user: "app",
  app_group: "app"

Prepare runtime configuration

Edit config/prod.exs with the runtime settings:

import Config

config :mix_deploy_example, MixDeployExample.Repo,
  username: "doadmin",
  password: "CHANGEME",
  database: "defaultdb",
  hostname: "db-postgresql-sfo2-xxx-do-user-yyy-0.db.ondigitalocean.com",
  port: 25060,
  ssl: true,
  pool_size: 15

config :mix_deploy_example, MixDeployExampleWeb.Endpoint,
  secret_key_base: "CHANGEME2"

You can generate a unique value for secret_key_base using this command:

mix phx.gen.secret

Build

Build the app and make a release:

bin/build

Initialize local system

Run this once to set up the system for the app, creating users and directories for releases, runtime configuration, etc.:

sudo bin/deploy-init-local

As this script changes group membership, you should log out and in again to reload user privileges.

Deploy the app

Deploy the release to the local machine:

# Extract release to target directory, creating current symlink
bin/deploy-release

# Run database migrations
bin/deploy-migrate

# Restart the systemd unit
sudo bin/deploy-restart

Check that it works

Make a request to the server:

curl -v http://localhost:4000/

You can get a console on the running release:

sudo -i -u app /srv/mix-deploy-example/bin/deploy-remote-console

You can also have a look at the logs:

sudo systemctl status mix-deploy-example
sudo journalctl -r -u mix-deploy-example

You can roll back the release with the following:

bin/deploy-rollback
sudo bin/deploy-restart

Configure the server to listen on port 80

Listening on port 4000 might be fine if it's behind a load balancer, otherwise we need to make the app available on port 80. There are two ways to do this:

After you complete this step, you should be able to access your website in the browser by navigating to your droplet's public IP address.

SSL

The steps necessary to get SSL set up with your Phoenix application depend on the approach that you took in the previous step. If you are forwarding ports using iptables, then you should set up SSL in your application's endpoint, as described in Phoenix docs. You can get an SSL certificate for free from Let's Encrypt.

If you are running behind an Nginx reverse proxy, you should instead set up SSL in Nginx. The necessary steps are described in Digital Ocean's tutorials.

How to prepare your Phoenix app for deployment

Following are the steps used to set up this repo. You can do the same to add it to your own project. This repo is built as a series of git commits, so you can see how it works step by step.

Generate Phoenix project

mix phx.new your_app
mix deps.get
cd assets && npm install && node node_modules/webpack/bin/webpack.js --mode development
  • Add mix.lock to git
  • Add package-lock.json to git

Configure releases

Elixir 1.9 has built in support for creating releases. For earlier versions, use the Distillery library.

Generate initial config files in the rel dir:

mix release.init

Check the rel directory into git.

In mix.exs, tell Mix not to include Windows executables in releases. In the main project configuration, add the option :releases:

def project do
  [
    app: :mix_deploy_example,
    version: "0.1.0",
    elixir: "~> 1.9",
    elixirc_paths: elixirc_paths(Mix.env()),
    compilers: [:phoenix, :gettext] ++ Mix.compilers(),
    start_permanent: Mix.env() == :prod,
    aliases: aliases(),
    deps: deps(),

    # add this line:
    releases: releases()
  ]
end

Then, in the same file, add a private function that returns your project's release configuration:

defp releases do
  [
    # change this to your application name
    mix_deploy_example: [
      include_executables_for: [:unix]
    ]
  ]
end

Tune the OS for performance

For optimal performance, it is recommended that you increase the default limits for open TCP ports and file handles. Follow the instructions in the post Tuning TCP ports for your Elixir app.

Add runtime config files

Loading runtime configuration from Elixir source files in Elixir 1.9 is very straightforward. The file config/releases.exs is copied into your release and evaluated at startup, and you can load your Elixir configuration file from within that file. Edit config/releases.exs:

import Config
import_config "/etc/mix-deploy-example/config.exs"

Create a config/prod.secret.exs.sample that you can use to generate production configuration files on the build server:

import Config

# Change these identifiers to ones specific to your application
config :mix_deploy_example, MixDeployExample.Repo,
  username: "CHANGEME",
  password: "CHANGEME",
  database: "CHANGEME",
  hostname: "CHANGEME",
  port: 5432,
  ssl: true,
  pool_size: 15

config :mix_deploy_example, MixDeployExampleWeb.Endpoint,
  secret_key_base: "CHANGEME2"

Add migrator module (optional)

In order for the bin/deploy-migrate script to work properly, you need to add a migrator module to your project. The instructions on how to do so are described in the post Running Ecto migrations in a release.

If your build server and production server are the same machine, you can also skip this step and just run your migrations with MIX_ENV=prod mix ecto.migrate.

Set up ASDF

Create a .tool-versions file in the root of your project, describing the versions of OTP, Elixir, and Node that you will be building with:

erlang 21.3
elixir 1.9.0
nodejs 10.16.0

Install mix_deploy and mix_systemd

Add libraries to deps from Hex:

{:mix_systemd, "~> 0.5.0"},
{:mix_deploy, "~> 0.5.0"}

Or from GitHub:

{:mix_systemd, github: "cogini/mix_systemd", override: true},
{:mix_deploy, github: "cogini/mix_deploy"},
end

Add rel/templates and bin/deploy-* to .gitignore:

echo '/rel/templates' >> .gitignore
echo '/bin/deploy-*' >> .gitignore

Copy build and utility scripts into your repo

Copy shell scripts from the bin/ directory of the mix-deploy-example repo to the bin/ directory of your project.

These scripts build your release or install the required dependencies:

  • build
  • build-install-asdf
  • build-install-asdf-deps-centos
  • build-install-asdf-deps-ubuntu
  • build-install-asdf-init
  • build-install-asdf-macos
  • build-install-deps-centos
  • build-install-deps-ubuntu

This script verifies that your application is running correctly:

  • validate-service

Check these scripts into git.

Configure for running in a release

In config/prod.exs, uncomment or add this line so that Phoenix can run correctly in a release:

config :phoenix, :serve_endpoints, true

In the same file, configure mix_deploy and mix_systemd to run your application as the app user. This step is mandatory:

config :mix_deploy,
  app_user: "app",
  app_group: "app"

# Minimal
config :mix_systemd,
  app_user: "app",
  app_group: "app"

Still in prod.exs, configure your application's endpoint to fetch port number from environment variables. The corresponding variable will be set by systemd:

config :your_app_name, YourAppNameWeb.Endpoint,
  http: [:inet6, port: System.get_env("PORT") || 4000],
  # ...

Confirm that everything compiles by building the app:

mix deps.get
mix deps.compile
mix compile

You should be able to run the app locally with:

# Create development database
mix ecto.create

# Compile assets with production settings
(cd assets && npm install && npm run deploy)

mix phx.server
open http://localhost:4000/

If everything seems to work, you can proceed with deployment just like you did with the mix-deploy-example sample application.