Phoenix: Automated build and deploy made simple

I’ll detail how we’ve made a straightforward build and deploy process for our Phoenix setup at Nomad Rental. I found some blog posts that led us most of the way but thought it would be great to have an A-Z guide, that can be easily modified for your particular setup.

This blog post by Piotr Włodarek helped us a long way.

What do we want?

• Build and deploy locally and on CI
• Automatic deploys with master branch builds from CI
• Build inside a Docker container
• Clean builds from master branch on GitHub (no dev pollution)
• Encrypted production secrets
• Date versioned releases
• Manually rollback if necessary

Quick note

We’re using my_app and MyApp to reference our app. Replace with your actual app name. There’s a sample repo on GitHub.

Ansible

We’ll use ansible to build and deploy our release. It can also be used to provision your production server. The sample repo contains all the roles and tasks we’ll go through here.

Docker

First, we’ll set up a docker image for our builds. Make sure that you’ve docker installed already. The docker image should match your own production server setup.

If the docker image below already matches your production setup, then you can skip the following steps and just use our docker image (docker pull dreamconception/phoenix-build-elixir-ubuntu18).

We’re using Ubuntu 18 on our production server, so we’ll add the following to .ansible/apps/build/Dockerfile:

FROM ubuntu:18.04

WORKDIR /app

RUN apt-get update && apt-get install -y curl locales

# Set locale
RUN locale-gen en_US.UTF-8
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
ENV HOME=/root
ENV PATH="$HOME/.asdf/bin:$HOME/.asdf/shims:$PATH"

# Install dependencies
RUN apt-get update && \
    apt-get install -y aptitude ca-certificates python python-dev python-pip \
                    python-virtualenv \
                    git \
                    nodejs \
                    automake autoconf libreadline-dev libncurses-dev libssl-dev libyaml-dev libxslt-dev libffi-dev libtool unixodbc-dev unzip && \
    rm -rf /var/lib/apt/lists/*

# Install asdf
RUN git clone https://github.com/asdf-vm/asdf.git $HOME/.asdf --branch v0.7.2

# Install node js
RUN asdf plugin-add nodejs https://github.com/asdf-vm/asdf-nodejs.git && \
    $HOME/.asdf/plugins/nodejs/bin/import-release-team-keyring && \
    asdf install nodejs 11.8.0 && \
    asdf global nodejs 11.8.0 && \
    rm -rf  /tmp/*

# Install erlang
RUN asdf plugin-add erlang https://github.com/asdf-vm/asdf-erlang.git && \
    asdf install erlang 22.0.2 && \
    asdf global erlang 22.0.2 && \
    rm -rf  /tmp/*

# Install elixir
RUN asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git && \
    asdf install elixir 1.9.0 && \
    asdf global elixir 1.9.0 && \
    rm -rf  /tmp/*

# Install hex and rebar
RUN mix local.hex --force
RUN mix local.rebar --force

# Set up ansible
RUN apt-get update && \
    apt-get install -y software-properties-common && \
    apt-add-repository ppa:ansible/ansible && \
    apt-get update && \
    apt-get install -y ansible

We’ll build and upload the docker image to Docker Hub. You should sign in and set up your organization if you don’t have one already, and create a repository.

Create .ansible/apps/build/update-docker-image.yml with:

---
- hosts: 127.0.0.1
  connection: local
  gather_facts: no

  tasks:
    - name: Build and upload docker image
      docker_image:
         path: ./
         name: YOUR_ORG/phoenix-build-elixir-ubuntu18
         repository: YOUR_ORG/phoenix-build-elixir-ubuntu18
         tag: 1.9.0
         pull: yes
         push: yes
         force: yes

Remember to replace YOUR_ORG with your organization name. Now you can run the following commando in .ansible/ to build and push your docker image:

ansible-playbook apps/build/update-docker-image.yml -vvv

We’re ready to build the app!

Mix release

We’ll use the build date and time for the build version in the form of 0.1.0-2019.6.12.4.55. Leading zeros are removed from the timestamp since mix release requires valid SemVer. Update mix.exs so the build :version can be set as an environment variable. We’ll also set the :releases option here:

defmodule MyApp.MixProject do
  use Mix.Project

  def project do
    [
      app: :my_app,
      version: System.get_env("BUILD_VERSION") || "0.1.0",
      elixir: "~> 1.9",
      elixirc_paths: elixirc_paths(Mix.env()),
      compilers: [:phoenix, :gettext] ++ Mix.compilers(),
      start_permanent: Mix.env() == :prod,
      aliases: aliases(),
      deps: deps(),
      releases: [
        my_app: [
          include_executables_for: [:unix],
          applications: [runtime_tools: :permanent]
        ],
      ]
    ]
  end

  # ...
end

Local build

Create .ansible/apps/build/inventory and use your docker container as the ansible_host:

---
all:
  hosts:
    build_server
  vars:
    ansible_connection: docker
    ansible_host: my_app_build_server
    build_dir: "/app/build"

We’ll need a vault secret for encrypting our secrets. Run the following command:

openssl rand -base64 280 > .ansible/.vault_pass.txt

Remember to add .vault_pass.txt to .ansible/.gitignore!

Add the following to .ansible/ansible.cfg:

[defaults]
vault_password_file = ./.vault_pass.txt
roles_path = ./.downloaded_roles:./roles
pipelining = False
callback_whitelist = profile_tasks

Create the build playbook .ansible/apps/build/build.yml with the following:

---
- hosts: all
  gather_facts: no
  vars:
    local_build_dir: "/tmp/my_app_build"
    git_repo: "GIT_REPO"
    production_vars_file: "../production/host_vars/MY_HOST"

  pre_tasks:
    - command: date +"0.1.0-%Y.%-m.%-d.%-H.%-M"
      delegate_to: localhost
      register: build_version_cmd

    - set_fact:
        build_version: "{{ build_version_cmd.stdout }}"

    - name: Checkout the master branch from git repo
      delegate_to: localhost
      git:
        repo: "{{ git_repo }}"
        dest: "{{ local_build_dir }}"
        version: master
        force: yes
      when: ansible_connection == "docker"

  roles:
    - role: docker_setup/0.0.1
      vars:
        container_name: "{{ ansible_host }}"
        image_name: YOUR_ORG/phoenix-build-elixir-ubuntu18:1.9.0
      when: ansible_connection == "docker"

    - role: build_app/0.0.1
      vars:
        mix_env: prod
        app_name: my_app

Replace GIT_REPO with your git repo url, MY_HOST to the domain name of your production website, and YOUR_ORG with your docker hub organization name (or dreamconception if you skipped creating the docker image).

Add the following scripts:

.ansible/roles/docker_setup/0.0.1/tasks/main.yml:

---
- name: Start docker container
  delegate_to: localhost
  docker_container:
    name: "{{ container_name }}"
    image: "{{ image_name }}"
    volumes:
      - "{{ local_build_dir }}:{{ build_dir }}"
    env:
      SSH_AUTH_SOCK: /ssh-agent
    auto_remove: yes
    detach: yes
    interactive: yes
    tty: yes

.ansible/roles/build_app/0.0.1/tasks/main.yml:

---
- name: Load production vars
  include_vars: "{{ production_vars_file }}"

- name: Create secrets
  template:
    src: prod.secret.exs.j2
    dest: "{{ build_dir }}/config/prod.secret.exs"
    mode: 0644

- name: Fetch mix dependencies
  command: bash -lc "mix deps.get" chdir="{{ build_dir }}"
  environment:
    MIX_ENV: "{{ mix_env }}"

- name: Fetch npm dependencies
  command: bash -lc "cd {{ build_dir }}/assets && npm install"
  environment:
    MIX_ENV: "{{ mix_env }}"

- name: Build assets
  command: bash -lc "cd {{ build_dir }}/assets && npm run deploy"
  environment:
    MIX_ENV: "{{ mix_env }}"

- name: Digest assets
  command: bash -lc "mix phx.digest" chdir="{{ build_dir }}"
  environment:
    MIX_ENV: "{{ mix_env }}"

- name: Remove previous build
  file:
    name: "{{ build_dir }}/_build/{{ mix_env }}/rel/{{ app_name }}"
    state: absent

- name: "Releasing {{ build_version }}"
  command: bash -lc "mix release" chdir="{{ build_dir }}"
  environment:
    MIX_ENV: "{{ mix_env }}"
    BUILD_VERSION: "{{ build_version }}"

- name: Adding BUILD_VERSION file with "{{ build_version }}"
  copy:
    content: "{{ build_version }}"
    dest: "{{ build_dir }}/_build/{{ mix_env }}/rel/{{ app_name }}/BUILD_VERSION"

- name: Get GIT version
  command: git rev-parse HEAD
  args:
    chdir: "{{ build_dir }}"
  register: git_result

- name: Adding COMMIT_HASH file with "{{ git_result.stdout }}"
  copy:
    content: "{{ git_result.stdout }}"
    dest: "{{ build_dir }}/_build/{{ mix_env }}/rel/{{ app_name }}/COMMIT_HASH"

Add the following file (you can also copy your current prod.secret.exs and replace the secrets with literals yourself):

.ansible/roles/build_app/0.0.1/templates/prod.secret.exs.j2:

use Mix.Config

# In this file, we keep production configuration that
# you'll likely want to automate and keep away from
# your version control system.
#
# You should document the content of this
# file or create a script for recreating it, since it's
# kept out of version control and might be hard to recover
# or recreate for your teammates (or yourself later on).
config :my_app, MyAppWeb.Endpoint,
  secret_key_base: "{{ secret_key_base }}"

# Configure your database
config :my_app, MyApp.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "{{ database_username }}",
  password: "{{ database_password }}",
  database: "my_app",
  pool_size: 15

Now we’ll add the secrets.

Create .ansible/apps/production/host_vars/MY_HOST, using the domain of your production website instead of MY_HOST as filename. Add the encrypted secrets one by one by running:

ansible-vault encrypt_string --vault-password-file ".ansible/.vault_pass.txt" --stdin-name "NAME"

Replace NAME in the above command with the variable name, e.g. secret_key_base. You should hit ctrl-d twice (not enter) to encrypt the secret. Paste them into your MY_HOST file so it looks similar to this:

---
database_username: "my_app"
database_name: "my_app"
secret_key_base: !vault |
  $ANSIBLE_VAULT;1.1;AES256
  33306165616163393530303239656263393439663134613565663930613339346664626538373062
  6438653532646332366339353839616162353765353266380a623432306235346364393433613038
  64333536643738366361666535666136346334303335636166326666643939616335653131396134
  3565306266613561320a306234343836313339663331626264333336633737633837663735663732
  34373564616163616238663336313234393638633230383831363564633162353861616261366632
  38376331663565633562366566303264616333383138623665383362326363373534383732396538
  65313865653936653764663330383334323633313035363966636431316263343733306237663564
  36656135333438656438
database_password: !vault |
  $ANSIBLE_VAULT;1.1;AES256
  66333166303832313166666632373539316139383232396266613130366265383165653431623661
  3136646563646131323237353037636666303433336538350a643334623532623539323461326134
  65623231633335356565393165353761366633646566633435613961346635316131356233373530
  3063333730636337350a653937376363343561356166613538626439616137653830313065386330
  3534

That’s it!

You’ll be able to do a local build by running the following in the .ansible subdirectory:

ansible-playbook -i apps/build/inventory apps/build/build.yml -vvv

To make sure that Phoenix can start in your build, uncomment this line in config/prod.exs:

config :phoenix, :serve_endpoints, true

CI builds

We’ll use CircleCI in this tutorial, but you can replace it with whatever CI you prefer. Just keep in mind that I’ll let CircleCI test and build inside the docker container we created earlier.

Copy the content from .ansible/.vault_pass.txt and use it for the VAULT_PASS environment value in your CI. With CircleCI I had to base64 encode the string first, as they don’t accept multiline strings. Hence why we decode the environment variable in the following CircleCI config.

Add the following to .circleci/config.yml:

# Elixir CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-elixir/ for more details
version: 2
jobs:
  build:
    docker:
      - image: YOUR_ORG/phoenix-build-elixir-ubuntu18:1.9.0
      - image: circleci/postgres:11

    environment:
      MIX_ENV: test

    working_directory: ~/repo
    steps:
      - checkout

      - restore_cache:
          keys:
            - deps-build

      # Test
      - run: mix deps.get
      - run: mix ecto.create
      - run: mix ecto.migrate
      - run: mix test

      # Build
      - run: cd ~/repo && git reset --hard && git clean -dfx
      - run: echo "$VAULT_PASS" | base64 -d > ~/repo/.ansible/.vault_pass.txt
      - run: cd .ansible && ansible-playbook -i apps/build/inventory-ci apps/build/build.yml -vvv

      - save_cache:
          key: deps-build
          paths:
            - ~/repo/_build
            - ~/repo/deps

Replace YOUR_ORG with your organization Docker Hub, or you can also just use dreamconception/phoenix-build-elixir-ubuntu18:1.9.0 as the image. If you use MySQL instead of Postgres, you should replace circleci/postgres:11 with circleci/mysql:8.

Create .ansible/apps/build/inventory-ci:

---
all:
  hosts:
    build_server
  vars:
    ansible_connection: local
    ansible_host: 127.0.0.1
    build_dir: ~/repo

Now every push will be automatically build on our CI server!

Deployment

Now that we have a full build setup ready, it’s time for deployment!

First, we’ll need to add a migration task. Add lib/release.ex with:

defmodule MyApp.Release do
  @app :my_app

  def migrate do
    for repo <- repos() do
      {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
    end
  end

  def rollback(repo, version) do
    {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
  end

  defp repos do
    Application.load(@app)
    Application.fetch_env!(@app, :ecto_repos)
  end
end

I’ll not dwell on the server setup, but I expect you have the following set up on your production server:

• A deploy user
• SSH pub key on deploy accessible by you
• SSH pub key on deploy accessible by CI
• /u/apps/my_app/releases directory
• The deploy user has a PORT environment variable set, or you’ve set the port specifically in config/prod.exs

We’ll create a systemd service file on our server so we can easily restart the app after deploy. Add the following to /etc/systemd/system/my_app.service on your server (replace postgres.service with mysql.service if necessary):

[Unit]
Description=Server for my_app
Wants=postgres.service
After=postgres.service
After=syslog.target
After=network.target

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/u/apps/my_app/current
ExecStart=/u/apps/my_app/current/bin/my_app start
KillMode=process
Restart=on-failure
SuccessExitStatus=143
TimeoutSec=10
RestartSec=5
SyslogIdentifier=my_app

[Install]
WantedBy=multi-user.target

Update .ansible/apps/production/host_vars/MY_HOST, and append the following:

releases_dir: "/u/apps/my_app/releases"
current_dir: "/u/apps/my_app/current"
releases_to_keep: 10

Create .ansible/apps/production/inventory, and replace MY_HOST with the domain name for your production website:

---
all:
  hosts:
    MY_HOST
  vars:
    local_build_dir: /tmp/my_app
    ansible_user: deploy

Add .ansible/apps/production/deploy.yml:

---
- hosts: all
  gather_facts: no

  vars:
    mix_env: prod
    local_release_dir: "{{ local_build_dir }}/_build/{{ mix_env }}/rel/my_app"
    build_version: "{{ lookup('file', local_build_dir + '/_build/{{ mix_env }}/rel/my_app/BUILD_VERSION') }}"
    commit_hash: "{{ lookup('file', local_build_dir + '/_build/{{ mix_env }}/rel/my_app/COMMIT_HASH') }}"

  pre_tasks:
    - name: Get git version
      delegate_to: localhost
      become: false
      shell: "git rev-parse master {{ local_build_dir }}"
      register: git_version_result

    - name: Check for newest build
      delegate_to: localhost
      fail:
        msg: "Latest GIT commit of {{ git_version_result.stdout_lines[0] }} does not match build version of {{ commit_hash }}. Please build a new release."
      when: commit_hash != git_version_result.stdout_lines[0]

  roles:
    - role: deploy_app/0.0.1
      vars:
        app_name: my_app
        username: "{{ ansible_user }}"

As you can see, we use the COMMIT_HASH file to ensure that the build version contains the latest git commit.

Add .ansible/roles/deploy_app/0.0.1/tasks/main.yml:

---
- name: Check for existing current directory
  stat:
    path: "{{ current_dir }}"
  register: current_dir_stat

- name: Check for existing release directory
  stat:
    path: "{{ releases_dir }}/{{ build_version }}"
  register: release_dir_stat

- name: Copy previous release (make faster release deploys)
  command: "cp -r -L {{ current_dir }}/ {{ releases_dir }}/{{ build_version }}"
  when: current_dir_stat.stat.exists and not release_dir_stat.stat.exists

- name: "Upload new {{ build_version }} release"
  synchronize:
    src: "{{ local_release_dir }}/"
    dest: "{{ releases_dir }}/{{ build_version }}"
    recursive: yes
    delete: yes

# Here you should symlink shared directory in case you have uploaded files, etc.

- name: Run migrations
  command: bash -lc "bin/{{ app_name }} eval MyApp.Release.migrate" chdir="{{ releases_dir }}/{{ build_version }}"

- name: Update current symlink
  file:
    dest: "{{ current_dir }}"
    src: "{{ releases_dir }}/{{ build_version }}"
    state: link
    force: yes
  notify:
    - "restart app"

- name: List all releases
  shell: "ls -t {{ releases_dir }} | tail -n +{{ releases_to_keep + 1 }}"
  register: ls_output

- name: Remove old releases
  file:
    name: "{{ releases_dir }}/{{ item }}"
    state: absent
  with_items: "{{ ls_output.stdout_lines }}"

Create .ansible/roles/deploy_app/0.0.1/handlers/main.yml:

---
- name: "restart app"
  raw: "sudo /bin/systemctl restart {{ app_name }}"

Note: The deploy user should be allowed to run the /bin/systmectl restart my_app without password.

Deploy from your machine

You’ll be able to do a deploy from your local environment now, by running the following in .ansible/:

ansible-playbook -i apps/production/inventory apps/production/deploy.yml -vvv

Deploy from CI

Now let’s automate the deploys.

Create .ansible/apps/production/inventory-ci, and replace MY_HOST with the domain name for your production website:

---
all:
  hosts:
    MY_HOST
  vars:
    local_build_dir: ~/repo
    ansible_ssh_extra_args: "-o StrictHostKeyChecking=no"
    ansible_user: deploy

Add the following deploy step to your .circleci/config.yml:

jobs:
  build:
    # ...

    steps:
      # ...

      # Install rsync for deploy
      - run: apt-get install rsync -y

      - deploy:
          name: "Deploy master to production"
          command: |
            if [ "${CIRCLE_BRANCH}" == "master" ]; then
              cd .ansible && ansible-playbook -i apps/production/inventory-ci production/deploy.yml -vvv;
            fi

CircleCI will deploy after successful test and build of any push on the master branch.

Congratulations, you got yourself an automated build and deploy setup for your Phoenix app. Happy coding!

The Author

Dan Schultzer is an active experienced entrepreneur, starting the Being Proactive groups, Dream Conception organization, among other things. You can find him at twitter

Like this post? More from Dan Schultzer

• Rails 4: How to set parent model on both left and right side in a has_many :trough relationship, models being created with nested attributes
• Javascript: Prevent blocking during load of inline background images
• Asynchronous (non-blocking) lazy loading of external javascript

Comments? We would love to hear from you, write us at @dreamconception.