This post covers storing settings like database configuration in environment variables with distillery. A prior post covered this with the Elixir Release Manager (exrm).

Software Versions

$ date -u "+%Y-%m-%d %H:%M:%S +0000"
2016-09-18 06:31:36 +0000
$ uname -vm
FreeBSD 12.0-CURRENT #0 r304324: Thu Aug 18 13:27:23 JST 2016     root@mirage.sennue.com:/usr/obj/usr/src/sys/MIRAGE_KERNEL  amd64
$ mix hex.info
Hex:    0.13.0
Elixir: 1.3.2
OTP:    19.0.7
* snip *
$ mix phoenix.new -v
Phoenix v1.2.1
$ cat mix.lock | grep distillery | cut -d" " -f 3,6 | sed 's/[",]//g'
distillery: 0.9.9

Creating a Sample Project

Create a new Phoenix project.

mix phoenix.new phoenix_environment_settings --no-brunch
cd phoenix_environment_settings
mix ecto.create

Create a simple JSON memo endpoint.

mix phoenix.gen.json Memo memos title:string body:string

Revise web/router.ex. Uncomment the “/api” scope and add the “/memos” route.

web/router.ex.partial listing

  scope "/api", PhoenixEnvironmentSettings do
    pipe_through :api

    resources "/memos", MemoController, except: [:new, :edit]
  end

Run the migration.

mix ecto.migrate

Tests should pass.

mix test

Start the server.

mix phoenix.server

POST and GET a memo to make sure the server works. A prior post covers a shell script for conveniently interacting with this sample app.

# POST new
curl -H 'Content-Type: application/json' -X POST -d '{"memo": {"title": "New Title", "body": "This is the new memo body."}}' http://localhost:4000/api/memos
# GET id 1
curl -H 'Content-Type: application/json' http://localhost:4000/api/memos/1

Environment Variable Definitions

Define the following environment variables. Feel free to use different values and a static value for SECRET_KEY_BASE. Note that depending on how your database is configured, DB_PASSWORD may be unnecessary for connections from localhost.

export NODE_NAME=leaf_node
export COOKIE=thin_mints
export DB_USER=phoenix_environment_settings
export DB_PASSWORD=phoenix_environment_settings_password
export DB_NAME=phoenix_environment_settings_prod
export DB_HOST=localhost
export HOST=host.example.org
export PORT=7777
export SECRET_KEY_BASE=$(elixir -e ":crypto.strong_rand_bytes(48) |> Base.encode64 |> IO.puts")

Configuring the Application

Set server to true in config/prod.exs. Also add a version entry. Replace static configuration with dynamic configuration that will be pulled in from environment variables. None of this needs to be kept out version control, so it is safe to delete config/prod.secret.exs. Those settings have been merged in.

config/prod.exs

use Mix.Config

config :phoenix_environment_settings, PhoenixEnvironmentSettings.Endpoint,
  http: [port: {:system, "PORT"}],
  url: [host: "${HOST}", port: {:system, "PORT"}],
  cache_static_manifest: "priv/static/manifest.json",
  server: true,
  root: ".",
  version: Mix.Project.config[:version]

config :logger, level: :info

config :phoenix_environment_settings, PhoenixEnvironmentSettings.Endpoint,
  secret_key_base: "${SECRET_KEY_BASE}"

config :phoenix_environment_settings, PhoenixEnvironmentSettings.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "${DB_USER}",
  password: "${DB_PASSWORD}",
  database: "${DB_NAME}",
  hostname: "${DB_HOST}",
  pool_size: 20

Alternatively, keep hard coded values in config/prod.secret.exs when running local release builds and use a version with dynamic environment variable settings in a real production environment.

config/prod.secret.exs

use Mix.Config

config :phoenix_environment_settings, PhoenixEnvironmentSettings.Endpoint,
  secret_key_base: "${SECRET_KEY_BASE}"

config :phoenix_environment_settings, PhoenixEnvironmentSettings.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "${DB_USER}",
  password: "${DB_PASSWORD}",
  database: "${DB_NAME}",
  hostname: "${DB_HOST}",
  pool_size: 20

Optionally, add dynamic configuration with a default values to config/dev.exs. The environment variable replacement above is only suitable for releases. The interpreted solution below is only suitable for development with mix.

config/dev.exs partial listing

config :phoenix_environment_settings, PhoenixEnvironmentSettings.Endpoint,
  http: [port: System.get_env("PORT") || 4000],

# ... at the end of the file ...

# Configure your database
config :phoenix_environment_settings, PhoenixEnvironmentSettings.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: System.get_env("DB_USER") || "postgres",
  password: System.get_env("DB_PASSWORD") || "postgres",
  database: System.get_env("DB_NAME") || "phoenix_environment_settings_dev",
  hostname: System.get_env("DB_HOST") || "localhost",
  pool_size: 10

Generating a Release

Add distillery to mix.exs as a project dependency.

mix.exs partial listing

  defp deps do
    [{:phoenix, "~> 1.2.1"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_ecto, "~> 3.0"},
     {:postgrex, ">= 0.0.0"},
     {:phoenix_html, "~> 2.6"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.11"},
     {:distillery, "~> 0.9"}, # this line is new
     {:cowboy, "~> 1.0"}]
  end

Install and initialize distillery. This will create the rel/ directory.

mix deps.get
mix deps.compile
mix release.init

Environment variables will be used as knobs to configure the app. Note that the REPLACE_OS_VARS=true environment variable needs to be defined to use environment variables for dynamic configuration.

The rel/vm.args file is primarily used to configure the erlang VM. It can also be used to define application configuration parameters. Application configuration parameters defined in this file can be passed into the program as atoms or integers. Note that the location of this file can be configured with the RELEASE_CONFIG_DIR environment variable. Add the following to rel/vm.args.

rel/vm.args

## Name of the node
-name ${NODE_NAME}

## Cookie for distributed erlang
-setcookie ${COOKIE}

## App Settings
-phoenix_environment_settings port ${PORT}

Alternatively, rel/sys.config can be used to pass in application configuration parameters. This file is written in Erlang.

rel/sys.config

[
  {phoenix_environment_settings, [
    {port, "${PORT}"}
  ]}
].

The Elixir config/config.exs file is probably a better place to define non-VM settings for an Elixir application. The distillery documentation seems to indicate that a release can either use sys.config or config.exs but not both. The exact config.exs settings for this project were covered in the Configuring the Application section above.

Adding Release Tasks

After deploying a release with distillery, mix is no longer available. Define a module for release tasks with a mix ecto.migrate equivalent. Note that this is an Erlang module written in Elixir.

lib/release_tasks.ex

defmodule :release_tasks do
  def migrate do
    {:ok, _} = Application.ensure_all_started(:phoenix_environment_settings)
    path = Application.app_dir(:phoenix_environment_settings, "priv/repo/migrations")
    Ecto.Migrator.run(PhoenixEnvironmentSettings.Repo, path, :up, all: true)
    :init.stop()
  end
end

After generating a release, this task can be run as follows.

REPLACE_OS_VARS=true rel/phoenix_environment_settings/bin/phoenix_environment_settings command release_tasks migrate

The mix ecto.create and mix ecto.drop tasks are run less frequently so it may just makes sense to just manually create or drop the database and user. PostgreSQL commands follow.

# PostgreSQL `mix ecto.create` equivalent
psql -c "CREATE USER ${DB_USER} WITH PASSWORD '${DB_PASSWORD}';"
createdb "${DB_NAME}"
psql -c "GRANT ALL PRIVILEGES ON DATABASE ${DB_NAME} to ${DB_USER};"

# PostgreSQL `mix ecto.drop` equivalent
dropdb "${DB_NAME}"
dropuser "${DB_USER}"

# PostgresSQL interactive terminal
PGPASSWORD="${DB_PASSWORD}" psql -U "${DB_USER}" "${DB_NAME}"

The MySQL commands look like this.

# MySQL `mix ecto.create` equivalent
mysql -e "CREATE USER '${DB_USER}'@'localhost' IDENTIFIED BY '${DB_PASSWORD}';"
mysql -e "CREATE DATABASE ${DB_NAME};"
mysql -e "GRANT ALL PRIVILEGES ON ${DB_NAME}.* TO '${DB_USER}'@'localhost' IDENTIFIED BY '${DB_PASSWORD}';"
mysql -e "FLUSH PRIVILEGES;"

# MySQL `mix ecto.drop` equivalent
mysql -e "DROP DATABASE ${DB_NAME};"
mysql -e "DROP USER '${DB_USER}'@'localhost';"

# MySQL interactive terminal
mysql -u"${DB_USER}" -p"${DB_PASSWORD}" "${DB_NAME}"

Note that the password security in the above commands is less than ideal. Also, an existing superuser and password may need to be explicily specified when creating the new user and database.

Adding Custom Commands for Release Tasks

Distillery supports custom commands. To wrap the above release tasks in custom commands, first create the rel/commands directory.

mkdir rel/commands

Next add a command for ecto.migrate.

rel/commands/ecto_migrate

#!/usr/bin/env sh

# mix ecto.migrate` equivalent
"${SCRIPT}" command release_tasks migrate

The PostgreSQL versions of the ecto.create and ecto.drop commands follow.

rel/commands/ecto_create

#!/usr/bin/env sh

# PostgreSQL `mix ecto.create` equivalent
psql -c "CREATE USER ${DB_USER} WITH PASSWORD '${DB_PASSWORD}';" &&
createdb "${DB_NAME}" &&
psql -c "GRANT ALL PRIVILEGES ON DATABASE ${DB_NAME} to ${DB_USER};" &&
echo "The database '${DB_NAME}' and role '${DB_USER}' have been created."

rel/commands/ecto_drop

#!/usr/bin/env sh

# PostgreSQL `mix ecto.drop` equivalent
dropdb "${DB_NAME}" &&
dropuser "${DB_USER}" &&
echo "The database '${DB_NAME}' and role '${DB_USER}' have been dropped."

Alternatively, the MySQL versions of ecto.create and ecto.drop look like this.

rel/commands/ecto_create

#!/usr/bin/env sh

# MySQL `mix ecto.create` equivalent
mysql -e "CREATE USER '${DB_USER}'@'localhost' IDENTIFIED BY '${DB_PASSWORD}';" &&
mysql -e "CREATE DATABASE ${DB_NAME};" &&
mysql -e "GRANT ALL PRIVILEGES ON ${DB_NAME}.* TO '${DB_USER}'@'localhost' IDENTIFIED BY '${DB_PASSWORD}';" &&
mysql -e "FLUSH PRIVILEGES;" &&
echo "The database '${DB_NAME}' and role '${DB_USER}' have been created."

rel/commands/ecto_drop

#!/usr/bin/env sh

# MySQL `mix ecto.drop` equivalent
mysql -e "DROP DATABASE ${DB_NAME};" &&
mysql -e "DROP USER '${DB_USER}'@'localhost';" &&
echo "The database '${DB_NAME}' and role '${DB_USER}' have been dropped."

Finally, modify rel/config.exs so that the custom commands are present. The following version is a rewrite.

rel/config.exs

use Mix.Releases.Config,
    default_release: :default,
    default_environment: :prod

environment :prod do
  set include_erts: true
  set include_src: false
  set commands: [
    "ecto.migrate": "rel/commands/ecto_migrate",
    "ecto.create": "rel/commands/ecto_create",
    "ecto.drop": "rel/commands/ecto_drop"
  ]
end

release :phoenix_environment_settings do
  set version: current_version(:phoenix_environment_settings)
end

Running the Release

Build the release with the configuration files.

# rm -rf rel/phoenix_environment_settings/ # optional, no stale release files
MIX_ENV=prod mix compile
# brunch build --production # if using brunch
MIX_ENV=prod mix phoenix.digest
MIX_ENV=prod mix release --env=prod

Run the ecto.create and ecto.migrate custom commands defined in the above sections.

REPLACE_OS_VARS=true rel/phoenix_environment_settings/bin/phoenix_environment_settings ecto.create
REPLACE_OS_VARS=true rel/phoenix_environment_settings/bin/phoenix_environment_settings ecto.migrate

Start the release.

REPLACE_OS_VARS=true rel/phoenix_environment_settings/bin/phoenix_environment_settings start

Make sure the server responds.

curl -H 'Content-Type: application/json' -X POST -d '{"memo": {"title": "Memo A", "body": "Alpha memo body."}}' "http://localhost:${PORT}/api/memos"
curl -H 'Content-Type: application/json' -X POST -d '{"memo": {"title": "Memo B", "body": "Beta memo body."}}' "http://localhost:${PORT}/api/memos"
curl "http://localhost:${PORT}/api/memos"

Stop the release.

REPLACE_OS_VARS=true rel/phoenix_environment_settings/bin/phoenix_environment_settings start

Custom Application Settings

Add this configuration gist to lib/config.ex. This configuration wrapper allows the same convenient {:system, "VARIABLE", "default"} convention to be used with both mix and releases. Note that this will not help configure things like PhoenixEnvironmentSettings.Repo because they were not written to get settings via this module. The linked to version has the typespecs and documentation.

lib/config.ex

defmodule Config do
  def get(app, key, default \\ nil) when is_atom(app) and is_atom(key) do
    case Application.get_env(app, key) do
      {:system, env_var} ->
        case System.get_env(env_var) do
          nil -> default
          val -> val
        end
      {:system, env_var, preconfigured_default} ->
        case System.get_env(env_var) do
          nil -> preconfigured_default
          val -> val
        end
      nil ->
        default
      val ->
        val
    end
  end

  def get_integer(app, key, default \\ nil) do
    case get(app, key, nil) do
      nil -> default
      n when is_integer(n) -> n
      n ->
        case Integer.parse(n) do
          {i, _} -> i
          :error -> default
        end
    end
  end
end

Add these lines to config/config.exs.

config/config.exs partial listing

config :phoenix_environment_settings,
  ecto_repos: [PhoenixEnvironmentSettings.Repo],
  welcome_message: {:system, "WELCOME_MESSAGE", "Hello, world!"},
  magic_number: {:system, "MAGIC_NUMBER", 42}

Access the settings in iex -S mix like this. You should get the default values.

iex -S mix

Config.get :phoenix_environment_settings, :welcome_message
Config.get_integer :phoenix_environment_settings, :magic_number

Define the environment variables and run the commands in iex again. You should get the environment variable values this time.

export WELCOME_MESSAGE="Welcome. Try, but there is no escape."
export MAGIC_NUMBER=-1

This is useful for defining environment variable knobs to control run time behavior. It is not a solution for problems that rely on compile time behavior, like using environment variables to specify dynamic routes.

Other Considerations

Note that when using REPLACE_OS_VARS=true, the environment variables in rel/sys.conf or config/config.exs will always be replaced with strings. The following almost certainly does not work as expected.

export DB_POOL_SIZE=20

config/prod.exs or config/prod.secret.exs partial listing

  pool_size: "${DB_POOL_SIZE}"

The following will work in development, but not production.

config/dev.exs partial listing

  pool_size: (System.get_env("DB_POOL_SIZE") || "10") |> String.to_integer

If integers or atoms need to be passed in directly, use vm.args. The author could not figure out how to pass DB_POOL_SIZE to Repo via vm.args.

Next Steps

Consider looking into conform, a library for working with init-style configuration. Also consider looking into edeliver for deployment.

References: