This post covers running the OTP’s observer on a node running on a remote machine. It is a step-by-step of this gist.

Software Versions

$ date -u "+%Y-%m-%d %H:%M:%S +0000"
2016-09-16 14:17:43 +0000

# localhost
$ uname -vm
Darwin Kernel Version 15.6.0: Mon Aug 29 20:21:34 PDT 2016; root:xnu-3248.60.11~1/RELEASE_X86_64 x86_64
$ mix
Hex:    0.13.0
Elixir: 1.3.2
OTP:    19.0
* snip *

# remotehost
$ uname -vm
FreeBSD 12.0-CURRENT #0 r304324: Thu Aug 18 13:27:23 JST 2016  amd64
$ mix
Hex:    0.13.0
Elixir: 1.3.2
OTP:    19.0.7
* snip *


On the remote host, start a node with known name and cookie values. iex or erl will be used for this example.

sh remotehost

HOST=$(ifconfig ${IF} | grep 'inet ' | cut -d ' ' -f 2)
printf "NAME='${NAME}'\nCOOKIE='${COOKIE}'\n"
iex --name "${NAME}" --cookie "${COOKIE}"
# OR
erl -name "${NAME}" -setcookie "${COOKIE}"

From the localhost, ssh into the remote host and run epmd.

sh localhost

# use your own remote host
ssh "${REMOTE_HOST}" "epmd -names"

With the previously started node running, the output should look something like this.

sh localhost

epmd: up and running on port 4369 with data:
name wild_exhibisionist at port 48341

Note the ports that empd and the node you want to debug are using. Reconnect to the remote host with these ports forwarded.

sh localhost

# below NODE_PORT setting assumes there is only one node running
EPMD_PORT=$(ssh "${REMOTE_HOST}" "epmd -names" | grep epmd | sed 's/[^0-9]//g')
NODE_PORT=$(ssh "${REMOTE_HOST}" "epmd -names" | grep name | sed 's/[^0-9]//g')

Start a hidden node running the observer app.

sh localhost

# both interfaces probably need to be on the same network
HOST=$(ifconfig ${IF} | grep 'inet ' | cut -d ' ' -f 2)
# use the same cookie value
printf "NAME='${NAME}'\nCOOKIE='${COOKIE}'\n"
iex --name "${NAME}" --cookie "${COOKIE}" --hidden -e ":observer.start"
# OR
erl -name "${NAME}" -setcookie "${COOKIE}" -hidden -run observer

In observer, go to Nodes -> Connect Node and enter the name of the remote node.

If you are having trouble connecting, try to manually connect the nodes from the iex or erl prompt. If the nodes do not connect manually, solve that problem. After the nodes have been manually connected, go to Nodes -> Connect Node and enter the name of the remote node in observer.

iex localhost

Node.connect :"wild_exhibisionist@"

iex remotehost

Node.connect :"observer@"

erl localhost


erl remotehost


Use the following to get a remote shell.

sh localhost

iex --name "${NAME}" --cookie "${COOKIE}" --remsh "${REMOTE_NODE}"
# OR
erl -name "${NAME}" -setcookie "${COOKIE}" -remsh "${REMOTE_NODE}"

Note that :runtime_tools must be in the applications list to observe an app released with exrm.

mix.exs partial listing

  def application do
    [mod: {MyApp, []},
     applications: [:runtime_tools]] # add to this list