Luis Rascão

Backend Developer @ Miniclip

View on GitHub
18 August 2016

Automatic release upgrades in Erlang


Automate your Erlang release upgrades using rebar3 all the way to the gen_server state conversion.


In this article i’ll demonstrate the use of a rebar3 plugin for handling .appup files that contain the instructions necessary to upgrade from one release to another. It supports the following features:

Application upgrade generation

This is achieved by first generating two releases, the one that is live right now and the one that we want to upgrade to, we then invoke the plugin and have it generate a special .appup file that OTP knows how to process in order to generate a release upgrade file (relup). This file contains low level Erlang VM instructions that will transition the Erlang application from one version to another without any downtime.

Using rebar3 the most simple flow necessary to generate the .appup files is:

Using the example relapp1 release (this app is also used for the plugin’s regression tests):

    $ git clone relapp1
    # checkout the original version that it's running
    # somewhere live
    $ git checkout 1.0.11
    $ rebar3 release
    # checkout the version with the hotfix we need to deploy
    $ git checkout 1.0.12
    $ rebar3 release
    # generate the .appup that contains the necessary
    # instructions for the upgrade from 1.0.11 to 1.0.12
    # and the downgrade from 1.0.12 to 1.0.11
    $ rebar3 appup generate
    $ ===> Generated appup ("1.0.11" <-> "1.0.12") for relapp in
    $ "_build/default/lib/relapp/ebin/relapp.appup"
    # generate the relup which essentially takes the appup
    # instructions and translates them to lower level
    # VM instructions
    $ rebar3 relup
    # finally generate a tarball with the 1.0.12 upgrade that
    # is ready for deployment
    $ rebar3 tar

The application upgrade format

In the previous example the appup that is generated looks like this:

%% appup generated for relapp by rebar3_appup_plugin ("2016/08/18 16:00:18")

The appup doc tells us that the acceptable format for an .appup is:

  [{UpFromVsn, UpgradeInstructions}, ...],
  [{DownToVsn, DowngradeInstructions}, ...]}.

So in this case we see that to upgrade from version 1.0.11 to 1.0.12 we need to update the relapp_srv2 module which is actually a gen_server process. The same update needs to occur to downgrade from 1.0.12 to 1.0.11.

Deploying the upgrade

Deploying the new version is also straightforward, it involves (after ssh’ing into the live machine):

In the case of the example release upgrading from 1.0.11 to 1.0.12:

    $ cd <live_release_dir>
    $ mkdir releases/1.0.12
    $ cp relapp-1.0.12.tar.gz releases/1.0.12/relapp.tar.gz
    $ bin/relapp upgrade 1.0.12

Soft vs Brutal purge

In the update and load_module appup directives there is a parameter that specifies the type of purge that the Erlang VM will apply when loading code. There are two phases when a purge is aplplied: pre and post purge and two types of purge: soft and brutal, according to the doc:

  Defaults to brutal_purge. It controls what action to take with
  processes executing old code before loading the new module
  If the value is brutal_purge, the processes are killed.
  If the value is soft_purge, release_handler:install_release/1
  returns {error,{old_processes,Mod}}.
  Defaults to brutal_purge. It controls what action to take with
  processes that are executing old code when the new module
  version has been loaded. If the value is brutal_purge, the code
  is purged when the release is made permanent and the processes
  are killed. If the value is soft_purge, the release handler
  purges the old code when no remaining processes execute the

A good example of this behaviour in practice is in a test app commit, in this example relapp_m1 (a simple library module) sends an anonymous function to the relapp_srv2 gen_server that will store in it’s state. If you look into it you will see something like this:


Now what happens if we load a new version of relapp_m1? This is where the purge option comes into play, a brutal_purge will kill the gen server process, that might not be the ideal method since the supervisor’s restart intensity could be reached, perhaps in this case a soft_purge should be chosen instead and then manually restart the gen server or supervisor tree. As a general rule it is best not to send functions across processes if this is to be avoided.

Using an .appup.src

You can generate the .appup file every time you pack a release upgrade with the rebar3 appup generate call. However when there is the need to add custom data or instructions to the .appup it’s useful to have it in source control alongside your .app.src file. The .appup.src file can contain Erlang code and it’s result should be a valid appup Erlang term.

The plugin will search for any files ending in .appup.src, evaluate them and The plugin will look for any files ending in .appup.src, evaluate them and have their results written to an .appup file that will then be used to generate the relup. The last term for the expression in this file should a properly formatted .appup term. Here is a simple example of this in practice in the relapp test release:

Version = "1.0.17",
{ok, V} = relapp_m1:test("1.0.16"),
UpFrom = DownTo = V,

Note that we are able to invoke methods in our codebase (ie. the relapp_m1 module).

State conversion through code injection

Use of records to contain a gen_server’s state is widespread in Erlang. Due to the fact that records are in fact just tuples a problem arises when performing relups between release versions that involve state changes. There must be a way of converting between to two records. Several solutions are available to overcome this:

Manual conversion

The most simple way is just taking the old record that arrives through code_change and by making several erlang:setelement/2 calls, convert the old tuple to the new one. You are in fact just converting one tuple into another. This is a bit of a hackish solution, but it works.

Record conversion using exprecs

Ulf Wiger’s excellent project parse_trans contains a parse_transform called exprecs that generates methods to manipulate your exported records, one interesting method is '#convert' which (as the name implies) converts one record to another. It needs the following to work:

Here is a simple example snippet with the relevant entries:

%% this is the current gen_server version, the one we want to
%% upgrade from is 1.0.0

%% ask exprecs to apply the parse transform to this file
-compile({parse_transform, exprecs}).

%% below is the previous state record definition with a new
%% name containing it's version, it only has one field.
%% Do note that in the 1.0.0 version the record is still
%% named 'state', the renaming to 'state__1.0.0' is just
%% so that exprecs is able to distinguish between versions
-record('state__1.0.0', {
    id = 0 :: non_neg_integer(),

%% this is our new state record definition
%% as you can see, we've added a new name field to it
-record(state, {
    id = 0 :: non_neg_integer(),
    name = <<>> :: binary()

%% ask the exprecs parse transform to generate it's methods
%% for our state record on compilation

%% pattern match on a requested code_change from module
%% version 1.0.0 to 1.0.1
code_change("1.0.0", OldState, Extra) ->
    lager:debug("code change from 1.0.0 version has been "
    lager:debug("   extra: ~p", [Extra]),
    {NewState, _} = m:'#convert-'("1.0.0", OldState),
    {ok, NewState};

exprecs will look at the OldState tuple to determine the record name, append the version suffix and convert from the old record structure to the new one while maintaining old values.

Plugin code injection

This plugin offers the choice of injecting the necessary code in gen_server:code_change/3 that takes care of record migration from the old version to the new. The plugin will be called into action when the new version tarball is being generated (hence the tar provider hook in rebar.config), it will:

Conversion method

The Erlang code that performs the conversion can be found in priv/templates/convert.tpl, based on a proplist of named record fields for the old and new versions it uses erlang:setelement/3 and erlang:element/2 to populate a new version record, new record fields are filled out with record defaults, updated fields are copied, renaming fields is not supported.

gen_server:code_change hook injection

In priv/templates/convert_call.tpl you can also see the call that is injected into the beginning of the code_change method, it simply is a call to the conversion method making sure that the original variable names are kept and setting them to their new values.

Module dependencies

OTP allows you to define a list of module dependencies that each upgraded module is dependent on, as described in the upgrade instructions doc:

    Defaults to [] and defines other modules that Mod is
    dependent on. In the relup file, instructions for suspending
    processes using Mod come before instructions for suspending
    processes using modules in DepMods when upgrading, and
    conversely when downgrading. In case of circular dependencies,
    the order of the instructions in the appup file is kept.

The plugin generates these dependencies list for you by using the xref tool, for each generated module in the .appup the plugin performs an xref:analyze/2 that yields a list of modules that it makes calls to, this list is then intersected with the modules being upgraded and added to the relevant entries in the .appup.