Using Lager with Cowboy

Lager is a popular logging framework for Erlang applications. To get it working with Cowboy (assuming you’re using erlang.mk), you need to add it as a dependency to your Makefile:

DEPS = cowboy lager

and then add the parse transform to the options for erlc (as shown here):

include erlang.mk
ERLC_OPTS += +'{parse_transform, lager_transform}'

(alternatively, you could add a compile header to each file that needs it). You also need to add lager to your app.src file:

    {applications, [
        kernel,
        stdlib,
        cowboy,
        lager
    ]},
]}.

so that it is started as part of your release. You can find an example repo here.

Building a Tic Tac Toe game with Cowboy & websockets (Part 2)

In Part 1 we dealt with session management, now we’ll take a look at starting a new game.

First, we send a message from the client:

newGameBtn.onclick = function() {
    var msg = JSON.stringify({type: 'new_game', sessionId: sessionId});
    send(msg);
    newGameBtn.disabled = true;
    clearBoard();
    updateStatus('Waiting to join game...');
};

and handle that on the other side of the websocket:

websocket_handle({text, Json}, Req, State) ->
    Msg = jiffy:decode(Json, [return_maps]),
    Resp = validate_session(Msg, fun() ->
        Type = maps:get(<<"type">>, Msg),
        handle_message(Type, Msg)
    end),
    {reply, make_frame(Resp), Req, State};

validate_session(Msg, Fun) ->
    SessionId = maps:get(<<"sessionId">>, Msg),
    case gen_server:call(t3_session_manager, {validate_session, SessionId}) of
        ok -> Fun();
        invalid_session -> #{type => <<"error">>, msg=> <<"invalid_session">>}
    end.

handle_message(<<"new_game">>, Msg) ->
    SessionId = maps:get(<<"sessionId">>, Msg),
    start_new_game(SessionId);

start_new_game(_SessionId) ->
    Res = try
         gen_server:call(t3_match_maker, {find_game}, 30000)
    catch
        exit:{timeout,_} -> timeout
    end,
    case Res of
        {ok, GameId} -> #{type => <<"new_game">>, id => GameId};
        timeout -> #{type => <<"no_game_available">>}
    end.

We decode the message, and validate the session; then we need to find a game, a task we delegate to the “match maker”, another gen_server. It’s possible a game won’t be found in time, so we need to handle timeouts too.

The match maker is pretty simple, like the session manager, but might be more interesting in a real app:

handle_call({find_game}, From, State) ->
    case find_game(From, State) of
        {ok, GameId, NewState} -> {reply, {ok, GameId}, NewState};
        {wait, NewState} -> {noreply, NewState}
    end;

find_game(From, #state{waiting=[]}) ->
    {wait, #state{waiting=[From]}};

find_game({P2,_}, #state{waiting=[From|Rest]}) ->
    GameId = uuid:uuid_to_string(uuid:get_v4(), binary_standard),
    {P1,_} = From,
    {ok, _Pid} = supervisor:start_child(t3_game_sup, [{P1, P2, GameId}]),
    gen_server:reply(From, {ok, GameId}),
    {ok, GameId, #state{waiting=Rest}}.

If someone is waiting, then we start a new game; otherwise, the pid of the websocket gets pushed onto the queue (this should probably be a more stable reference, like a user id, and we would look up the pid. We also don’t handle the case that a queued socket is closed, or times out waiting).

We need to add two more items to our supervision tree:

init([]) ->
    Procs = [
        {t3_session_manager, {t3_session_manager, start_link, []}, permanent, 5000, worker, [t3_session_manager]},
        {t3_match_maker, {t3_match_maker, start_link, []}, permanent, 5000, worker, [t3_match_maker]},
        {t3_game_sup, {t3_game_sup, start_link, []}, permanent, 5000, supervisor, [t3_game_sup]}
    ],
    {ok, {{one_for_one, 1, 5}, Procs}}.

The game supervisor is also very simple:

init([]) ->
    Procs = [{t3_game_fsm, {t3_game_fsm, start_link, []}, transient, 5000, worker, [t3_game_fsm]}],
    {ok, {{simple_one_for_one, 5, 10}, Procs}}.

The important thing to note is the “simple one-for-one” restart strategy, where children are added dynamically, on demand.

We’ll look at the game FSM in more detail in Part 3, but when started it notifies both players:

init(Args) ->
    io:format("New game started: ~p~n", [Args]),
    [{P1, P2, GameId}] = Args,
    true = gproc:reg({n, l, GameId}),
    State = #state{p1 = P1, p2 = P2},
    P1 ! {your_turn, State#state.board},
    P2 ! {wait, State#state.board},
    {ok, p1_turn, State}.

Finally, we need to handle some new messages on the client:

    } else if (msg.type === 'new_game') {
        gameId = msg.id;
        updateStatus('New game!');
    } else if (msg.type === 'your_turn') {
        updateStatus('Your turn!');
        updateBoard(msg.data);
        enableBoard();
    } else if (msg.type === 'wait') {
        updateBoard(msg.data);
        updateStatus('Waiting for other player...');
    }

Building a Tic Tac Toe game with Cowboy & websockets (Part 1)

Tic-tac-toe (or noughts and crosses) is a good example project, as the game itself is pretty simple and won’t become a distraction. I’m going to try and build a version using Erlang, and more specifically Cowboy: an http server/framework.

I’m going to assume you know the basics of setting up a new project, and skip straight to the more interesting bits. The goal is to create a multi-player, websockets & html based version of tic-tac-toe. As it’s turn-based, we avoid a lot of the really thorny problems of “real-time” multi-player games.

My first step was to create a session as soon as the socket is opened:

socket.onopen = function () {
    socket.send('new_session')
}

The backend then needs to handle this frame. I decided to add a session manager as a gen_server:

-record(state, {sessions=#{}}).

handle_call(new_session, _From, State) ->
    SessionId = uuid:uuid_to_string(uuid:get_v4(), binary_standard),
    NewState = update_session(SessionId, State),
    {reply, {ok, SessionId}, NewState};

update_session(SessionId, State) ->
    Expiry = half_an_hour_from_now(),
    prune_expired_sessions(State#state{sessions=maps:put(SessionId, Expiry, State#state.sessions)}).

half_an_hour_from_now() ->
    Now = calendar:universal_time(),
    calendar:gregorian_seconds_to_datetime(calendar:datetime_to_gregorian_seconds(Now) + (30 * 60)).

prune_expired_sessions(State) ->
    SessionIds = maps:keys(State#state.sessions),
    {_, ExpiredSessions} = lists:partition(fun(S) -> session_valid(S, State#state.sessions) end, SessionIds),
    State#state{sessions=maps:without(ExpiredSessions, State#state.sessions)}.

session_valid(SessionId, Sessions) ->
    case maps:find(SessionId, Sessions) of
        error -> false;
        {ok, Expires} ->
            Expires > calendar:universal_time()
    end.

In a more realistic app, you would probably want to perform some authentication here, and use a datastore of some kind; but we’ll just create a new session and return the id. We also need to add the new server to the supervision tree:

init([]) ->
    Procs = [ 
        {t3_session_manager, {t3_session_manager, start_link, []}, permanent, 5000, worker, [t3_session_manager]}
    ],  
    {ok, {{one_for_one, 1, 5}, Procs}}.

We can then call the server from the ws handler:

websocket_handle({text, <<"new_session">>}, Req, State) ->
    Resp = start_new_session(),
    {reply, make_frame(Resp), Req, State};

start_new_session() ->
    {ok, SessionId} = gen_server:call(t3_session_manager, new_session),
    #{type => <<"new_session">>, sessionId => SessionId}.

make_frame(Msg) ->
    Json = jiffy:encode(Msg),
    {text, Json}.

We can call the server using it’s module name, rather than a pid, because it registered itself when starting:

start_link() ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

(Watch out though, this effectively makes it a singleton, which could be a choke-point in a real app). We need to handle the response on the client:

socket.onmessage = function(ev) {
    console.log('Received data: ' + ev.data);
    var msg = JSON.parse(ev.data);

    if (msg.type === 'new_session') {
        sessionId = msg.sessionId;
        newGameBtn.disabled = false;
    }
}

And we’ll probably also need to be able to validate a session at some point:

handle_call({validate_session, SessionId}, _From, State) ->
    case session_valid(SessionId, State#state.sessions) of
        false ->
            {reply, {error, invalid_session}, State};
        true ->
            %% sliding expiry window
            NewState = update_session(SessionId, State),
            {reply, ok, NewState}
    end;

You can find the full source code here. In Part 2, we’ll look at starting a new game.

Using Bootstrap alerts with Express.js 4

It’s often handy to be able to display a notification to users, noting a successful operation for example, or showing errors.

With Express this is normally done using a “flash message”. You’ll need the connect-flash middleware, as it was unbundled in Express 3.

var express = require('express'),
    flash = require('connect-flash'),
    app = express();

app.use(flash());
app.use(function(req, res, next){
    res.locals.success = req.flash('success');
    res.locals.errors = req.flash('error');
    next();
});

The 2nd piece of middleware ensures that flash messages will be available to the template as locals.

router.get('/account/name', function (req, res) {
        var data = {
            firstName: req.user.firstName,
            lastName: req.user.lastName
        };
        res.render('settings_name', data);
    });

    router.post('/account/name', function (req, res) {
        req.user.updateName(req.user.id, req.body.firstName, req.body.lastName, function(err) {
            if (err) {
                req.flash('error', 'Could not update your name, please contact our support team');
            } else {
                req.flash('success', 'Your name was updated');
            }
            res.redirect('/account/name');
        });
    });

The final step is to display the messages when necessary. Bootstrap alerts provide an easy way to do this:

doctype html
html
  head
    link(rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css")
  body
    if error && error.length > 0
      .alert.alert-danger.alert-dismissible.fade-in(role="alert")
        button.close(type="button" data-dismiss="alert" aria-label="Close")
          span(aria-hidden="true") ×
        p= error
    if success && success.length > 0
      .alert.alert-success.alert-dismissible.fade-in(role="alert")
        button.close(type="button" data-dismiss="alert" aria-label="Close")
          span(aria-hidden="true") ×
        p= success

    script(src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js")
    script(src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/js/bootstrap.min.js")

Hello world with Cowboy and Websockets

Cowboy is a web server/framework written in Erlang, that provides pretty seamless websocket support.

The “getting started” guide is very good, and I’m going to assume you’ve read it, and followed it up to the “cowboy setup” section.

We’re going to add a ws handler, rather than an http one, using the template:

make new t=cowboy_ws n=ws_handler

And wire it up in the app:

start(_Type, _Args) ->
    Dispatch = cowboy_router:compile([{'_', [
        {"/connect", ws_handler, []} 
    ]}]),
    Port = 8080,
    cowboy:start_http(my_http_listener, 100, [{port, Port}], [{env, [{dispatch, Dispatch}]}]),
    ping_pong_sup:start_link().

You could now build and run the release, and connect to a websocket by calling http://localhost:8080/connect. But it wouldn’t be very interesting… yet.

The next job is to add a web page that will call the server, in priv/static/index.html:

<html>
    <head>
        <title>Ping-Pong</title>
    </head>

    <body>
        <h1>Ping-Pong</h1>
    </body>

    <script>
var socket = new WebSocket('ws://localhost:8080/connect');
    
socket.onopen = function() {
    console.log('connected');
};
    </script>
</html>

and add a static handler to the dispatcher:

    Dispatch = cowboy_router:compile([{'_', [
        {"/", cowboy_static, {priv_file, ping_pong, "static/index.html"}},
        {"/connect", ws_handler, []} 
    ]}]),

Still not hugely exciting, but if you look in the js console in your browser you should be connected to a websocket! We can now send a message to the server:

socket.onopen = function() {
    console.log('connected');
    send('ping');
};

function send(data) {
    console.log('Sending data: ' + data);
    socket.send(data);
}

and return the standard response:

websocket_handle({text, <<"ping">>}, Req, State) ->
    Reply = {text, <<"pong">>},
    {reply, Reply, Req, State};

which we can listen for on the client:

socket.onmessage = function(ev) {
    console.log('Received data: ' + ev.data);
};

If all that went well, then we can liven things up by adding some json into the mix. First, add jiffy as a dependency (other json libraries are available!), then update the js to send some json instead:

socket.onopen = function() {
    console.log('connected');
    var msg = {type: 'ping', count: 1};
    send(JSON.stringify(msg));
};

and handle that on the server:

websocket_handle({text, Json}, Req, State) ->
    Map = jiffy:decode(Json, [return_maps]),
    Count = maps:get(<<"count">>, Map),
    Reply = #{type => <<"pong">>, count => Count + 1},
    {reply, {text, jiffy:encode(Reply)}, Req, State};

Just remember that the socket receives the json as text, you need to parse it!

socket.onmessage = function(ev) {
    console.log('Received data: ' + ev.data);
    var msg = JSON.parse(ev.data);
}

An alternative to parsing the json is to use pattern matching:

websocket_handle({text, <<"{\"type":\"ping\",\"count\":", Count:1/binary, "}">>}, Req, State) ->
    Reply = #{type => <<"pong">>, count => list_to_integer(binary_to_list(Count)) + 1}, 
    {reply, {text, jiffy:encode(Reply)}, Req, State};

but in this case, it has the disadvantage that the length of Count has to be specified.

You can find the source code for these examples here, and there’s another example in the Cowboy repo.

Using the erlang.mk gen_server template

I can never remember how to do this when the time comes, so here’s a note to future self:

make new t=gen_server n=foo_server

The generated file will be in src/foo_server.erl. You can also list all the available templates using:

make list-templates

Available templates: cowboy_http cowboy_loop cowboy_rest cowboy_ws gen_server ranch_protocol supervisor

Or just look at the source.

The promise handler

Continuing from my last adventures with promises and express.js, it turns out I can have what I want. Sort of. Behold the promise handler:

module.exports = function(inner) {
    this.handle = function(req, res) {
        res.promise(inner.handle(req));
    };
};

So now my express handlers look like this:

module.exports = function() {
    this.handle = function(req) {
        return somethingThatReturnsAPromise();
    };
};

And you would wire it all together like this:

var app = express();
app.use(promiseMiddleware());
app.get("/foo", function(req, res) {
    var handler = new PromiseHandler(new FooHandler());
    return handler.handle(req, res);
});

(As a side-note, I’m a huge fan of the Russian doll model of composing behaviour of handlers)

Debugging an ansible module

Debugging an ansible module can be a pretty thankless task; luckily the team has provided some tools to make it a little easier. While it’s possible to attach a debugger (e.g. epdb), good old fashioned println debugging is normally enough.

If you just add a print statement to the module, and run it normally, then you’ll be disappointed to see that your output is nowhere to be seen. This is due to the way that ansible modules communicate, using stdin & stdout.

The secret is to run your module using the test-module script provided by the ansible team. You can then pass in the arguments to your module as a json blob:

hacking/test-module -m library/cloud/rax -a "{ \"credentials\": \"~/.rackspace_cloud_credentials\", \"region\": \"LON\", \"name\": \"app-prod-LON-%02d\", \"count\": \"2\", \"exact_count\": \"yes\", \"group\": \"app_servers\", \"flavor\": \"performance1-1\", \"image\": \"11b0cefc-d4ec-4f09-9ff6-f842ca97987c\", \"state\": \"present\", \"wait\": \"yes\", \"wait_timeout\": \"900\" }"

The output of this script can be a bit verbose, so if you’re only interested in your output it can be worthwhile commenting out the code that prints out the parsed output, and just keeping the raw version.

Connecting to postgres using domain sockets and libpq

We recently started using Goose to run our migrations, and I wanted to connect to the Postgres instance on a local machine using a domain socket. The documentation states:

host
Name of host to connect to. If this begins with a slash, it specifies Unix-domain communication rather than TCP/IP communication; the value is the name of the directory in which the socket file is stored. The default behavior when host is not specified is to connect to a Unix-domain socket in /tmp (or whatever socket directory was specified when PostgreSQL was built). On machines without Unix-domain sockets, the default is to connect to localhost.

So I assumed that specifying a connection string without a host would “just work”:

db, err := sql.Open("postgres", "dbname=foo")

This doesn’t return an error, but any attempt to use the connection resulted in a “bad connection” failure. For some reason I needed to specify the path to the socket (which wasn’t in /tmp on the Debian instance I was using!):

db, err := sql.Open("postgres", "host=/run/postgresql dbname=foo sslmode=disable")

Adding instances to multiple host groups using the Ansible rax module

We use the Ansible rax module to create new instances of our “cloud servers”. It’s pretty easy to add them to one group:

- name: Build a Cloud Server
  tasks:
      local_action:
        module: rax
        name: rax-test1
        wait: yes
        state: present
        networks:
          - private
          - public
        group: app-servers

But it’s also quite handy to be able to place a server in multiple groups (e.g. test / production, different regions etc). There’s nothing in the documentation about this, but a bit of code spelunking reveals that a metadata key named “groups” can contain a comma-separated list of extra host groups:

- name: Build a Cloud Server
  tasks:
      local_action:
        module: rax
        name: rax-test1
        wait: yes
        state: present
        networks:
          - private
          - public
        group: app-servers
        meta:
            groups: test, london