Node.js and systemd-notify

Running a nodejs app as a systemd service is pretty simple. But the downside to this approach is that if the app dies straight away (e.g. a config file is malformed), then systemd remains unaware.

An alternative is to switch to using notify:

[Service]
Type=notify
ExecStart=/var/www/app_name/app.js
Restart=always
...

There are various ways to notify systemd that your service is ready; but the simplest is to use systemd-notify, a console wrapper around sd_notify.

this.start = function(port) {
    var deferred = Q.defer();

    server = app.listen(port, '127.0.0.1', function(error) {
        if (error) {
            deferred.reject(new Error(error));
        } else {
            logger.info('Listening on port %d', port);
            exec('systemd-notify --ready');
            deferred.resolve();
        }
        });

        return deferred.promise;
    };
};

If the process that calls systemd-notify is not the one in ExecStart (e.g. you are using cluster), you will also need to set NotifyAccess to “all”.

Now when you use systemctl to start your service, it will wait for the notification before deeming the start process complete.

Parsing json from syslog entries with logstash

A consequence of moving to Debian 8 (and hence systemd), is that all our log data now goes to syslog. So long logrotate!

It does however require a change to the way we filter them, once they’ve been aggregated:

filter {
    if [type] == "syslog" {
        grok {
            match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:syslog_message}" }
        }
    }

    if [program] == "foo" {
        json {
            source => "syslog_message"
        }
        mutate {
            convert => [ "level", "integer" ]
            remove_field => [ "hostname "]
        }
        date {
            match => [ "time", "ISO8601" ]
        }
    }
}

First, we parse the syslog entry, and put the free form message into a property named “syslog_message”. We could overwrite the existing message, but this makes it easier to investigate if it goes wrong.

Then, if the “program” (set by the SyslogIdentifier in your systemd unit file) matches, we parse the message as json and tidy up a few fields.

Ansible & systemctl daemon-reload

(UPDATE: there is now a PR open to add this functionality)

We recently migrated to Debian 8 which, by default, uses systemd. I can appreciate why some people have misgivings about it, but from my point of view it’s been a massive improvement.

Our unit files look like this now:

[Service]
ExecStart=/var/www/{{ app_name }}/app.js
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier={{ app_name }}
User={{ app_name }}
Group={{ app_name }}
Environment=NODE_ENV={{ env }}
WorkingDirectory=/var/www/{{ app_name }}

[Install]
WantedBy=multi-user.target

Compared to a 3 page init script, using start-stop-daemon. And we no longer need a watchdog like monit.

We do our deployments using ansible, which already knows how to play nice with systemd. One thing missing though, is that if you change a unit file you need to call systemctl daemon-reload before the changes will be picked up.

There’s a discussion underway as to whether ansible should take care of it. But for now, the easiest thing to do is add another handler:

- name: Install unit file
  sudo: true
  copy: src=foo.service dest=/lib/systemd/system/ owner=root mode=644
  notify:
    - reload systemd
    - restart foo

with a handler like this:

- name: reload systemd
  sudo: yes
  command: systemctl daemon-reload

UPDATE: if you need to restart the service later in the same play, you can flush the handlers to ensure daemon-reload has been called:

- meta: flush_handlers