LogStash & Bunyan, a match made in heaven

Every platform has a plethora of logging libraries; each with their own good points, and bad points. The first node.js library we tried was winston. It had all the usual features: multiple logging levels, different outputs (we actually just use stdout, and the daemon pipes it to a logfile); but in production we wanted to use logstash to aggregate and parse the logs.

Unfortunately winston suffers from a common problem: exception stack traces are split across multiple lines. It should be possible to use the multiline codec to merge the lines, but around this time I ran across bunyan.

The manifesto from Joyent explains the merits of logging in json (or some other machine parseable format), and it’s very easy to configure a logstash file input for a json file just by setting the codec:

input {
    file {
        type => "foo-app"
        path => [ "/var/log/foo-app/*.log" ]
        codec => "json"
    }   
}

Now, when your logs are shipped to the logstash server, each field will be indexed individually without needing to grok any crazy patterns.

It’s also very simple to add fields to the logger at runtime:

var logger = bunyan.createLogger({
    name: name,
    stream: process.stdout,
    level: 'info',
    anExtraField: 'foo'
});

...

logger.fields.anotherExtraField = 'bar';

which will also be searchable in logstash; making it easy to trace a request across multiple services, for example.

UPDATED: to add some sample output

{"name":"foo","hostname":"pikachu","pid":7645,"level":30,"msg":"Listening on port 1234","time":"2014-08-12T16:14:19.667Z","v":0}
{"name":"foo","hostname":"pikachu","pid":7645,"level":30,"msg":"Service started","time":"2014-08-12T16:14:19.673Z","v":0}
{"name":"foo","hostname":"pikachu","pid":7645,"level":30,"msg":"Received request for abc","time":"2014-08-12T16:32:28.600Z","v":0}
{"name":"foo","hostname":"pikachu","pid":7645,"level":50,"msg":"ERROR: bad request","time":"2014-08-12T16:32:28.600Z","v":0}

As the levels are integers (e.g. 30 => INFO), you can ask logstash for all entries where level >= ? (e.g. all ERRORs & FATALs).

Advertisements

4 thoughts on “LogStash & Bunyan, a match made in heaven

  1. Saichovsky (@Saichovsky) August 12, 2014 / 3:48 pm

    Wish you shared a snippet of what a simple log (e.g. syslog log file) would look like after being passed through bunyan

    • Graham Hay August 12, 2014 / 4:40 pm

      I’ve added some sample bunyan output above.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s