Parsing json from syslog entries with logstash

A consequence of moving to Debian 8 (and hence systemd), is that all our log data now goes to syslog. So long logrotate!

It does however require a change to the way we filter them, once they’ve been aggregated:

filter {
    if [type] == "syslog" {
        grok {
            match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:syslog_message}" }
        }
    }

    if [program] == "foo" {
        json {
            source => "syslog_message"
        }
        mutate {
            convert => [ "level", "integer" ]
            remove_field => [ "hostname "]
        }
        date {
            match => [ "time", "ISO8601" ]
        }
    }
}

First, we parse the syslog entry, and put the free form message into a property named “syslog_message”. We could overwrite the existing message, but this makes it easier to investigate if it goes wrong.

Then, if the “program” (set by the SyslogIdentifier in your systemd unit file) matches, we parse the message as json and tidy up a few fields.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s