We use Bunyan in our node apps, for “structured logging”. The output json string is passed to syslog, by systemd, and then fed into ELK.
{
"name":"foo-service",
"hostname":"app-01",
"pid":30988,
"ip":"1.19.24.8",
"requestId":"1c11f448-73f2-4efa-bc63-3de787618d49",
"level":50,
"err": {
"message":"oh noes!"
}
}
Unfortunately, if that string is longer than 2048 chars (usually a stacktrace, or html returned from a web service instead of json), then the json blob ends up split over 2 lines in syslog.
This causes ELK to barf when attempting to parse the broken lines (assuming you are parsing as json), and means you won’t see those errors in Kibana.
It is possible to detect the error parsing the error, by searching for the string “Trouble parsing json”, but that’s not really a solution.
I would prefer to see a truncated error, than have the current situation, but that means either wrapping or patching Bunyan itself.
