ELK

We’re running a relatively old version of ELK (1.x), which is working fine but I don’t want to slip too far behind (the beta for 5.0 is out, although that was a jump from 2.x).

The last time I tried to upgrade I ran into problems with “mapping conflicts”, which are no longer acceptable. I tried to fix the individual conflicts, yet somehow ended up with more conflicts than I had before.

This time, I decided to nuke it from orbit, and use separate indices for different log types:

output {
    elasticsearch {
        host => localhost
        index => "%{type}-%{+YYYY.MM.dd}"
    }
}

This means, for example, that my nginx and postgresql logs are in separate indices; and therefore, similarly named fields no longer conflict.

The main benefit and downside are intertwined: it makes querying simpler as you don’t have to include the type, but you can’t query across log types (this hasn’t been a problem, so far). It also makes it far simpler to check which logs are taking up all the space on disk, and probably in memory.

I recently started upgrading to a newer version of ES (4.5), and found that it refused to start:

IllegalStateException: unable to upgrade the mappings for the index

In fact, this mapping conflict was one of the things I was hoping the upgrade would solve. After a bit of reading it became clear that I would have to make some changes.

The mapping in question was a field from the logs called “level”. In the postgres logs it was a string (e.g. “INFO”), and in our application logs (using bunyan) it was an integer (40 => “WARN”).

To allow me to search using a range (e.g. level:[40 TO 60]), I was using a mutate filter to convert the string “40” to an integer, and this was the cause of the conflict.

My first thought was to copy the field before converting:

mutate {
    add_field => { "level_int" => "%{level}" }
    convert => { "level_int" => "integer" }
}

But it turns out that that’s not enough to avoid a conflict (possibly because ES guesses the type, and saw an int first?). So I went with the nuclear option, and renamed the field:

mutate {
    rename => { "level" => "level_int" }
    convert => { "level_int" => "integer" }
}

Now my new documents were conflict free. Unfortunately, the only solution provided for existing data is to export and re-import it, which I wasn’t really in the mood for.

Luckily, I’m not in any rush to upgrade, and we close indices after 30 days. So I plan to wait for a month, and hope my data is clean by then!

Look on my works, ye Mighty, and despair!

Using multiple indices with ELK

Mapping conflicts with ELK