Using multiple indices with ELK

We’re running a relatively old version of ELK (1.x), which is working fine but I don’t want to slip too far behind (the beta for 5.0 is out, although that was a jump from 2.x).

The last time I tried to upgrade I ran into problems with “mapping conflicts”, which are no longer acceptable. I tried to fix the individual conflicts, yet somehow ended up with more conflicts than I had before.

This time, I decided to nuke it from orbit, and use separate indices for different log types:

output {
    elasticsearch {
        host => localhost
        index => "%{type}-%{+YYYY.MM.dd}"

This means, for example, that my nginx and postgresql logs are in separate indices; and therefore, similarly named fields no longer conflict.

The main benefit and downside are intertwined: it makes querying simpler as you don’t have to include the type, but you can’t query across log types (this hasn’t been a problem, so far). It also makes it far simpler to check which logs are taking up all the space on disk, and probably in memory.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s