We’re running a relatively old version of ELK (1.x), which is working fine but I don’t want to slip too far behind (the beta for 5.0 is out, although that was a jump from 2.x).
The last time I tried to upgrade I ran into problems with “mapping conflicts”, which are no longer acceptable. I tried to fix the individual conflicts, yet somehow ended up with more conflicts than I had before.
This time, I decided to nuke it from orbit, and use separate indices for different log types:
output { elasticsearch { host => localhost index => "%{type}-%{+YYYY.MM.dd}" } }
This means, for example, that my nginx and postgresql logs are in separate indices; and therefore, similarly named fields no longer conflict.
The main benefit and downside are intertwined: it makes querying simpler as you don’t have to include the type, but you can’t query across log types (this hasn’t been a problem, so far). It also makes it far simpler to check which logs are taking up all the space on disk, and probably in memory.