Warm indexes not moving

We have an ES cluster, using the hot-warm architecture. The ILM policy was rolling the indexes, and moving them to the warm phase; but the indexes were stuck on the (expensive) hot nodes, and the warm nodes were sitting there, with empty disks.

You can check exactly what is where, using cat shards, and the metrics were correct. I decided to try and force a move, using cluster reroute:

POST /_cluster/reroute
{
  "commands": [
    {
      "move": {
        "index": "foo", "shard": 0,
        "from_node": "hotnode1", "to_node": "warmnode2"
      }
    }
}

And got told:

[NO(node does not match index setting [index.routing.allocation.require] filters [data:\"hot\"])]

On closer inspection of the index settings, I realised that although the ILM policy was adding the correct attributes to prefer a warm node:

     "routing": {
        "allocation": {
          "include": {
            "_tier_preference": "data_warm,data_hot"
          },
          "require": {
            "data": "hot"
          }
        }
      }

It wasn’t removing the existing attribute forcing it to use a hot node. It was (relatively) easy to fix that for the existing indexes:

PUT /foo/_settings
{ 
    "routing.allocation.require.data": null
}

Once the require attribute was removed, the indexes were relocated automatically. Unfortunately, I couldn’t find a way to do the same thing using ILM, other than explicitly flipping the require to warm:

            "warm": {
                "actions": {
                    ...
                    "allocate" : { 
                        "require" : { 
                            "data": "warm"
                        }
                    }
                }
            },

Testing ES ingest pipelines

If you are working with ElasticSearch, it’s useful to be able to test locally. Thanks to the magic of docker, that’s simpler than ever:

version: '3' 
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - "discovery.type=single-node"
    volumes:
      - ./data:/usr/share/elasticsearch/data
  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.1
    ports:
      - "5601:5601"

(I need to use a volume, because my root partition is tiny). With these containers running, you can set up filebeat (e.g. in vagrant), and start shipping logs. It’s then simple to test an ingest pipeline:

curl "http://localhost:9200/_ingest/pipeline/foo" -X "PUT" -d @ingest/foo.json -H 'content-type: application/json'

Or an index template:

curl "http://localhost:9200/_template/foo" -X "PUT" -d @index-template/foo.json -H 'content-type: application/json'

Or an ILM policy:

curl "http://localhost:9200/_ilm/policy/foo" -X "PUT" -d @ilm-policy/foo.json -H 'content-type: application/json'

If you want to play with APM too, you also need that server running:

  apm:
    image: docker.elastic.co/apm/apm-server:7.6.1
    ports:
      - "8200:8200"