Look on my works, ye Mighty, and despair!

We have an ES cluster, using the hot-warm architecture. The ILM policy was rolling the indexes, and moving them to the warm phase; but the indexes were stuck on the (expensive) hot nodes, and the warm nodes were sitting there, with empty disks.

You can check exactly what is where, using cat shards, and the metrics were correct. I decided to try and force a move, using cluster reroute:

POST /_cluster/reroute
{
  "commands": [{
      "move": {
        "index": "foo", "shard": 0,
        "from_node": "hotnode1", "to_node": "warmnode2"
      }
    }]
}

And got told:

[NO(node does not match index setting [index.routing.allocation.require] filters [data:\"hot\"])]

On closer inspection of the index settings, I realised that although the ILM policy was adding the correct attributes to prefer a warm node:

     "routing": {
        "allocation": {
          "include": {
            "_tier_preference": "data_warm,data_hot"
          },
          "require": {
            "data": "hot"
          }
        }
      }

It wasn’t removing the existing attribute forcing it to use a hot node. It was (relatively) easy to fix that for the existing indexes:

PUT /foo/_settings
{ 
    "routing.allocation.require.data": null
}

Once the require attribute was removed, the indexes were relocated automatically. Unfortunately, I couldn’t find a way to do the same thing using ILM, other than explicitly flipping the require to warm:

            "warm": {
                "actions": {
                    ...
                    "allocate" : { 
                        "require" : { 
                            "data": "warm"
                        }
                    }
                }
            },