Testing ES ingest pipelines

If you are working with ElasticSearch, it’s useful to be able to test locally. Thanks to the magic of docker, that’s simpler than ever:

version: '3' 
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - "discovery.type=single-node"
    volumes:
      - ./data:/usr/share/elasticsearch/data
  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.1
    ports:
      - "5601:5601"

(I need to use a volume, because my root partition is tiny). With these containers running, you can set up filebeat (e.g. in vagrant), and start shipping logs. It’s then simple to test an ingest pipeline:

curl "http://localhost:9200/_ingest/pipeline/foo" -X "PUT" -d @ingest/foo.json -H 'content-type: application/json'

Or an index template:

curl "http://localhost:9200/_template/foo" -X "PUT" -d @index-template/foo.json -H 'content-type: application/json'

Or an ILM policy:

curl "http://localhost:9200/_ilm/policy/foo" -X "PUT" -d @ilm-policy/foo.json -H 'content-type: application/json'

If you want to play with APM too, you also need that server running:

  apm:
    image: docker.elastic.co/apm/apm-server:7.6.1
    ports:
      - "8200:8200"

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s