Filter field value of a document while inserting into elastic search index-CodePudding

I have a simple requirement where in i need to create a template which will filter the value of a field before inserting it into Elastic search index.

Elastic search version : 7.10.2

Example : I have created a template like below.

PUT /index_test
{
  "mappings": {
    "properties": {
    "id": {
        "type": "keyword"
       
      },
    "id_type": {
        "type": "keyword"
      }
    }
  }
}

but, I want to put a filter on the value of id_type as "social", only then this filed should be inserted into the document otherwise it should be only id or other fields(in future) will be inserted.

I tried to explore fieldData option but it does not access the value rather it takes true or false. that not expected.

CodePudding user response：

You can do this using an ingest pipeline that will filter the value before indexing the document.

First, create a pipeline that will remove the id_type field if it doesn't satisfy a given condition:

PUT _ingest/pipeline/type-filter
{
  "description": "Filter the id_type field",
  "processors": [
    {
      "remove": {
        "if": "ctx.id_type != 'social'",
        "field": "id_type"
      }
    }
  ]
}

Then, when indexing your documents, you can reference this ingest pipeline so that all documents will have to go through it before being indexed.

PUT index_test/_doc/1?pipeline=type-filter
{
   "id": "123",
   "id_type": "social"       <--- this field will be indexed
}

PUT index_test/_doc/1?pipeline=type-filter
{
   "id": "456",
   "id_type": "anything"     <--- this field won't be indexed
}