Bulk Delete an attribute from a document in ElasticSearch 6+

Question

I need to delete a particular attribute from all documents in ElasticSearch. Is there a way to do it and can it be done via Bulk API. I am unable to find an API to Bulk Delete an attribute from ElasticSearch.

Is there any way to achieve the same.

Sample part of my document:

{
  "media": {
    "list1": [
      {"title":"a"}
    ],
    "list2": [
      {"title":"b"}
    ]
  }
}

Script to delete the same :

{
  "script": "ctx._source.remove('media.list1')",
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "media.list1"
          }
        }
      ]
    }
  }
}

score 2 · Answer 1 · answered Feb 07 '19 at 19:23

2

If you are looking to just delete a field from the doc (rather than deleting the entire doc), you need to use a painless script. See this question post as reference for writing a script to delete a field from a document.

To update all docs in an index, put the painless script to remove a field inside an Update By Query API, using a match_all as the 'query' to update all documents.

You wouldn't want to use Bulk API for this. Bulk API would be used if you wanted to replace entire documents (update) with new information.

answered Feb 07 '19 at 19:23

Dennis

415
4
13

Suppose I have json in the form below : {"media":{"list1":[{"title":"a"}],"list2":[{"title":"b"}]}} I am able to delete entire media object but unable to delete let's say list1. Can you please let me know, how can i shape my script to do that? – jatin mahajan Feb 07 '19 at 20:02
My script to do the same goes like this : {"script":"ctx._source.remove('media.list1')","query":{"bool":{"must":[{"exists":{"field":"media.list1"}}]}}} – jatin mahajan Feb 07 '19 at 20:06
You have to access media before the remove function. I tested this script to delete list1: {"script":"ctx._source.media.remove('list1')","query":{"bool":{"must":[{"exists":{"field":"media.list1"}}]}}} – Dennis Feb 07 '19 at 20:38
You need to give more information about your problem. The script I gave works to delete list1 from the sample you gave. What exactly is different and what is your error? – Dennis Feb 08 '19 at 16:58
Actually i was able to delete the same but i faced a strange issue. The query sometimes work and update the same but other times it fails and nothing happens. For non-nested objects in the document, it works perfectly fine but ambiguous otherwise. I was able to achieve the same using Java RestHighLevelClient for Elasticsearch version 6.5.4 by using DeleteByQuery class. Also, I am using aws managed elasticsearch and it has support for version 6.4 as of now and DeleteByQuery is not available in it. What can be done in such scenario? – jatin mahajan Feb 12 '19 at 08:42
I haven't used aws managed elasticsearch before -- I would check to see if someone has written a plugin to add that command. If not, you would have to write a manual solution. Query the docs you want to delete (using scroll api for big number of hits), store the IDs in a collection, then finally send batches of bulk api requests to delete those documents in the collection. – Dennis Feb 13 '19 at 22:46

Bulk Delete an attribute from a document in ElasticSearch 6+

1 Answers1