How to remove duplicate search result in elasticsearch?

Question

First Create some example data (e1,e2,e3 are types and test is the index name):

PUT test/e1/1
{
  "id":1
  "subject": "subject 1"
}
PUT test/e2/1
{
  "id":1
  "subject": "subject 2"
}
PUT test/e3/2
{
  "id":2
  "subject": "subject 3"
}

Now my question is: how can I get just these two data? remove duplicate data with the same id in the curl -XGET _search result.

test/e1/1
{
  "id":1
  "subject": "subject 1"
}
test/e3/2
{
  "id":2
  "subject": "subject 3"
}

score 19 · Accepted Answer · edited May 23 '17 at 12:26

19

First you will need to search across multiple index.
Then, on the result remove the duplicate ID.

POST  http://myElastic.com/test/e1,e2,e3/_search
{
  "aggs":{
    "dedup" : {
      "terms":{
        "field": "id"
       },
       "aggs":{
         "dedup_docs":{
           "top_hits":{
             "size":1
           }
         }
       }    
    }
  }
}

This might help you:

edited May 23 '17 at 12:26

Community

1
1

answered Apr 27 '15 at 03:50

Francois Combet

316
2
8

Does aggregation keep the highest score result? And is there a method to put the `aggs` result to `hits` part in json result. – navins Apr 27 '15 at 04:07
@navins Yes, by default result are order by score. This kind of request will only keep the first result, who is also the highest score result. I don't know the 2nd part of your question (maybe ask a 2nd question on SO) – Francois Combet Apr 27 '15 at 04:12
thanks, just post another question here: http://stackoverflow.com/questions/29887583/how-to-paging-aggregation-result-in-elasticsearch – navins Apr 27 '15 at 05:07

score 7 · Answer 2 · answered Mar 09 '21 at 16:40

Check out Field Collapsing - its designed to give you 1 search result per "field".

GET /test/_search
{
  "collapse": {
    "field": "id"                
  }                      
}

Prior to this feature being added into Elasticsearch, using a terms aggregation with top hits was the best way to achieve this.

score 0 · Answer 3 · answered Jan 26 '22 at 16:18

in my case, I have a term query too

 user: {
      type: 'object',
      properties: {
        id: {
          type: 'keyword', // using keyword for removing duplicate documents from searc results
          normalizer: 'useLowercase',
        },
        name: {
          type: 'text',
        },
        thumbnail: {
          type: 'text',
        },
      },
    },

    const { body } = await elasticWrapper.client.search({
        index: ElasticIndex.Payment,
        body: {
          from: f ? f : 0,
          size: s ? s : 10,
          _source: ['user'],
          query: {
            bool: {
              must: [
                {
                  nested: {
                    path: 'cart',
                    query: {
                      term: {
                        'cart.product.id': req.params.id,
                      },
                    },
                  },
                },
              ],
            },
          },
          collapse: {
            field: 'user.id',
          },
          aggs: {
            user_count: {
              cardinality: {
                field: 'user.id',
              },
            },
          },
        },
      });

How to remove duplicate search result in elasticsearch?

3 Answers3

Linked