How to remove duplicates from mongoDB and keeping unique data using aggregate function

Question

Hi i have been trying to show an output of unique data without the duplicates in the Mongodb.

Below is one sample of my entire document in my mongoDB

{
        "_id" : ObjectId("5a43aa19d4b45e362428e2da"),
        "comments_data" : {
                "id" : "28011986676_10155780942281677",
                "comments" : {
                        "paging" : {
                                "cursors" : {
                                        "after" : "WTI5dGJXVnVkRjlqZAFhKemIzSTZAN4TlRBeE5EWXlPQT09",
                                        "before" : "WTI5dGJXVnVk4TVRZAMk56YzZANVFV4TlRBeE5EWXlPQT09"
                                }
                        },
                        "data" : [
                                {
                                        "created_time" : "2018-01-03T21:23:47+0000",
                                        "message" : "Poor customer care service after became the Singtel customer.I did my          re contract they send acknowledgement email confirmation after no followup.I called again and remains no proper response and action extremely worst customer care service.",
                                        "from" : {
                                                "name" : "Sundararaju G",
                                                "id" : "1020391"
                                        },
                                        "id" : "10155780942281677_10155811924116677"
                                }
                        ]
                }
        },
        "post_id" : "28011986676_10155780942281677",
        "post_message" : "\"Singtel TV celebrated our 10th birthday with 10 awesome experiences for our customers! Each of our winners won a trip of a lifetime - from attending the Emmy Awards, getting a magical princess treatment at Disneyland, to catching a Premier League game live in London! We thank all our customers for your support and we look forward to more great years to come!\"",
        "reactions_data" : {
                "reactions" : {
                        "paging" : {
                                "cursors" : {
                                        "after" : "TVRBd01EQXpNVEF5T1Rje4TXc9PQZDZD",
                                        "before" : "TVRjNE56TTBBek56a3hNek14TWc9PQZDZD"
                                },
                                "next" : "https://graph.facebook.com/v2.7/280119866761677/reactions?access_token=EAA"
                        },
                        "data" : [
                                {
                                        "type" : "ANGRY",
                                        "id" : "1020573391",
                                        "name" : "Sundararaju Gh"
                                },
                                {
                                        "type" : "LIKE",
                                        "id" : "64721496",
                                        "name" : "Zhiang Xian"
                                }
                        ]
                },
                "id" : "28011986676_102281677"
        }
}

after that i have been trying to extract out the message comments at the comments_data field. i have tried using this query

db.sInsert.find({post_id: "28011986676"}, {post_id:1, 'comments_data.comments.data.message':1})

where the results i get is all duplicates data.

{ "_id" : ObjectId("5a43aa19d4b45e362428e2ec"), "comments_data" : { "comments" : { "data" : [ { "message" : "Who else loves Apple ?" } ] } }, "post_id" : "28011986676" }
{ "_id" : ObjectId("5a4660f2d4b45e3698398320"), "comments_data" : { "comments" : { "data" : [ { "message" : "Who else loves Apple ?" } ] } }, "post_id" : "28011986676" }
{ "_id" : ObjectId("5a47ae92d4b45e2148941901"), "comments_data" : { "comments" : { "data" : [ { "message" : "Who else loves Apple ?" } ] } }, "post_id" : "28011986676" }
{ "_id" : ObjectId("5a4928b1d4b45e208cfd6916"), "comments_data" : { "comments" : { "data" : [ { "message" : "Who else loves Apple ?" } ] } }, "post_id" : "28011986676" }

Thereafter, i have tried using the aggregate function inside the MongoDB

db.sInsert.aggregate([   { $match: {post_id: {"$eq": "28011986676" } } },   { $project: {'message': '$comments_data.comments.data.message', _id:0} }, ])

The result i have get is

{ "message" : [ "Who else loves Apple ?" ] }
{ "message" : [ "Who else loves Apple ?" ] }
{ "message" : [ "Who else loves Apple ?" ] }
{ "message" : [ "Who else loves Apple ?" ] }

How do i get unique result by using the aggregate function ? I also have tried using ensureIndex, it still does not helps in returning unique results from the mongoDB.

Saravana · Accepted Answer · 2018-01-28T06:52:03.960

0

If you need just the distinct messages you can simply do a distinct query

db.sInsert.distinct(
    "comments_data.comments.data.message", 
    {post_id: {"$eq": "28011986676" }}
)

also you can do the same with aggregation as well, $group with $addToSet to get unique messages

db.sInsert.aggregate(
    {$match :  {post_id: {"$eq": "28011986676" }}},
    {$group : {_id:null, messages: {$addToSet : "$comments_data.comments.data.message"}}},
    {$project : {_id:0, messages: {$arrayElemAt : ["$messages", 0]}}}
)

edited Jan 28 '18 at 06:52

answered Jan 28 '18 at 06:46

Saravana

12,647
2
39
57

Thanks!! May i ask one more question? sorry, it will sounds like another question. what if i want to export them out, can i just query them out ? – xxxSL Jan 28 '18 at 07:03
either you can iterate the results or `$out` to another collection – Saravana Jan 28 '18 at 07:04

How to remove duplicates from mongoDB and keeping unique data using aggregate function

1 Answers1