1

I have a problem but it seems to me that all the solutions I've seen are solving a problem more complex than mine.

I'm relatively new to MongoDB, but I've been using it with Apache Nutch to store crawled documents. As I am crawling in stages, and assessing each crawled result differently, to get the stats, I have been storing the results in different collections.

All I want to do is put these collections together into one big collection so I can translate and classify. All the fields are the SAME, so I don't need to add any new field.

This is the insert query I have been using for each collection:

new_crawl_130422_data.insert_one(
                        {
                        "database_url": proj_database_url,
                        "database_project_id": proj_database_id,
                        "projectname": proj_database_name,
                        "version": version,
                        "boost": boost,
                        "content": content,
                        "digest": digest,
                        "title": title,
                        "timestamp": timestamp,
                        "url": website,
                        "language": language

                        }

So, collection 1, and collection 2, have these same fields. Obviously, there would be multiple documents in each collection that have the same database_project_id as these are records mapped based on urls that match, and they have been transferred from Solr. Where the project name and url match, the document has been assigned the project_id, to show that it contains data for that project.

I thought it would be possible to just add on one collection to the end of the other like a list, but I have looked at these solutions - Copy few documents in a collection to another collection in mongo DB , Insert all documents from one collection into another collection in MongoDB database , Spring data mongodb- copy a collection and they all seem to be doing more complicated things than I need.

I don't want to change any collection, just create one large collection by putting two or more collections together.

I also saw this - https://www.mongodb.com/docs/v4.2/reference/method/db.collection.copyTo/, but it is deprecated.

Thanks in advance.

Rose_Trojan
  • 95
  • 1
  • 8
  • To combine collections into 1, we can use [$merge](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) another way to do it is with [$unionWith](https://www.mongodb.com/docs/manual/reference/operator/aggregation/unionWith/), and [$out](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out/) to create the new combined collection. Not sure what you need to do, but maybe check them. – Takis Apr 14 '22 at 11:57
  • @Takis thanks. Will try this with Wernfried's answer below. – Rose_Trojan Apr 14 '22 at 14:15

1 Answers1

6

If you like to copy all data then simply run:

db.new_crawl_130422_data.aggregate([
    {$out: "secondCollection"}
])

Note, all existing data in secondCollection will be deleted. If you like to add, i.e. merge the documents and keep existing data, then use:

db.new_crawl_130422_data.aggregate([
    {$merge: "secondCollection"}
])

If you have a large data set, then see How to copy a collection from one database to another in MongoDB

Wernfried Domscheit
  • 54,457
  • 9
  • 76
  • 110
  • thank you for this. How can this be amended to merge the two collections, say collection1 and collection2 into a third collection - collection3? This would mean that collection1 and collection2 would still be available, and collection3 will be a combination of both collections. – Rose_Trojan Apr 14 '22 at 13:59
  • 1
    Run two commands: `db.collection1.aggregate([{$merge: "collection3"}])` and `db.collection2.aggregate([{$merge: "collection3"}])` Would be the simplest. Or use [$unionWith](https://www.mongodb.com/docs/manual/reference/operator/aggregation/unionWith/) – Wernfried Domscheit Apr 14 '22 at 14:09