1

I have MongoDB version 5.0.7 installed in the default location in my PC running on Windows 10 Pro. I have a database named mydb which contains a collection called products. I want to copy the mydb database into a new database called newdb. For that, I’m trying to use mongodump and mongorestore commands as mentioned here. While doing so, although mongodump is executing and creating the dump file properly, the mongorestore command fails everytime generating a duplicate key error.

The command I use to dump the mydb database to an archive:

mongodump --archive="mongodump-mydb" --db=mydb

The command I use to restore the database from the dump to a new database:

mongorestore --archive="mongodump-mydb" --nsFrom='mydb.*' --nsTo='newdb.*'

The error message that I get:

2022-04-17T19:09:04.136+0530    preparing collections to restore from
2022-04-17T19:09:04.146+0530    reading metadata for mydb.products from archive 'mongodump-mydb'
2022-04-17T19:09:04.150+0530    restoring to existing collection mydb.products without dropping
2022-04-17T19:09:04.157+0530    restoring mydb.products from archive 'mongodump-mydb'
2022-04-17T19:09:04.209+0530    continuing through error: E11000 duplicate key error collection: mydb.products index: _id_ dup key: { _id: ObjectId('625974fe025923931b6e5e7f') }
2022-04-17T19:09:04.211+0530    continuing through error: E11000 duplicate key error collection: mydb.products index: _id_ dup key: { _id: ObjectId('62597508025923931b6e5e80') }
2022-04-17T19:09:04.211+0530    continuing through error: E11000 duplicate key error collection: mydb.products index: _id_ dup key: { _id: ObjectId('62597508025923931b6e5e81') }
2022-04-17T19:09:04.211+0530    finished restoring mydb.products (0 documents, 3 failures)
2022-04-17T19:09:04.211+0530    no indexes to restore for collection mydb.products
2022-04-17T19:09:04.211+0530    0 document(s) restored successfully. 3 document(s) failed to restore.

Surprisingly, it seems to restore the collection products in mydb database instead of newdb. As mydb already contains the products collection having rows of documents with unique ids, it’s creating a clash generating the duplicate key error.

As an alternative, while executing mongorestore I put the name of the new database in both --nsFrom and --nsTo arguments to check if it makes some difference:

mongorestore --archive="mongodump-mydb" --nsFrom='newdb.*' --nsTo='newdb.*'

Yet, it generates the same error.

Somehow, mongo seems to be ignoring the new database argument altogether. Why is this happening and how do I solve this problem? Because, copying existing data from a database into a new one is a common requirement of my project, and if I can’t restore my data from dump, all it takes is a crash to loose my records forever. So please help me in resolving this issue.

Regards

priyamtheone
  • 517
  • 7
  • 22
  • How do you like to handle the error? Ignore it (i.e. keep existing document), replace them or merge them? – Wernfried Domscheit Apr 17 '22 at 15:59
  • You don't need to save the dump to your disk, you can pipe the dump directly into mongorestore: `mongodump --db=mydb --archive=- | mongorestore --nsFrom='mydb.*' --nsTo='newdb.*' --archive=-` – Wernfried Domscheit Apr 17 '22 at 16:04
  • @WernfriedDomscheit: I don't want to restore the dump to the existing database. I want to COPY the existing database "mydb" to a new one called "newdb" while keeping the old one intact. For that, I'm dumping the old database in my PC and while restoring, I'm assigning the name of the new database through `--nsTo='newdb.*'` to which the old mydb database will be copied. That's what my objective is. Hope I could make it more clear to you this time. Is my objective possible to do? – priyamtheone Apr 17 '22 at 17:09
  • Also, instead of dumping on my disc, I tried to pipe the output of the `mongodump` to `mongorestore` with the command that you mentioned. But the same error is generated in that case too. Somehow, mongo seems to be restoring the dump from the disc to the old mydb database every time instead of copying it to the new mydb database, which is why it is causing a clash of **unique keys** generating the **duplicate jey error**. – priyamtheone Apr 17 '22 at 17:15
  • Obviously the collection already exist when you import it. Maybe use `mongorestore --drop` option. Or use [$out](https://www.mongodb.com/docs/datalake/reference/pipeline/out/) - however this copies only a single collection rather than the entire database. But you could run it in a loop. – Wernfried Domscheit Apr 17 '22 at 19:47
  • I found the root of the problem and the solution too. Check my solution answer. – priyamtheone Apr 18 '22 at 17:30

1 Answers1

3

I found the reason. The problem lies in using single quotes while passing the values of --nsFrom and --nsTo arguments. While doing so, in some environments, mongo keeps on restoring the dump to the existing database instead of the new one, thus creating a clash between the existing data and the data being restored generating the “duplicate key error”. When you encounter such a scenario, either use double quotes to pass the values of --nsFrom and --nsTo arguments or don’t use any quote at all.

This definitely seems to be a bug in MongoDB Server system or its syntax. MongoDB development team must take note of this and rectify it in the next update.

priyamtheone
  • 517
  • 7
  • 22
  • Did you create a ticket at https://jira.mongodb.org/? – Wernfried Domscheit Apr 18 '22 at 19:01
  • I just did at: https://jira.mongodb.org/browse/TOOLS-3082?jql=project%20%3D%20TOOLS – priyamtheone Apr 22 '22 at 16:28
  • Just realized single quotes in CMD Window seems to be a general problem. So, they may reject your ticket. See https://stackoverflow.com/questions/24173825/what-does-single-quoting-do-in-windows-batch-files – Wernfried Domscheit Apr 22 '22 at 16:37
  • "The inconsistent escape and quote rules used by cmd.exe vs. various command line utilities often makes it a challenge to discover the correct syntax needed for any given situation." This statement by dbenham in that thread says it all succinctly. If there's an issue involved with single quotes, they should've stated it clearly in the MongoDB documentation. Otherwise, it ends up being extremely puzzling for users, especially beginners, who will definitely try to implement the code form the documentation for testing and development purpose. – priyamtheone Apr 23 '22 at 13:29