mongodb impossible (?) E11000 duplicate key error dup key when upserting

Question

My understanding is that update with upsert:true on a single document is an atomic operation so this should never result in a duplicate key error, especially not on the primary _id key, when the collection has no unique-ly indexed fields:

Order.update({ _id: order._id }, query, { upsert: true }, cb) // with mongoose

But this appears in the mongod.log:

    2015-03-27T09:39:10.349-0400 I WRITE    [conn258236] update xyz.orders 
query: { _id: "6353f880-c6a7-4260-809f-98e0af27b9a2" } update: { $set: { ... 
} keyUpdates:0 writeConflicts:0 **exception: E11000 duplicate key error dup 
key: { : "6353f880-c6a7-4260-809f-98e0af27b9a2" } code:11000** numYields:1 
locks:{} 138ms


    2015-03-27T09:39:10.349-0400 I COMMAND  [conn258236] command xyz.$cmd 
command: update { update: "orders", writeConcern: { w: 1 }, ordered: true, 
updates: [ { q: { _id: "6353f880-c6a7-4260-809f-98e0af27b9a2" }, u: { $set: { 
... } }, multi: false, upsert: true } ] } keyUpdates:0 writeConflicts:0 
numYields:0 reslen:235 locks:{} 139ms

Here is the output from db.orders.getIndexes():

{
    "v" : 1,
    "key" : {
        "_id" : 1
    },
    "name" : "_id_",
    "ns" : "xyz.orders"
},

We are using MongoDB version 3.0.0 with WiredTiger.

What is the value of `order._id` and `query` - do both have the same `_id` value? — DaveStSomeWhere, Mar 27 '15 at 19:13
Yes. The lines in the log are for a particular execution of the mongoose call with order._id set as "6353f880-c6a7-4260-809f-98e0af27b9a2". — Jason, Mar 27 '15 at 19:17
The calls are coming from the same connection with `{w : 1 }`, so it's likely they were serial and not concurrent - how are you generating the `order._id` value? Are you sure you didn't call the update twice with the same `order._id` in the same thread/process? — wdberkeley, Mar 31 '15 at 15:06
The update was indeed called twice with the same order._id. But I thought that should be okay with upsert, the first should create with that _id and the second should find that _id and update, right? — Jason, Mar 31 '15 at 17:57
@jason : what was the conclusion here? I faced a similar issue today — Mandeep Singh, Aug 24 '15 at 22:25
Facing the same problem, anybody knows why this might be happening? — Ismael, Sep 30 '15 at 18:37
My solution was to just handle the error code E11000. Sorry still have no explanation for why error occurs. Might have to ask a MongoDB developer. — Jason, Dec 16 '15 at 20:06

score 34 · Accepted Answer · answered Apr 18 '16 at 15:01

34

I am afraid that this is an ongoing problem. I had the same problem and I found a jira ticket about this:

https://jira.mongodb.org/browse/SERVER-14322

It is possible that two updates come in with upsert:true, resulting in neither finding a document and both inserting new documents which conflict on unique index violations of the query predicate.

The "solution" here is to add a retry code into the client.

answered Apr 18 '16 at 15:01

jcoll

740
9
10

5

Yes it is super annoying bug, cant believe it is still there in 3.4. Doing retries client side ad infinitum is an ugly workaround. – user2667976 Jun 18 '17 at 09:38
You only need to retry once at most. And if at all the retry is needed (due to duplicate conflict), it'll anyway result in an update. – Shashank Jul 21 '17 at 10:26
1

Still having this issue with version 3.6 – tony Jan 25 '18 at 18:20
5

This is ridiculous.. "not a bug"? It's a *race*. MongoDB's atomicity guarantee sux. – rustyx Sep 26 '18 at 07:09
1

it seems they added the retry logic in the server side "if it is safe to do so" (look at the ticket linked in the answer for details). They say that the retry logic will be added for mongo 4.1.6 and later, and 4.2. – jmmut Jul 15 '19 at 15:39
Still having with 3.6 – Vim Sep 19 '19 at 10:11
Still having the issue with a `findAndModify` and `upsert: true` in Mongo 4.2. Is it supposed to be fixed as well for `findAndModify`? – Gaël J Sep 02 '21 at 13:18

mongodb impossible (?) E11000 duplicate key error dup key when upserting

1 Answers1

Linked