25

EDIT: Basically I'm looking for some hints on how to understand what background operations are running on my MongoDB instance and possibly reduce/disable them when necessary so they don't interfere with running tests. I've tried mongostat and mongotop but haven't found anything with them that helped me understand what background operations are running and what is initiating them. db.currentOp() consistently returns an empty array when run before I start running my tests.

I regularly run tests while developing with node (mocha, cucumber). Since yesterday, about 25% of the time server initialization fails trying to connect to mongodb with the following error:

**Unhandled rejection MongoError: exception: cannot perform operation: a background operation is currently running for collection** somecollection
    at Function.MongoError.create (/somepath/node_modules/pow-mongodb-fixtures/node_modules/mongodb/node_modules/mongodb-core/lib/error.js:31:11)
    at /somepath/node_modules/pow-mongodb-fixtures/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:793:66
    at bound (domain.js:254:14)
    at runBound (domain.js:267:12)
    at Callbacks.emit (.../node_modules/pow-mongodb-fixtures/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:94:3)
    at null.messageHandler (/somepath/node_modules/pow-mongodb-fixtures/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:235:23)
    at Socket.<anonymous> (/somepath/node_modules/pow-mongodb-fixtures/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:294:20)
    at Socket.emit (events.js:107:17)
    at readableAddChunk (_stream_readable.js:163:16)
    at Socket.Readable.push (_stream_readable.js:126:10)
    at TCP.onread (net.js:538:20)

We use pow-mongodb-fixtures to clear the db and fill it with some basic data before running tests, which is where this is happening. AFAIK nothing major changed when this started happening. Any ideas where I can even start researching the source of this error?

joniba
  • 3,339
  • 4
  • 35
  • 49
  • 1
    check running background process using db.currentOp(). In case of active background process your client is configured to throw this error – undefined_variable Nov 21 '15 at 13:54
  • This command always returns an empty array. Probably by the time I run it the "background operation" has finished. Nevertheless, I sometimes need to run 3 or 4 times to get past this error. – joniba Nov 24 '15 at 11:46
  • You might be on the right track with `db.currentOp()` but maybe just not using it correctly. This article goes more in depth; [Viewing and killing currently executing operations in MongoDB](https://masteringmean.com/lessons/104-Viewing-and-killing-currently-executing-operations-in-MongoDB) – Philip O'Brien Nov 26 '15 at 15:30
  • Fascinating. A normal query should not be disrupted by any background operation. Is it possible that you try to create an index while another background index creation is already in progress? – Markus W Mahlberg Nov 28 '15 at 10:30
  • 1
    Before running tests we 1. Clear the entire db, then 2. Insert some basic documents. Step 2 is run using mongoose so it also adds indexes, but this happens only after clearing is done. As far as I can tell it's the clearing that fails. I could be wrong and the error could be from the insertion step where indexes are ensured (indexes that should all already exist). – joniba Dec 07 '15 at 05:06
  • 1
    @robocode, again db.currentOp() returns an empty array when I run it. The article you linked assumes you have an operation that takes a minute or more to complete and you have plenty of time to investigate. In my case these background operations seem to run for a second or two and then stop, but frequently, occasionally interfering with clearing the database. – joniba Dec 07 '15 at 20:32
  • This isn't an *elegant* solution, but what I did was surround my index update scripts in a try/catch, and added some logic to wait 1 second then retry, up to 10 times. Whenever it fails, it tends to work on the first retry. :) – Kip Feb 23 '21 at 19:01

2 Answers2

11

Well I'm not going to mark this as the answer because I was looking for a way to monitor what is happening and figure out the issue through mongo db. Nevertheless, I did figure out the problem. Not the most helpful solution in the world, but it turned out that we were asynchronously starting the server, which would run all the mongoose schemas and thereby check/recreate all indexes, and more or less simultaneously I was running the fixtures code to clear my collections.

In short, don't try to delete collections while creating indexes on them :)

joniba
  • 3,339
  • 4
  • 35
  • 49
  • 2
    If you were using Mongoose, it's possible that you missed using the option {"background": false} in the index creation. Then the indexes could potentially be running in the background while some operation is trying to drop the collections. Setting background to false makes the index to be run in foreground. – João Otero Jan 17 '19 at 23:34
  • Another workaround when using Mongoose is call "model.init()" before dropping the collection. This should work even when indexes are created in background. More info at https://thecodebarbarian.com/whats-new-in-mongoose-5-2-syncindexes – kctang Jan 31 '20 at 06:24
8

Your friend in this case would be:

db.currentOp(true)

Specifying true includes operations on idle connections and system operations.

See here: https://docs.mongodb.org/manual/reference/method/db.currentOp/

Roi Tal
  • 1,000
  • 6
  • 14
  • 1
    Of course I did, but you kept on specifying running db.currentOp() without the extra parameter which might have helped in your case. Anyway, very glad to hear your problem was solved :) – Roi Tal Dec 20 '15 at 22:20