mongodb cursor id not valid error

Question

I am trying to iterate through this loop:

for doc in coll.find()

I get the following error at the 100,000th plus record.

File "build\bdist.win32\egg\pymongo\cursor.py", line 703, in next
File "build\bdist.win32\egg\pymongo\cursor.py", line 679, in _refresh
File "build\bdist.win32\egg\pymongo\cursor.py", line 628, in __send_message
File "build\bdist.win32\egg\pymongo\helpers.py", line 95, in _unpack_response
pymongo.errors.OperationFailure: cursor id '1236484850793' not valid at server

what does this error mean?

Possible duplicate of [MongoDB - Error: getMore command failed: Cursor not found](https://stackoverflow.com/questions/44248108/mongodb-error-getmore-command-failed-cursor-not-found) — vovchisko, Jun 07 '18 at 18:58

Reto Aebersold · Accepted Answer · 2014-06-10T13:50:12.733

38

Maybe your cursor timed out on the server. To see if this is the problem, try to set timeout=False`:

for doc in coll.find(timeout=False)

See http://api.mongodb.org/python/1.6/api/pymongo/collection.html#pymongo.collection.Collection.find

If it was a timeout problem one possible solution is to set the batch_size (s. other answers).

edited Jun 10 '14 at 13:50

answered Apr 24 '12 at 13:05

Reto Aebersold

16,306
5
55
74

2

The FAQ suggests you have this correct: http://api.mongodb.org/python/current/faq.html#what-does-operationfailure-cursor-id-not-valid-at-server-mean – Steven Rumbalski Apr 24 '12 at 13:10
im at the 50,000th record. waiting to see if im thru :) – codious Apr 24 '12 at 13:14
1

Please note that this consumes memory on the server until you explicitly close it. See http://www.mongodb.org/display/DOCS/Queries+and+Cursors#QueriesandCursors-ClosingandTimeouts – Craig Younkins Nov 07 '12 at 16:07
@CraigYounkins correct me if I'm wrong, but with pymongo, I believe a cursor object is closed when it is garbage collected, so this isn't such a problem unless there is something like a network failure (so pymongo can't tell the server to close it). – drevicko Mar 20 '15 at 11:22
@drevicko sorry but I have no idea – Craig Younkins Mar 22 '15 at 22:47
1

I just looked it up in the pymongo source: yes, it does get closed when the cursor is garbage collected. Network failures and broken garbage collection (e.g.: circular refs in python destructors) could stop that from working though. – drevicko Mar 23 '15 at 04:31
If this doesn't work, no_cursor_timeout=True might work. – akki Jul 03 '15 at 05:52

score 35 · Answer 2 · edited Dec 23 '15 at 07:54

35

Setting the timeout=False is dangerous and should never be used, because the connection to the cursor can remain open for unlimited time, which will affect system performance. The docs specifically reference the need to manually close the cursor.
Setting the batch_size to a small number will work, but creates a big latency issue, because we need to access the DB more often than needed.
For example:
5M docs with a small batch will take hours to retrieve the same data that a default batch_size returns in several minutes.

In my solution it is mandatory to use sort on the cursor:

done = False
skip = 0
while not done:
    cursor = coll.find()
    cursor.sort( indexed_parameter ) # recommended to use time or other sequential parameter.
    cursor.skip( skip )
    try:
        for doc in cursor:
            skip += 1
            do_something()
        done = True
    except pymongo.errors.OperationFailure, e:
        msg = e.message
        if not (msg.startswith("cursor id") and msg.endswith("not valid at server")):
            raise

edited Dec 23 '15 at 07:54

Shuai Zhang

2,011
3
22
23

answered Oct 01 '13 at 13:34

Oran

877
7
13

This is a nice solution, although if you have a few million entries and no "time or other sequential parameter" it's impractical. Can't believe there is no off the self solution. – Mario Alemi Jan 30 '14 at 08:31
1

Just to make it clear. This solution (either the batch one) do not ensure iterate all the documents just once. Some document could be yield more than once or skipped if the database get updated between queries. For statistical proposes this is usually not a problem however if you need exactitude this may be a problem in some cases. – David Mabodo Jun 23 '15 at 10:12
"timeout=False is dangerous and should never be used" this is untrue - if I have a script which iterates over collection elements, one by one and on each I spent 5s-15min period of time, then I should be using this option. There is no risk of cursor remaining afterwards since it will be destroyed along with the collection at the end of script. – Maciej Urbański Jan 06 '20 at 12:10
@MaciejUrbański This was true according to Mongo 2.4/2.6 docs. This parameter was changed, even by name, and probably by functionality. The question doesn't have a tag to the Mongo version, so this is a bit confusing. Anyway, keeping cursors open is dangerous due to potentially high memory usage, if you open many cursors and forget to close them. Consider this option as an advanced user feature, which is usually not something you want to do - although there are always exceptions. Thanks for you feedback :) – Oran Jan 09 '20 at 15:23

score 26 · Answer 3 · answered Jul 14 '13 at 21:36

Setting timeout=False is a very bad practice. A better way to get rid of the cursor id timeout exception is to estimate how many documents your loop can process within 10 minutes, and come up with an conservative batch size. This way, the MongoDB client (in this case, PyMongo) will have to query the server once in a while whenever the documents in the previous batch were used up. This will keep the cursor active on the server, and you will still be covered by the 10-minute timeout protection.

Here is how you set batch size for a cursor:

for doc in coll.find().batch_size(30):
    do_time_consuming_things()

I agree, this sounds like the better solution – Tom Swifty Sep 25 '13 at 12:54 — Tom Swifty, Sep 25 '13 at 12:54

score 0 · Answer 4 · answered Aug 01 '17 at 23:45

0

you should choose a low value of batch_size to fix the issue:

col.find({}).batch_size(10)

see the following answer

answered Aug 01 '17 at 23:45

HISI

4,557
4
35
51

score -1 · Answer 5 · answered Nov 27 '15 at 18:37

-1

You can also force evaluation by using:

for doc in list(coll.find())

answered Nov 27 '15 at 18:37

laura112

50
2

@peterh The question is about fixing the cursor timeout issue, not about explaining how cursors and batches work. I agree a more detailed explanation would be great, but this answer is still valid, as converting the `cursor` to `list` will force it to retrieve all the batches and close, most likely before the 10 minutes default expiration time. – Danziger May 30 '17 at 00:29

mongodb cursor id not valid error

5 Answers5

Linked

Related