1

I have some code like:

cursor = self.db.coll.find({})
cursor.batch_size(100)
last   = next(cursor)
while True:
   ...
   while a>b:
       last = next(cursor)
   ...

Except that I want to use an idiom to pull from cursor with next, rather than use for cycle. What I see is that next raises StopIteration after the 100 block, where I expected it to refresh by the end of the block. If I remove the batch_size it works as expected, but I need to bring down the batch_size. I looked at the source code of the driver an it has something like:

def next(self):
     if self.__empty:
         raise StopIteration
     db = self.__collection.database
     if len(self.__data) or self._refresh():
         next = db._fix_outgoing(self.__data.pop(0), self.__collection)
     else:
         raise StopIteration
     return next

I don't get the purpose of that self.__empty which is explained as some ugly way to support some slicing. What I see on the next line makes more sense: if it has data or if refreshes successfully otherwise it goes on, but my invocation doesn't seem to reach that point... What do I have to do for the next() to continue pulling the next batch ?

displayName
  • 13,888
  • 8
  • 60
  • 75
user2123288
  • 1,103
  • 1
  • 13
  • 22
  • Just some program logic : I have two "streams" that I'm processing on the while (the cursor and a file) which have data sorted in he same way. I keep reading from two sources but if cursor is getting behind of file stream, I perform the next() to catch up. Don't think is relevant for the problem at hand, i.e. what is needed for the cursor to refresh past the first 100 without raising the end of the iteration. – user2123288 Jun 02 '14 at 04:11

1 Answers1

2

In the while condition where you are checking for a>b, please also put the check to see if the cursor has more items. It is done with alive in pymongo.

Reference: http://api.mongodb.org/python/current/api/pymongo/cursor.html#pymongo.cursor.Cursor.alive

displayName
  • 13,888
  • 8
  • 60
  • 75
  • 1
    Ok, did that. The result is that now I know that the cursor reached the end of the first block and avoid the exception, but my question is then, how can I get it to refresh past the first 200 ? – user2123288 Jun 02 '14 at 04:03
  • 1
    I meant the first 100. – user2123288 Jun 02 '14 at 04:12
  • 1
    I think we will need to see a little bit more of your code. There are two ways of raising StopIteration in the source code of the driver. So it is probably the second one that is being raised. There may be some problem with the database or the connection. For a debug, try outputting the count() of the cursor to see whether it is even loading the correct output. And as a good programming practice, always use `alive` before printing next value. Otherwise your code will always always crash after it has reached the last value of cursor. – displayName Jun 02 '14 at 05:36
  • Update reference to https://pymongo.readthedocs.io/en/stable/api/pymongo/cursor.html#pymongo.cursor.Cursor.alive – LeMoussel Jan 17 '22 at 12:28