How avoid pooling data in memory. When iterate cursor object in pymongo?

Question

Example:

def iter():
    c=pymongo.Connection()
    cursor=c.db.media.find().skip(0).limit(50000)
    for item in cursor:
        yield item

Before it goes in cycle for there is pause about 2 minus. It loads all data in memory before start iterate for some reason. Can i somehow avoid it?

If I do it in mongodb shell everything is ok.

Why are you wrapping the cursor (that can be iterated itself) in a generator function? — Bernie Hackett, Mar 30 '11 at 15:05

score 0 · Answer 1 · answered Jun 02 '14 at 04:15

Look at cursor's block_size method. With it you should be able to set how much you read in advance. I say should, because I'm facing some problems with it now (Getting StopIteration exception on next(cursor) when modifying batch_size in pymongo), but I'm probably making some mistake. block_size should solve your problem.

score 0 · Accepted Answer · answered Dec 21 '10 at 18:01

0

Do you know if this is possible? If c.db.media.find() returns everything instead of an iterator, I'm not sure there's much you can do.

answered Dec 21 '10 at 18:01

Falmarri

2

It does not returns everything. It returns cursor object. – Pol Dec 21 '10 at 18:11

2 Answers2