How to paginate an aggregation pipeline result in pymongo?

Question

I have a web app where I store some data in Mongo, and I need to return a paginated response from a find or an aggregation pipeline. I use Django Rest Framework and its pagination, which in the end just slices the Cursor object. This works seamlessly for Cursors, but aggregation returns a CommandCursor, which does not implement __getitem__().

cursor = collection.find({})
cursor[10:20] # works, no problem

command_cursor = collection.aggregate([{'$match': {}}])
command_cursor[10:20] # throws not subscriptable error

What is the reason behind this? Does anybody have an implementation for CommandCursor.__getitem__()? Is it feasible at all?

I would like to find a way to not fetch all the values when I need just a page. Converting to a list and then slicing it is not feasible for large (100k+ docs) pipeline results. There is a workaround with based on this answer, but this only works for the first few pages, and the performance drops rapidly for pages at the end.

score 3 · Accepted Answer · answered Jul 06 '20 at 14:10

Mongo has certain aggregation pipeline stages to deal with this, like $skip and $limit that you can use like so:

aggregation_results = list(collection.aggregate([{'$match': {}}, {'$skip':  10}, {'$limit':  10}]))

Specifically as you noticed Pymongo's command_cursor does not have implementation for __getitem__ hence the regular iterator syntax does not work as expected. I would personally recommend not to tamper with their code unless you're interested in becoming a contributer to their package.

score 2 · Answer 2 · answered Jul 06 '20 at 14:16

The MongoDB cursor for find and aggregate functions in a different way since cursor result from aggregation query is a result of precessed data (in most cases) which is not the case for find-cursors as they are static and hence documents can be skipped and limitted to your will.

You can add the paginator limits as $skip and $limit stages in the aggregation pipeline.

For Example:

command_cursor = collection.aggregate([
    {
        "$match": {
            # Match Conditions
        }
    },
    {
        "$skip": 10  # No. of documents to skip (Should be `0` for Page - 1)
    },
    {
        "$limit": 10  # No. of documents to be displayed on your webpage
    }
])

How to paginate an aggregation pipeline result in pymongo?

2 Answers2