How to implement Cursor based pagination for mongodb in node.js using official mongodb client when _id is overrided?

Question

I want to implement pagination for mongodb in node.js enviroment using offical mongodb package.

I have mongodb collection whose rows are like
db.registrations.findOne()

{ "_id" : {"mtid" : 156568417304061, "uid" : 156568728002501}, "tid" : 2 }

Here _id is not instance of ObjectId() nor numeric. How can I implement Cursor based pagination on this collection? I will avoid offset-based and time-based pagination.
I treid to use idea given in https://stackoverflow.com/a/58411020/1762051 but I guess this will not fit in my case as _id is not numeric.

For pagination, I suggest to use Mongodb-aggregation. that's much more fast and flexible. like this: https://stackoverflow.com/questions/42111369/mongo-aggregation-with-paginated-data-and-totals/59247367 — Amin Shojaei, Dec 12 '19 at 10:29

score 3 · Accepted Answer · answered Dec 12 '19 at 11:15

I don't think you have much to do.

You have by default an index on _id and according to mongodoc, the field you index can be a whole embedded document whose (sub) fields order matters.

That is {mtid, uid} is your embedded document and as long as you query in the same (sub)field order you should be fine.

You would then just need to sort by _id.

generate some data:

arr = Array(25).fill(0).map((x,i)=>i).sort(x=>Math.random()-0.5).map(x=>[(x-x%5)/5,x%5])
arr.forEach(([mtid, uid],i)=>db.dums.insert({_id:{mtid, uid},n:i}))

in my case:

{ "_id" : { "mtid" : 2, "uid" : 4 }, "n" : 0 }
{ "_id" : { "mtid" : 2, "uid" : 2 }, "n" : 1 }
{ "_id" : { "mtid" : 0, "uid" : 1 }, "n" : 2 }
{ "_id" : { "mtid" : 0, "uid" : 0 }, "n" : 3 }
{ "_id" : { "mtid" : 0, "uid" : 2 }, "n" : 4 }
{ "_id" : { "mtid" : 2, "uid" : 3 }, "n" : 5 }
{ "_id" : { "mtid" : 3, "uid" : 1 }, "n" : 6 }
...

consider below query

function query(str){
  let [mtid, uid] = str.split('_')
  mtid = parseInt(mtid)
  uid = parseInt(uid)

  const res = db.dums.find({
    _id: {
      $gt: { mtid, uid }
    }
  }).limit(2).sort({ _id: 1 }).toArray()

  if(!res.length) return {items:[], next:null}
  const last = res[res.length-1]._id
  const next = last.mtid + '_' + last.uid
  print(res.map(x=>x._id.mtid+'_'+x._id.uid).join(';'))
  return {
    items: res,
    next,
    goNext(){return query(next)}
  }
}

results:

> var a=query('0_0');
0_1;0_2
> var a=a.goNext();
0_3;0_4
> var a=a.goNext();
1_0;1_1
> var a=a.goNext();
1_2;1_3
> var a=a.goNext();
1_4;2_0
> var a=a.goNext();
2_1;2_2

we can observe that the items are indeed ordered by mtid, uid

How to implement Cursor based pagination for mongodb in node.js using official mongodb client when _id is overrided?

1 Answers1