Mongoose/MongoDB perform where after populate

Question

I am currently facing a problem where I want to retrieve listings from MongoDB using mongoose which a user has not previously liked.

Listings table:

const listingSchema = new Schema({
   _id,
   ...etc
})
listingSchema.virtual('listingLikes',{
    ref:'ListingLikes',
    ...etc
})

Listing likes table:

const listingLikesSchema = new Schema({
   _id,
   listingId,
   userId
})

What I want to be able to is perform a query such as this one:

ListingModel.find().populate('listingLikes').where({'listingLikes.userId':{$ne:userId}}).limit(10).exec()

Basically obtaining all listings where the user has not liked. The following query works to achieve this:

Listing.find().where({_id:{$nin:[ ...All previously liked user listings in the listing likes model ) ]}})

However this is I believe is largely inefficient (have to load all users previously liked listings into memory before finding the listings).

How can I do this in a more efficient manner? ideally without modifying the current Schema.

Answer 1

If you want to bring in the userId to exclude as a variable, an aggregate() with a $lookup would likely work:

let loggedInUser = 'Eladian'
ListingModel.aggregate([
  {
    // $lookup is instead of .populate() but does roughly the same thing,
    // bringing in the data from the listingLikes table
    '$lookup': {
      'from': 'listingLikesModel', 
      'localField': 'listingLikes', 
      'foreignField': '_id', 
      'as': 'listingLikes'
    }
  }, {
    // $lookup puts its results in an array in your table. 
    // We flatten it using $unwind
    '$unwind': '$listingLikes'
  }, {
    // $match is equivalent to 'where' 
    '$match': {
      'listingLikes.userId': {
        '$ne': loggedInUser
      }
    }
  }, {
    // limit to 10 results
    '$limit': 10
  }
])

Alternatively, you could add a $pipeline to the $lookup, which I believe would be more efficient and allow you to bring in userId from ListingModel or from anywhere else it might be stored:

ListingModel.aggregate([
  {
    '$lookup': {
      'from': 'listingLikesModel',
      // below assumes there is a 'userId' field in ListingModel,
      // which we put in to a variable (listingUserId) so we can use it in
      // the pipeline
      'let': {'listingUserId': '$userId'}, 
      'pipeline': [
        {
          '$match': {
            '$expr': {
              '$ne': [ '$userId', '$$listingUserId']
            }
          }
        },
        { '$limit': 10 }
      ],
      // It puts the 10 posts in an array called 'notLikedList' in the ListingModel
      'as': 'notLikedList'
    }
  }
])

More info on $lookup pipelines here

Answer 2

W.r.t efficiency cursor (Which is a pointer to the result set of a query) can be used to extract data of the query results one by one into the the memory and filter:

Note: According to listingLikes schema the actual reference of Listings (by _id as listingId) is stored, would suggest to tune the listingSchema to directly define listingLikes as field by ref. And not have it as virtuals.

As the following solution is in accordance with above. One of the things to note is match cannot be used directly populating virtuals.

The idea here is to:

• Prepare a query to populate listingLikes for Listings with a match on userId
• Iterate on the cursor of this query and push docs where listingLikes [] or null is empty/null since those will the listings that user didn't like.

Since cursors implement AsyncIterator interface, allowing them to be used in for…await loops.

let requiredListingDocs = [];
//with cursor limit can be removed if it was there to bring only some documents in memory
const cursor = ListingModel.find({})
  .populate({
    path: "listingLikes",
    match: { userId: userId },
    options: { limit: 10 }
  })
  .cursor();
//will contain listingLikes [] empty/or null for Listing docs that user didn't like

//looping on cursor which returns promise
for await (const doc of cursor) {
    if (doc["listingLikes"] === null || doc["listingLikes"].length === 0) {
        requiredListingDocs.push(doc);
    }
  }
console.info("LISTING DATA::", requiredListingDocs);