If you want a "virtual populate" with a sort and limit then that's not actually how you do it. What you created is just a "virtual getter" which is actually returning the result of an async function. You can use that, but it's a lot trickier to manage resolving the returned Promise and it really has nothing to do with populate(), which is where you are raising an error.
There are also different options to doing this.
Mongoose Virtual Populate
For this which is closest to what you attempted, Instead you want something like this:
const dashboardSchema = new Schema({
userId: Schema.Types.ObjectId
},
{
toJSON: { virtuals: true },
toObject: { virtuals: true }
});
dashboardSchema.virtual('TopReports', {
ref: 'Report',
localField: 'userId',
foreignField: 'userId',
options: { sort: { date: -1 }, limit: 10 } // optional - could add with populate
});
const reportSchema = new Schema({
userId: Schema.Types.ObjectId,
seq: Number,
date: Date
});
const Dashboard = mongoose.model('Dashboard', dashboardSchema);
const Report = mongoose.model('Report', reportSchema);
This is actually almost identical to the documentation example on "Populate Virtuals" in that the given example there also includes the options which are the same as passing options to the populate method itself. When you set options in the virtual you only need invoke like this:
let result = await Dashboard.findOne().populate('TopReports');
The defaults set for sort and limit are automatically applied on this "virtual" field as the populate() is performed. If you chose NOT to include the options you would just add the options manually:
let result2 = await Dashboard.findOne().populate({
path: 'TopReports',
options: { sort: '-date', limit: 5 }
});
log(result2);
IMPORTANT - Setting the options within the virtual will ALWAYS OVERRIDE any options passed to populate() as shown above. If you want to use different options on different requests, then you call on the method as above rather than defining on the virtual method attached to the schema.
That's all you really need do. The definition of course includes localField and foreignField as well as the ref all so the populate() call knows where to get the data from and which fields to relate to. There is also an optional justOne which differentiates between singular and Array results, as well as a few other options.
MongoDB $lookup
The other option here is that MongoDB basically has the same functionality built in anyway, with the exception that this is single request as opposed to populate() which is actually multiple requests in order to return the data from separate collections:
let result = await Dashboard.aggregate([
{ "$lookup": {
"from": Report.collection.name,
"let": { "userId": "$userId" },
"pipeline": [
{ "$match": { "$expr": { "$eq": [ "$userId", "$$userId" ] } } },
{ "$sort": { "date": -1 } },
{ "$limit": 10 }
],
"as": "TopReports"
}}
]);
So that's one request with a single response as compared to the "many" find() requests actually issued by populate(). It's the same result and just used the "sub-pipeline" form of $lookup in order to apply the $sort and $limit to the returned array of related items.
With a little work, you can even inspect the schema definition ( including a defined virtual ) from the mongoose schema and construct the same $lookup statement. There is a basic demonstration of this "schema inspection" on Querying after populate in Mongoose.
So it depends on which suits your needs best. I suggest trying both and even bench-marking application performance.
As a full demonstration, here is an example listing. It inserts 20 things and just returns the most "recent" 10 results in the array:
const { Schema, Types: { ObjectId } } = mongoose = require('mongoose');
const uri = 'mongodb://localhost:27017/test';
const opts = { useNewUrlParser: true };
mongoose.set('useFindAndModify', false);
mongoose.set('useCreateIndex', true);
mongoose.set('debug', true);
const dashboardSchema = new Schema({
userId: Schema.Types.ObjectId
},
{
toJSON: { virtuals: true },
toObject: { virtuals: true }
});
dashboardSchema.virtual('TopReports', {
ref: 'Report',
localField: 'userId',
foreignField: 'userId',
options: { sort: { date: -1 }, limit: 10 } // optional - could add with populate
});
const reportSchema = new Schema({
userId: Schema.Types.ObjectId,
seq: Number,
date: Date
});
const Dashboard = mongoose.model('Dashboard', dashboardSchema);
const Report = mongoose.model('Report', reportSchema);
const log = data => console.log(JSON.stringify(data, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri, opts);
await Promise.all(
Object.entries(conn.models).map(([k,m]) => m.deleteMany())
);
// Insert some things
let { userId } = await Dashboard.create({ userId: new ObjectId() });
const oneDay = ( 1000 * 60 * 60 * 24 );
const baseDate = Date.now() - (oneDay * 30); // 30 days ago
await Report.insertMany(
[ ...Array(20)]
.map((e,i) => ({ userId, seq: i+1, date: baseDate + ( oneDay * i ) }))
);
// Virtual populate
let popresult = await Dashboard.findOne().populate('TopReports');
log(popresult);
// Aggregate $lookup
let result = await Dashboard.aggregate([
{ "$lookup": {
"from": Report.collection.name,
"let": { "userId": "$userId" },
"pipeline": [
{ "$match": { "$expr": { "$eq": [ "$userId", "$$userId" ] } } },
{ "$sort": { "date": -1 } },
{ "$limit": 10 }
],
"as": "TopReports"
}}
]);
log(result);
} catch (e) {
console.error(e)
} finally {
mongoose.disconnect()
}
})()
And the output:
Mongoose: dashboards.deleteMany({}, {})
Mongoose: reports.deleteMany({}, {})
Mongoose: dashboards.insertOne({ _id: ObjectId("5cce3d9e16302f32acb5c572"), userId: ObjectId("5cce3d9e16302f32acb5c571"), __v: 0 })
Mongoose: reports.insertMany([ { _id: 5cce3d9e16302f32acb5c573, userId: 5cce3d9e16302f32acb5c571, seq: 1, date: 2019-04-05T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c574, userId: 5cce3d9e16302f32acb5c571, seq: 2, date: 2019-04-06T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c575, userId: 5cce3d9e16302f32acb5c571, seq: 3, date: 2019-04-07T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c576, userId: 5cce3d9e16302f32acb5c571, seq: 4, date: 2019-04-08T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c577, userId: 5cce3d9e16302f32acb5c571, seq: 5, date: 2019-04-09T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c578, userId: 5cce3d9e16302f32acb5c571, seq: 6, date: 2019-04-10T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c579, userId: 5cce3d9e16302f32acb5c571, seq: 7, date: 2019-04-11T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c57a, userId: 5cce3d9e16302f32acb5c571, seq: 8, date: 2019-04-12T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c57b, userId: 5cce3d9e16302f32acb5c571, seq: 9, date: 2019-04-13T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c57c, userId: 5cce3d9e16302f32acb5c571, seq: 10, date: 2019-04-14T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c57d, userId: 5cce3d9e16302f32acb5c571, seq: 11, date: 2019-04-15T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c57e, userId: 5cce3d9e16302f32acb5c571, seq: 12, date: 2019-04-16T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c57f, userId: 5cce3d9e16302f32acb5c571, seq: 13, date: 2019-04-17T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c580, userId: 5cce3d9e16302f32acb5c571, seq: 14, date: 2019-04-18T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c581, userId: 5cce3d9e16302f32acb5c571, seq: 15, date: 2019-04-19T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c582, userId: 5cce3d9e16302f32acb5c571, seq: 16, date: 2019-04-20T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c583, userId: 5cce3d9e16302f32acb5c571, seq: 17, date: 2019-04-21T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c584, userId: 5cce3d9e16302f32acb5c571, seq: 18, date: 2019-04-22T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c585, userId: 5cce3d9e16302f32acb5c571, seq: 19, date: 2019-04-23T01:34:22.554Z, __v: 0 }, { _id: 5cce3d9e16302f32acb5c586, userId: 5cce3d9e16302f32acb5c571, seq: 20, date: 2019-04-24T01:34:22.554Z, __v: 0 } ], {})
Mongoose: dashboards.findOne({}, { projection: {} })
Mongoose: reports.find({ userId: { '$in': [ ObjectId("5cce3d9e16302f32acb5c571") ] } }, { sort: { date: -1 }, limit: 10, projection: {} })
{
"_id": "5cce3d9e16302f32acb5c572",
"userId": "5cce3d9e16302f32acb5c571",
"__v": 0,
"TopReports": [
{
"_id": "5cce3d9e16302f32acb5c586",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 20,
"date": "2019-04-24T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c585",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 19,
"date": "2019-04-23T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c584",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 18,
"date": "2019-04-22T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c583",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 17,
"date": "2019-04-21T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c582",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 16,
"date": "2019-04-20T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c581",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 15,
"date": "2019-04-19T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c580",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 14,
"date": "2019-04-18T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c57f",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 13,
"date": "2019-04-17T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c57e",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 12,
"date": "2019-04-16T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c57d",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 11,
"date": "2019-04-15T01:34:22.554Z",
"__v": 0
}
],
"id": "5cce3d9e16302f32acb5c572"
}
Mongoose: dashboards.aggregate([ { '$lookup': { from: 'reports', let: { userId: '$userId' }, pipeline: [ { '$match': { '$expr': { '$eq': [ '$userId', '$$userId' ] } } }, { '$sort': { date: -1 } }, { '$limit': 10 } ], as: 'TopReports' } } ], {})
[
{
"_id": "5cce3d9e16302f32acb5c572",
"userId": "5cce3d9e16302f32acb5c571",
"__v": 0,
"TopReports": [
{
"_id": "5cce3d9e16302f32acb5c586",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 20,
"date": "2019-04-24T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c585",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 19,
"date": "2019-04-23T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c584",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 18,
"date": "2019-04-22T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c583",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 17,
"date": "2019-04-21T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c582",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 16,
"date": "2019-04-20T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c581",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 15,
"date": "2019-04-19T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c580",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 14,
"date": "2019-04-18T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c57f",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 13,
"date": "2019-04-17T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c57e",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 12,
"date": "2019-04-16T01:34:22.554Z",
"__v": 0
},
{
"_id": "5cce3d9e16302f32acb5c57d",
"userId": "5cce3d9e16302f32acb5c571",
"seq": 11,
"date": "2019-04-15T01:34:22.554Z",
"__v": 0
}
]
}
]
The Wrong Way
Just to demonstrate what is wrong with the "getter" approach, here is an example listing showing actually resolving the returned Promise on each returned object:
const { Schema, Types: { ObjectId } } = mongoose = require('mongoose');
const uri = 'mongodb://localhost:27017/test';
const opts = { useNewUrlParser: true };
mongoose.set('useFindAndModify', false);
mongoose.set('useCreateIndex', true);
mongoose.set('debug', true);
const dashboardSchema = new Schema({
userId: Schema.Types.ObjectId
},
{
toJSON: { virtuals: true },
toObject: { virtuals: true }
});
dashboardSchema.virtual('TopReports').get(function() {
return Report.find({ userId: this.userId }).sort("-date").limit(10);
});
const reportSchema = new Schema({
userId: Schema.Types.ObjectId,
seq: Number,
date: Date
});
const Dashboard = mongoose.model('Dashboard', dashboardSchema);
const Report = mongoose.model('Report', reportSchema);
const log = data => console.log(JSON.stringify(data, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri, opts);
await Promise.all(
Object.entries(conn.models).map(([k,m]) => m.deleteMany())
);
// Insert some things
let { userId } = await Dashboard.create({ userId: new ObjectId() });
const oneDay = ( 1000 * 60 * 60 * 24 );
const baseDate = Date.now() - (oneDay * 30); // 30 days ago
await Report.insertMany(
[ ...Array(20)]
.map((e,i) => ({ userId, seq: i+1, date: baseDate + ( oneDay * i ) }))
);
// Mimic the virtual populate with the getter
let results = await Dashboard.find();
for ( let r of results ) {
let obj = { ...r.toObject() }; // copy the plain object data only
obj.TopReports = await r.TopReports; // Resolve the Promise
log(obj);
}
} catch (e) {
console.error(e)
} finally {
mongoose.disconnect()
}
})()
And output:
Mongoose: dashboards.deleteMany({}, {})
Mongoose: reports.deleteMany({}, {})
Mongoose: dashboards.insertOne({ _id: ObjectId("5cce45193134aa37e88c4114"), userId: ObjectId("5cce45193134aa37e88c4113"), __v: 0 })
Mongoose: reports.insertMany([ { _id: 5cce45193134aa37e88c4115, userId: 5cce45193134aa37e88c4113, seq: 1, date: 2019-04-05T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4116, userId: 5cce45193134aa37e88c4113, seq: 2, date: 2019-04-06T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4117, userId: 5cce45193134aa37e88c4113, seq: 3, date: 2019-04-07T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4118, userId: 5cce45193134aa37e88c4113, seq: 4, date: 2019-04-08T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4119, userId: 5cce45193134aa37e88c4113, seq: 5, date: 2019-04-09T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c411a, userId: 5cce45193134aa37e88c4113, seq: 6, date: 2019-04-10T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c411b, userId: 5cce45193134aa37e88c4113, seq: 7, date: 2019-04-11T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c411c, userId: 5cce45193134aa37e88c4113, seq: 8, date: 2019-04-12T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c411d, userId: 5cce45193134aa37e88c4113, seq: 9, date: 2019-04-13T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c411e, userId: 5cce45193134aa37e88c4113, seq: 10, date: 2019-04-14T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c411f, userId: 5cce45193134aa37e88c4113, seq: 11, date: 2019-04-15T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4120, userId: 5cce45193134aa37e88c4113, seq: 12, date: 2019-04-16T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4121, userId: 5cce45193134aa37e88c4113, seq: 13, date: 2019-04-17T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4122, userId: 5cce45193134aa37e88c4113, seq: 14, date: 2019-04-18T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4123, userId: 5cce45193134aa37e88c4113, seq: 15, date: 2019-04-19T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4124, userId: 5cce45193134aa37e88c4113, seq: 16, date: 2019-04-20T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4125, userId: 5cce45193134aa37e88c4113, seq: 17, date: 2019-04-21T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4126, userId: 5cce45193134aa37e88c4113, seq: 18, date: 2019-04-22T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4127, userId: 5cce45193134aa37e88c4113, seq: 19, date: 2019-04-23T02:06:17.518Z, __v: 0 }, { _id: 5cce45193134aa37e88c4128, userId: 5cce45193134aa37e88c4113, seq: 20, date: 2019-04-24T02:06:17.518Z, __v: 0 } ], {})
Mongoose: dashboards.find({}, { projection: {} })
Mongoose: reports.find({ userId: ObjectId("5cce45193134aa37e88c4113") }, { sort: { date: -1 }, limit: 10, projection: {} })
{
"_id": "5cce45193134aa37e88c4114",
"userId": "5cce45193134aa37e88c4113",
"__v": 0,
"TopReports": [
{
"_id": "5cce45193134aa37e88c4128",
"userId": "5cce45193134aa37e88c4113",
"seq": 20,
"date": "2019-04-24T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c4127",
"userId": "5cce45193134aa37e88c4113",
"seq": 19,
"date": "2019-04-23T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c4126",
"userId": "5cce45193134aa37e88c4113",
"seq": 18,
"date": "2019-04-22T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c4125",
"userId": "5cce45193134aa37e88c4113",
"seq": 17,
"date": "2019-04-21T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c4124",
"userId": "5cce45193134aa37e88c4113",
"seq": 16,
"date": "2019-04-20T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c4123",
"userId": "5cce45193134aa37e88c4113",
"seq": 15,
"date": "2019-04-19T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c4122",
"userId": "5cce45193134aa37e88c4113",
"seq": 14,
"date": "2019-04-18T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c4121",
"userId": "5cce45193134aa37e88c4113",
"seq": 13,
"date": "2019-04-17T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c4120",
"userId": "5cce45193134aa37e88c4113",
"seq": 12,
"date": "2019-04-16T02:06:17.518Z",
"__v": 0
},
{
"_id": "5cce45193134aa37e88c411f",
"userId": "5cce45193134aa37e88c4113",
"seq": 11,
"date": "2019-04-15T02:06:17.518Z",
"__v": 0
}
],
"id": "5cce45193134aa37e88c4114"
}
Of course the Promise returned by the getter needs to be resolved for each returned document. By contrast the populate() uses $in in order to return the matching entries for ALL results in the find() with a single request, and this would issue a new find() for every Dashboard document instead of a single find() based on ALL the userId values found in every Dashboard document in the result.
Basically as opposed to either populate() or $lookup you are essentially splitting out the "join" logic into parts of the control flow, where it really does not belong, and becomes difficult to manage as well as generating even more requests back to the server.
