Mongodb find() using Node not returning all documents (strange behavior)

Question

I'm using Meteor with Node to retrieve a list of vehicles from a MongoDB collection, hosted on mLab. I recently noticed that find() in my js app was not returning all the matching documents in the collection. Using Mongo shell and the search on mLab both return the correct number of results.

Here's a sample document from the collection that isn't returned when it should be:

{
    "_id": "VIN",
    "updatedOn": "Fri Aug 11 2017 11:27:40 GMT-0400 (EDT)",
    "clientId": "1001",
    "crushVersion": "v.3.42",
    "yardName": "YARD NAME",
    "yardCity": "CITY",
    "yardState": "STATE",
    "stockNumber": "STK123447",
    "iStatus": "0",
    "location": "YARD",
    "year": "2003",
    "make": "AUDI",
    "model": "A6",
    "vehicleRow": "32",
    "yardDate": "2017-08-10T18:09:38.363",
    "batchNumber": "STK123447",
    "lastUpdate": "08/11/2017 01:31:31 AM",
    "color": "SILVER",
    "vin": "VIN",
    "reference": "",
    "milage": "24"
}

...and one that's returned as expected:

{
    "_id": "VIN",
    "updatedOn": "Fri Aug 11 2017 11:27:40 GMT-0400 (EDT)",
    "clientId": "1112",
    "crushVersion": "v.3.42",
    "yardName": "YARD NAME",
    "yardCity": "CITY",
    "yardState": "STATE",
    "stockNumber": "STK02752",
    "iStatus": "0",
    "location": "YARD",
    "year": "2003",
    "make": "AUDI",
    "model": "A6",
    "vehicleRow": "600",
    "yardDate": "2017-07-20T10:28:54.407",
    "batchNumber": "STK02752",
    "lastUpdate": "08/11/2017 08:30:30 AM",
    "color": "GRAY",
    "vin": "VIN",
    "reference": "",
    "milage": "1"
}

...and the js code for 'find';

MongoClient.connect(
    url,
    Meteor.bindEnvironment((err, db) => {
        if (err) throw new Meteor.Error(err)
        const inv = db.collection('inventory'),
            userId = user._id,
            picks = _picks, 
            /* _picks comes from a user-populated form as an array of objects. */
            makes = picks.map(pick => {
                return pick.make.toUpperCase()
            }),
            models = picks.map(pick => {
                return pick.model.toUpperCase()
            }),
            yards = user.profile.yards.map(yard => {
                return yard.yard
            })
        if (picks.length > 0) {
            inv
                .find({
                    $or: user.profile.picks.map(p => ({
                        year: { $gte: p.minYear, $lte: p.maxYear }
                    })),
                    make: { $in: makes },
                    model: { $in: models },
                    yardName: { $in: yards }
                })
                .toArray(
                    Meteor.bindEnvironment((err, docs) => {
                        if (err) throw new Meteor.Error(err)

                        console.log(docs)
                    })
                )
        }
    })
)

Now here's the weird part. All the documents returned have a yardDate of 2017-7-31 or older. This may be a coincidence, but I don't think so. My collection has around 21000 documents and all queries exhibit the same behavior.

I looked at this question Link, but some of my queries only have one or two results, so I don't think it's a buffer overflow issue.

Thanks for any help you can give!

Update: I made a little test file to see if there's something up the chain from my find() that may be messing with the query. Here's the code for that:

const MongoClient = require('mongodb').MongoClient,
    url = 'mongodb://user:password@ipaddress/dbname'

MongoClient.connect(url, (err, db) => {
    const inv = db.collection('inventory')

    inv.find({ year: '2003', model: 'A6' }).toArray((err, docs) => {
        if (err) throw new Error(err)
        console.log(docs)
        db.close()
    })
})

This should return five documents, but only returns two... all with a yardDate older than 2017-07-31.

Update:

I added a 'missing' result and a 'found' result.

It sounds like there is some kind of data problem. Have you tried exporting the data and reimporting it to a new database? Maybe the indexes are corrupted somehow. OR... Your example shows the year is a string, do the missing records have numbers in the year field? — Mikkel, Aug 11 '17 at 16:40
I like both ideas, but nope. I dropped the collection and created it again with the same result. In each document, the properties are all strings, whether they contain numbers or not. Thanks, though! — RationalExhuberance, Aug 11 '17 at 17:02
I just added examples of two documents, one that isn't returned and one that is. — RationalExhuberance, Aug 11 '17 at 17:07

score 0 · Answer 1 · answered Aug 11 '17 at 22:13

0

Turns out... when I moved my database to mLab, I didn't find all the urls in the app and change them. The original database was still talking, so I was getting the 'old' results.

answered Aug 11 '17 at 22:13

RationalExhuberance

121
8

It happens to the best of us :) At least you had enough guts to own up to the mistake. Put it down to experience - that one won't happen to you again, will it? – Mikkel Aug 16 '17 at 14:23

Mongodb find() using Node not returning all documents (strange behavior)

1 Answers1