Can't fetching all documents from my mongodb on Azure Cosmos DB

Question

I am trying to get ALL documents from a collection in my Cosmos DB I have on Azure. The collection contains approx 50.000 documents.

I get this error: MongoError: cursor does not exist, was killed or timed out when I am doing this:

const mongoose = require('mongoose');
const mongooseOptions = { useNewUrlParser: true };
mongoose.connect(connectionString, mongooseOptions);
mongoose.set('useCreateIndex', true);
mongoose.Promise = global.Promise;
const mongoDB = mongoose.connection;
mongoDB.on('error', console.error.bind(console, 'MongoDB connection error:'));

const Schema = mongoose.Schema;
const MongoEidModelSchema = new Schema({
    uid: { type: String, unique: true },
    eid: { type: String, unique: true }
});

const MongoEidModel = mongoose.model('eids', MongoEidModelSchema);
MongoEidModel.find({}, {timeout: false}).then(data => {
    console.log(data);
    console.log(Object.keys(data).length);
});

When I set a limit of 1000 or 1500 on the find() it works.

I have also tested to change the RU/s on the collection from 400 to 10.000 (in the Azure Portal / console) which also works, but that seems like an expensive solution... doesn't it?

I have also tested to fetch this with find() in batches in a recursive loop until there is no more documents left, with a sleep between each iteration (otherwise Cosmos DB gives me "429: Too many requests" after a while.

Is there a way in which I can get ALL the 50.000 documents using Node.js and Mongoose without changing RU/s or doing recursive loops?

Thanks in advance!

/Daniel

score 0 · Accepted Answer · answered Sep 03 '18 at 11:38

To avoid confusion, I assume you're using the MongoDB driver to access Cosmos in Azure?

For MongoDB, there is a query limit of 16Mb (which you may well be shooting past if you are returning 50k documents). See here: https://docs.mongodb.com/manual/reference/limits/

It is possible that the limitation isn't enforced in the node driver (I haven't inspected its source), in which case it's worth consulting the Azure docs: https://learn.microsoft.com/en-us/azure/cosmos-db/faq

The upshot is, you should really use a cursor to walk across the collection when you are dealing with large numbers of documents like this. See here: How can I use a cursor.forEach() in MongoDB using Node.js?

Hope this helps :)

Thanks for your help! I eventually solved how to use a cursor and I iterated over it with a stream. But even though I did so, I sometimes get error from mongodb. Sometimes this works, and sometimes it doesn't. Fortunately this was only for a maintenance job to be run once or twice locally, but still... :( — Daniel_swe, Sep 05 '18 at 12:43

Can't fetching all documents from my mongodb on Azure Cosmos DB

1 Answers1