Cancel previous MongoDB operation from the same client

Question

I have a MongoDB collection of 3257477 cities, and I'm using Mongoose on NodeJS to access it. I'm making requests to it repeatedly (once per 500ms). Requests are usually answered very quickly. However, when I make a bad typo the query takes a long time and requests start to pile up until the initial request is answered. Here are some logs I collected of requests and responses:

21:48:50    started query for "new"
21:48:50    finished query for "new"
21:48:52    started query for "newj ljl"  // blockage
21:48:54    started query for "newj"
21:48:55    started query for "new"
21:48:57    started query for "new ye"
21:48:59    started query for "new york"
21:49:08    finished query for "newj ljl" // blockage removed, quick queries flood in
21:49:08    finished query for "new"
21:49:08    finished query for "new york"
21:49:08    finished query for "new ye"
21:49:23    finished query for "newj"

I'm able to cancel the requests made by the client so I'm not worried about queries coming back in the wrong order. And I'm not interested in how to make that query faster at this point, since queries for actual correct spellings are quick.

I'm wondering how a new request can cancel an old request that was made by the same client. In other words "newj ljl" gets canceled when "newj" arrives, "newj" gets canceled when "new" arrives, and so on. If it's just going to be thrown out, why tie up the database?

Is there a proper way to do this?

Update:

I'm aware of db.currentOp().inprog and I'm thinking I can use the client property of the documents within that array to know whether it's a repeat request, but I can't quite figure out how to access that from Mongoose. I'm also not sure when to do that, or how I know which request was spawned from this client (and therefore which to cancel). I'd like an actual code example using Mongoose, or the native NodeJS MongoDB driver if possible!

Here's some sample code to go off of:

models.City.find({ ... })
    .exec(function (err, cities) {

    });

Answer 1

Below is what I came up with to solve the issue.

I can easily do db.currentOp().inprog and db.killOp() from the Mongo shell, but I really need this to happen automatically, when it needs to, from Mongoose. Since you can reference the MongoDB driver using require('mongoose').connection.db, you can execute those commands by doing "queries" on the following collections:

db.collection('$cmd.sys.inprog');
db.collection('$cmd.sys.killop');

The full solution:

var db = require('mongoose').connection.db,
    // get the client IP address
    ip = request.headers['x-forwarded-for'] || 
         request.connection.remoteAddress || 
         request.socket.remoteAddress ||
         request.connection.socket.remoteAddress;

// same thing as db.currentOp().inprog
db.collection('$cmd.sys.inprog').findOne(function (err, data) {
    if (err) throw err;

    data.inprog.filter(function (op) {
        // get the operation's client IP address without the port
        return ip == op.client.split(':')[0];
    }).forEach(function(op){
        // same thing as db.killOp()
        db.collection('$cmd.sys.killop')
          .findOne({ 'op': op.opid }, function (err, data) {
              if (err) throw err;
          });
    });

    // start the new cities query
    models.City.find({ ... })
        .exec(function (err, cities) {

        });
});

Helpful links:

• https://groups.google.com/forum/#!topic/mongodb-user/1wFp7AqWnM4
• drop database with mongoose
• How to determine a user's IP address in node

Answer 2

You can try using db.killOp()

http://docs.mongodb.org/manual/reference/method/db.killOp/#db.killOp

UPDATE: You can get the list of current operations from db.currentOp() and identify the operation to be cancelled by matching fields like op, query and client

http://docs.mongodb.org/manual/reference/method/db.currentOp/#db.currentOp

Answer 3

You can definitely do this with killop, and the above solution looks like it could work for the problem as stated. However, I think it may be worthwhile to dig a bit deeper.

The fact that you have a noticeably slow query when you've got a query that's going to return no results seems unusual. That reeks of a full collection scan. The questions to ask are, first, do you have indices set up, and second, are you querying with a general regex? MongoDB doesn't really handle regex searches like { "name" : /.*new york.*/ } particularly well.

Also, the whole "send an http request every time the user hits a key" approach is simple and elegant, but also causes some unnecessary server load. Perhaps a search button or a client-side timeout where you only send a request if a user hasn't hit a key for 1 second could help alleviate the need for the killop approach.