MeteorJS: Save API call results for up to 24 hours

Question

I am working on a small app that consumes an XML data feed. I am wanting to take the XML data that comes back and save it in my MongoDB, so that I can then populate a template with the information (pagination, etc).

Getting the XML data is not the problem, I am just not sure how to expire information or even if this is the best option

Answer 1

You could add a timestamp to each document, and then periodically remove documents older than a day. In this example I'll assume your collection is called Feed.

When you insert a new document:

Feed.insert({data: '...', createdAt: new Date()});

Then sometime later you can run this on the server:

var cleanupFeed = function() {
  var now = new Date().valueOf();
  var yesterday = new Date(now - 86400*1000);
  Feed.remove({createdAt: {$lt: yesterday}});
};

An easy way to have cleanupFeed run periodically is just to use setInterval. For more ideas on task scheduling, have a look at this question.

Answer 2

Dennis,

You have 2 options in this case:

1. Application Based Expiration (Add a timestamp field and periodically query for those docs and remove them). I would generally recommend against this as it adds a lot of overhead to your application.
2. If you are using MongoDB 2.2 or newer, the best option would be to use a Time-To-Live (TTL) index

A TTL index runs a background task to delete expired documents every 60 seconds, so can effectively spread out the impact of deletions.

There are two ways to use a TTL expiry:

1. expire documents after a certain number of seconds
2. use a custom expiry time per document

The first approach is probably more appropriate for your use case. You would set a 24-hour expiry using something similar to:

db.api.results.ensureIndex( { "createdAt": 1 }, { expireAfterSeconds: 86400 } )