Background tasks not connected with any client in Meteor

Question

I would like to run some client-independent regular tasks in the background of a Meteor app (like scraping some pages). So they should not be inside any client thread, but once they finish, I would like to update all clients with information. What is the best way to achieve this?

Please note that there is no such thing as a "client thread" in Meteor, contrary to more traditional solutions. Client requests are processed asynchronously, not multi-threaded. — Kyll, Jul 31 '15 at 21:26
You are right. They are fibers, a cooperative multi-tasking. — Mitar, Jul 31 '15 at 21:35

score 6 · Answer 1 · edited Jan 24 '14 at 17:48

6

Run them on your server side code. If by regular you mean timed tasks every day or something:

You could use a cron job with Tom Coleman's cron package : https://github.com/tmeasday/meteor-cron.

You'll need to install the meteorite package manager first : npm install meteorite -g and then install the cron package in your project dir mrt add cron-tick

Server js

var MyCron = new Cron();

// this job will happen every day (60 seconds * 60 * 24)
MyCron.addJob(60*60*24, function() {
    //Scrape your stuff

    //Update your collections
});

As soon as you run your update/insert/edit they will be pushed to all clients.

edited Jan 24 '14 at 17:48

Dan Dascalescu

143,271
52
317
404

answered Mar 14 '13 at 08:39

Tarang

75,157
39
215
276

Is there a way to push something to clients which is not based on the collection (so not based on something in the database)? Like just information that something has been done? A notification? – Mitar Mar 15 '13 at 00:44
1

Its a bit tricker than that, javascript in node will in a single thread, so this is where Fibers comes in. If you use a Meteor.call in the cron to a meteor.method with your task containing `this.unblock` the content in the method from `this.unblock` would run in a new fiber so as not to block other clients. Regarding what other stuff meteor can send, I'd recommend the counts by room collection: http://stackoverflow.com/questions/10565654/how-does-the-messages-count-example-in-meteor-docs-work a good place to start, it depends what exactly you want to send down. – Tarang Mar 15 '13 at 01:28
But in which fiber does cron tasks run if you do not use `this.unblock`? – Mitar Mar 15 '13 at 05:04

score 5 · Accepted Answer · edited May 23 '17 at 12:00

5

To do this in a way that allows arbitrary external processes update the Meteor clients, use the DDP protocol that's associated with Meteor. Your server processes can write to the DDP channel, and when they do your clients will update. Have a look at this post for an example and a use case, which may be similar to yours:

Using node ddp-client to insert into a meteor collection from Node

The protocol is fairly straight forward, and the post shows an example of a node.js process writing to a Mongo collection that updates the clients in real time.

edited May 23 '17 at 12:00

Community

1
1

answered May 04 '13 at 13:06

mcauth

924
7
15

Ehm, you linked back to this question? – Mitar May 04 '13 at 19:45
Gah - sorry about that. Just fixed. – mcauth May 07 '13 at 01:19
An excellent solution. A couple things to call out: 1. To achieve periodic execution, set up a regular cron job which runs your program at whatever interval you'd like; 2. Your periodic program doesn't even have to be written in javascript, as long as the language has a DDP client library you're all set. – alanning Sep 02 '13 at 12:22

score 1 · Answer 3 · answered Apr 09 '13 at 09:04

1

You could try calling a Meteor.setInterval on the server (perhaps in Meteor.startup). That should work, though it might not be as flexible as the cron solution.

answered Apr 09 '13 at 09:04

benaiah

21
1

1

The idea is that I can have independent processes, even split them among multiple servers. So, really, heavy lifting on worker machines. – Mitar Apr 09 '13 at 22:39

score 0 · Answer 4 · answered Jul 31 '15 at 20:33

Go to http://atmospherejs.com and search for cron

The best one I found is percolate:synced-cron

Installation

meteor add percolate:synced-cron

Basics

SyncedCron.add({
  name: 'Crunch some important numbers for the marketing department',
  schedule: function(parser) {
    // parser is a later.parse object
    return parser.text('every 2 hours');
  },
  job: function() {
    var numbersCrunched = CrushSomeNumbers();
    return numbersCrunched;
  }
});

SyncedCron.start();

Advanced

See their documentation

Background tasks not connected with any client in Meteor

4 Answers4