59

I'm wondering, is there any way to implement background tasks? Maybe with workers pool. Can you show me direction, I'm thinking about writing package for this?

StormTrooper
  • 1,731
  • 4
  • 23
  • 37
Leonid Bugaev
  • 1,048
  • 1
  • 9
  • 9

4 Answers4

108

2019 update

Before thinking about writing a package for anything, first look if there are existing packages that do what you need. In the Meteor world, this means looking on Atmosphere for "job/queue/task/worker management/scheduling" packages, then on npm for the same search terms. You also need to define your requirements more precisely:

  • do you want persistence, or would an in-memory solution work?
  • do you want to be able to distribute jobs to different machines?

Meteor-specific

  • job-collection - reliable (I used it in 2014 in production at a startup), but currently in maintenance mode. Lets you schedule persistent jobs to be run anywhere (servers, clients).
  • SteveJobs - actively maintained by Max Savin, the author of several powerful Meteor tools
  • littledata:synced-cron - "A simple cron system for Meteor. It supports syncronizing jobs between multiple processes."

Abandoned packages:

Npm packages

Meteor has been able to use npm packages directly for several years now, so this question amounts to finding job/worker/queue management packages on NPM. If you don't care about persistence:

  • Async "provides around 70 functions that include the usual 'functional' suspects (map, reduce, filter, each...) as well as some common patterns for asynchronous control flow (parallel, series, waterfall...)"
  • d3-queue - minimalistic, written by D3 author Mike Bostock

If you do want persistence, since Meteor uses MongoDB already, it may be advantageous to use a job scheduling package with persistence to MongoDb. The most powerful and popular seems to be Agenda, but unfortunately it hasn't been maintained in months, and it has a significant backlog of issues.

If you're willing to add a dependency backed by redis to your project, there are more choices:

Like MongoDB, Redis can also provide high-availability (via Redis Sentinel), and if you want to distribute jobs among multiple worker machines, you can point them all at the same Redis server.

Dan Dascalescu
  • 143,271
  • 52
  • 317
  • 404
  • 1
    This one looks pretty popular: https://atmospherejs.com/percolatestudio/synced-cron – Adam Monsen Oct 08 '14 at 04:47
  • 1
    This is a phenomenal list, thanks for keeping it up to date, a real service. – chmac Nov 27 '14 at 21:11
  • 2
    This package seams interesting: https://atmospherejs.com/differential/workers I didn't try it yet but I wanted to mention it to increase the options list – Humber Jan 11 '15 at 12:27
  • 1
    @DanDascalescu I'm yet to do a full write-up on this one - but you can use celeryproject.org (Distributed task queue built on Python & RabbitMQ) - there's a meteor package we're currently using - https://atmospherejs.com/3stack/celery (See also https://atmospherejs.com/3stack/celery-connect for an easier setup) – nathan-m Jan 30 '15 at 01:00
  • @chmac: It would be easier to keep it up-to-date [if SO editing worked properly](http://meta.stackoverflow.com/questions/326100/useful-edit-rejected-by-editors-not-familiar-with-the-topic). – Dan Dascalescu Jun 14 '16 at 13:51
  • 1
    Thanks for sharing such a complete list! – Juliomac Oct 13 '16 at 14:24
  • I'm curious about `Kue` being marked as *avoid* - the meteor package wrapper is deprecated, sure, but kue is still a great (non-meteor) option – zeroasterisk Jan 28 '17 at 20:24
  • 1
    @zeroasterisk: thanks for the prompt. I've updated the answer. – Dan Dascalescu Jan 29 '17 at 21:51
2

There is a package based on Cron jobs which can be used to schedule tasks on certain intervals or dates. Here is the package: https://atmosphere.meteor.com/package/cron

And if you happen to look into the source of that package, you'll notice they're simply using:

Meteor.setInterval( ... , delay );

So if you save your tasks in a database, then load them into intervals during startup, then you'll probably be on the right track.

miketucker
  • 1,074
  • 1
  • 15
  • 27
  • And there is a good possibility that those calls in the cron package can and should be changed to Meteor.setTimeout() calls. – Rob_vH May 08 '15 at 13:15
  • The only issue is these intervals would run on each server. My new package, Steve Jobs, takes care of this issue and other issues. See https://github.com/msavin/SteveJobs..meteor.jobs.scheduler.queue.background.tasks – Max Savin Jun 23 '18 at 11:14
2

If you are looking for something that is specific to Meteor, I am happy to share there is a new package called Steve Jobs. It makes running background jobs as easy as calling a Method.

It has all the standard features you would expect, such as running a job only once, retrying failed jobs, and so on. You can learn more about it on GitHub:

http://github.com/msavin/stevejobs

Max Savin
  • 152
  • 4
0

I'm guessing proper support is on their roadmap, but in the meantime I've managed to get it going in a hacky way via setInterval. See the cron-tick package.

Dan Dascalescu
  • 143,271
  • 52
  • 317
  • 404
Tom Coleman
  • 3,037
  • 18
  • 16