1

If what i understand is correct, processing background tasks is a good way to free the main thread of cpu bound tasks.

What i don't get is what is used by systems like bull or kue to run the tasks out of the main thread.

Do they use threads ? Do they require an entire node process fork? Do they spawn child processes?

Robert Brax
  • 6,508
  • 12
  • 40
  • 69

2 Answers2

1

Bull is based on Redis which handles in it's own process the handling and Queuing of data for these jobs. It is a lightweight, robust and fast job processing queue. It uses redis for persistence, so the queue is not lost if the server goes down for any reason.

the internal implementation of the Job can be seen here

The same is with Kue module which is a priority job queue backed by redis processes, built for node.js. The background tasks are powered by Redis.

This means that these module depends on Redis external process that enables different creating background jobs.

Job-specific events are fired on the Job instances via Redis pubsub:

  • enqueue the job is now queued
  • promotion the job is promoted from delayed state to queued
  • progress the job's progress ranging from 0-100
  • failed attempt the job has failed, but has remaining attempts yet
  • failed the job has failed and has no remaining attempts
  • complete the job has completed
  • remove the job has been removed

The delayed Jobs are powered by Redis Queue which notifies/triggers back the callbacks in the module.

Tal Avissar
  • 10,088
  • 6
  • 45
  • 70
  • That's very interesting i had no clue bg tasks could benefit from external non node process. And what are the capabilities of redis process? Is it multi process? Threaded? Is it single thread limited? – Robert Brax Apr 13 '16 at 05:13
  • Redis is an open source (BSD licensed), in-memory data structure store, used as database, cache and message broker. – Tal Avissar Apr 13 '16 at 05:15
  • Redis is single-threaded with epoll/kqueue and scales indefinitely in terms of I/O concurrency. --@antirez (creator of Redis). http://stackoverflow.com/questions/21304947/redis-performance-on-a-multi-core-cpu – Tal Avissar Apr 13 '16 at 05:21
  • @TalAvissar You seem to understand Kue very well. Would you mind taking a second to view my [question](http://stackoverflow.com/questions/37103639/how-to-schedule-a-job-with-kue-on-parse-server)? Thanks! – Baylor Mitchell May 09 '16 at 20:56
0

That's not how node.js works. Node.js internally uses an event loop to handle requests (and thus, making it an event driven framework)

This entire event loop is running in a single thread. When you execute a long running command (such as I/O or network operations) the request is enqueued to the loop and the process doesn't block. When the operation completes, it triggers a callback in your code

Event Loop