3

I'm using the great NodeJS Agenda task scheduler (https://github.com/agenda/agenda) to schedule background jobs in my Node/Express app.

Agenda can also run in a Node Cluster configuration, however I'm running into a problem with that (maybe I overlooked something because this seems pretty basic).

So I've used the code example from the README (https://github.com/agenda/agenda#spawning--forking-processes) to set up a cluster with N workers, each worker (Node cluster process) has an agenda instance.

Now suppose I have 2 workers (processes) and I call "agenda.now()" from worker 1, then it can be picked up (processed) by any of the 2 workers, because all of them are monitoring the queue, right?

However I always see the job being picked up by the first worker, the other one(s) aren't picking up the job.

What am I overlooking? All the workers should be monitoring the queue so all of them should be picking up jobs.

Kaneki21
  • 1,323
  • 2
  • 4
  • 22
leo
  • 1,175
  • 12
  • 13
  • 4
    P.S. I think I'm a step closer to the answer. Apparently Agenda will only use the second processor (worker) if the first one is "full" (completely "occupied") - this depends on how high you set the "defaultConcurrency" parameter. If I set "defaultConcurrency" to 1 (the standard value is 10 I think) then I do see the second worker being used. So, apparently, Agenda first "fills up" the first Node worker (agenda instance) and only after that starts using the second (and third, etc) worker. This SEEMS to be how it's working. – leo Aug 12 '17 at 14:46
  • thanks for clarifying that. I'm also looking on how to run agenda with node in cluster mode. I was scared having one agenda instance per process was not right, but it seems to be the right way. Am I right ? – marcvander Mar 04 '20 at 09:22

1 Answers1

1

setting the lockLimit would distribute the jobs evenly in case of cluster. It can be setup while instantiation of agenda

Takes a number which specifies the max number jobs that can be locked at any given moment. By default it is 0 for no max.

const agenda = new Agenda({lockLimit:3,
  db: { address: __mongodb_connection__ , collection: "jobCollectionName"} 
});
Kaneki21
  • 1,323
  • 2
  • 4
  • 22