Stack Overflow
Stack Overflow

Questions tagged [job-queue]

In system software, a job queue (sometimes batch queue), is a data structure maintained by job scheduler software containing jobs to run.

A job queue is a queue (that is, a First In First Out (FIFO) collection) of tasks to be carried out at an unspecified time in the future. Workers may take on the task at the head of the queue and carry it out, returning to take another when they are finished.

This is often applied to multi-threaded programming environments, in which there are fewer threads than available tasks so tasks must be queued until a thread is available to process them.

113 questions
45
votes
8 answers

What's the best way of implementing a messaging queue table in mysql

It's probably the tenth time I'm implementing something like this, and I've never been 100% happy about solutions I came up with. The reason using mysql table instead of a "proper" messaging system is attractive is primarily because most application…
taw
  • 18,110
  • 15
  • 57
  • 76
36
votes
6 answers

What is the difference between "event loop queue" and "job queue"?

I can not understand how the following code run. Why "1" is after "b" but "h" is after "3"? Should'n the order be: a, b, 1, 2, h, 3? Some articles said that the difference between "event loop queue" and "job queue" leads to the following output. But…
Sam Yang
  • 559
  • 2
  • 6
  • 10
25
votes
6 answers

The best way to use a DB table as a job queue (a.k.a batch queue or message queue)

I have a databases table with ~50K rows in it, each row represents a job that need to be done. I have a program that extracts a job from the DB, does the job and puts the result back in the db. (this system is running right now) Now I want to allow…
BCS
  • 75,627
  • 68
  • 187
  • 294
25
votes
8 answers

Stats/Monitor/Inspector for beanstalkd

Does anyone know of an app that can monitor a beanstalkd queue? I'm looking for something that shows stats on tubes and jobs, and allows you to inspect the details. I'm not really picky about language/platform, just want to know if there's…
Tim Lytle
  • 17,549
  • 10
  • 60
  • 91
20
votes
1 answer

Why does celery need a message broker?

As celery is a job queue/task queue, name illustrates that it can maintain its tasks and process them. Then why does it need a message broker like rabbitmq or redis?
Shailendra
  • 528
  • 2
  • 10
  • 21
20
votes
2 answers

What is a good Sidekiq-like job system for node.js?

(Most of the questions asked here about this subject are a bit old, and I was wondering what had change in the Node ecosystem it all those years.) I'm basically looking to implement a job queue in an app of mine in node.js. I've heard about and seen…
Guillaume Flandre
  • 8,936
  • 8
  • 46
  • 54
19
votes
9 answers

Job queue with job affinity

I am currently facing a problem for which I am pretty sure there is an official name, but I don't know what to search the web for. I hope that if I describe the problem and the solution I have in mind, somebody is able to tell me the name of the…
Golo Roden
  • 140,679
  • 96
  • 298
  • 425
14
votes
9 answers

job queue implementation for python

Do you know/use any distributed job queue for python? Can you share links or tools
user90150
14
votes
0 answers

Resque vs. Sidekiq

I'm working on a project with a friend where we've been using Resque for processing various commands from data input inside our rails application on a minute to minute basis. We've been messing around with the idea of using Sidekiq because it is…
James Newton
  • 402
  • 4
  • 14
7
votes
3 answers

How to use Jobqueue in Python-telegram-bot

I have able to make a bot very easily by reading the docs but Jobqueue is not working as per it is written. The run_daily method uses a datetime.time object to send the message at a particular time but this code neither does its job of sending a…
Rajas Rasam
  • 175
  • 2
  • 3
  • 9
7
votes
3 answers

Stack size becomes negative after instruction

It has been pointed out that the reason I am having this problem is because of proguard 4.9, so I went ahead and updated to proguard-5.2.1. But I am still having the same problem. Has anyone found a solution? I confirmed the new version through …
learner
  • 11,490
  • 26
  • 97
  • 169
7
votes
3 answers

How should I best structure my web application using job queues [and Perl/Catalyst]?

I'm writing a web application using the Catalyst framework. I'm also using a Job Queue called TheSchwartz. I'm wanting to use a job queue because I'm wanting as much of the application specific code decoupled from the web application interface…
Adam Taylor
  • 7,534
  • 8
  • 44
  • 54
7
votes
6 answers

Queueing solutions for ASP.NET MVC

I looking into the concept of queueing for web apps (i.e. putting some types of job in a queue for completion by a seperate worker, rather than being completed in the web request cycle). I would like to know if there are any good solutions existing…
UpTheCreek
  • 31,444
  • 34
  • 152
  • 221
7
votes
6 answers

What's the best way to build a queue for long-running jobs in a Grails app?

I have a Grails app that has some computationally intensive optimizations with a running time of ~5 minutes (perhaps more). Currently, I'm doing these in the main request thread, i.e. it takes 5 minutes for the request to return. It works, but is of…
Michael Borgwardt
  • 342,105
  • 78
  • 482
  • 720
6
votes
2 answers

Method for self-rearranging job queue

I have a job queue (using Amazon SQS) which hands off jobs to many machines for fetching and processing various documents over HTTP. There are hundreds of different hosts which are accessed, and there is no predictable order for the jobs. In order…
friedo
  • 65,762
  • 16
  • 114
  • 184
1
2 3 4 5 6 7 8