Stack Overflow
Stack Overflow

Questions tagged [task-queue]

With the Task Queue API, applications can perform work outside of a user request, initiated by a user request.

Tasks queues are an efficient and powerful tool for background processing; they allow your application to define tasks, add them to a queue, and then use the queue to process them in aggregate. You name queues and configure their properties in a configuration file named queue.yaml. A queue can be one of two types—push or pull—based on your needs.

Push queues function only within the App Engine environment. These queues are the best choice for applications whose tasks work only with App Engine tools and services. With push queues, you simply configure a queue and add tasks to it. App Engine handles the rest. Push queues are easier to implement, but are restricted to use within App Engine. For more information about push queues and examples of how to use them, see Using Push Queues.

In summary, push queues allow you to process tasks within App Engine at a steady rate and App Engine scales computing resources according to the number of tasks in your queue.

757 questions
131
votes
6 answers

Background processes in Node.js

What is a good aproach to handle background processes in a NodeJS application? Scenario: After a user posts something to an app I want to crunch the data, request additional data from external resources, etc. All of this is quite time consuming, so…
Ole Spaarmann
  • 15,845
  • 27
  • 98
  • 160
64
votes
3 answers

Is there a performance difference between pooling connections or channels in rabbitmq?

I'm a newbie with Rabbitmq(and programming) so sorry in advance if this is obvious. I am creating a pool to share between threads that are working on a queue but I'm not sure if I should use connections or channels in the pool. I know I need…
Lostsoul
  • 25,013
  • 48
  • 144
  • 239
50
votes
9 answers

Parallel processing in PHP - How do you do it?

I am currently trying to implement a job queue in php. The queue will then be processed as a batch job and should be able to process some jobs in parallel. I already did some research and found several ways to implement it, but I am not really aware…
enricog
  • 4,226
  • 5
  • 35
  • 54
36
votes
1 answer

What is TombstonedTaskError from App Engine's Task Queue?

What does the TombstonedTaskError mean? It is being raised while trying to add a task to the queue, from a cron-job: Traceback (most recent call last): File "/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 501, in…
dbr
  • 165,801
  • 69
  • 278
  • 343
26
votes
2 answers

RabbitMQ: What Does Celery Offer That Pika Doesn't?

I've been working on getting some distributed tasks working via RabbitMQ. I spent some time trying to get Celery to do what I wanted and couldn't make it work. Then I tried using Pika and things just worked, flawlessly, and within minutes. Is there…
Jason Champion
  • 2,670
  • 4
  • 35
  • 55
24
votes
1 answer

What is meant by 'bucket-size' of queue in the google app engine?

Google app engine task queues have configuration as (example) mail-queue 5/m 10 Here, what does the 'bucket-size' mean? I could not find a comprehensive…
Gopi
  • 10,073
  • 4
  • 31
  • 45
24
votes
1 answer

Push vs Pull Queues on Google App Engine

So far, no one (not even the GAE docs) has been able to give me a really clear description of what the difference is between a push queue and a pull queue. My understanding is that the Task Queue API allows you to define task queues where work can…
IAmYourFaja
  • 55,468
  • 181
  • 466
  • 756
23
votes
2 answers

how to use q.js promises to work with multiple asynchronous operations

Note: This question is also cross-posted in Q.js mailing list over here. i had a situation with multiple asynchronous operations and the answer I accepted pointed out that using Promises using a library such as q.js would be more beneficial. I am…
Kim Stacks
  • 10,202
  • 35
  • 151
  • 282
19
votes
3 answers

Return data from Laravel Jobs

TL;DR: How can I return data from a queued Job without saving it anywhere and handling the situation when the job might run more than once because of retries? Or is there another way if Jobs are not suitable for this? I am developing API on Laravel…
Bushikot
  • 783
  • 3
  • 10
  • 26
19
votes
1 answer

Dynamic queue creation with RabbitMQ

I've been learning RabbitMQ various topologies, however, I couldn't find any reference to dynamic queue creation (aka Declare Queue) emitted from a producer. The idea would be to create queues dynamically depending on a particular event (e.g a HTTP…
3XX0
  • 1,315
  • 1
  • 13
  • 25
18
votes
4 answers

GAE: unit testing taskqueue with testbed

I'm using testbed to unit test my google app engine app, and my app uses a taskqueue. When I submit a task to a taskqueue during a unit test, it appears that the task is in the queue, but the task does not execute. How do I get the task to execute…
new name
  • 15,861
  • 19
  • 68
  • 114
18
votes
1 answer

Running out of PermGen space when using DeferredTasks in GAE 1.7.3

I switched to Google App Engine Java SDK 1.7.3 recently. Since then, I am running out of PermGen space every time I am submitting DeferredTasks into the task queue. This does not happen when the app is deployed to App Engine. It only happens…
Ingo
  • 1,552
  • 10
  • 31
17
votes
5 answers

Throttle JavaScript function calls, but with queuing (don't discard calls)

How can a function rate-limit its calls? The calls should not be discarded if too frequent, but rather be queued up and spaced out in time, X milliseconds apart. I've looked at throttle and debounce, but they discard calls instead of queuing them up…
Dan Dascalescu
  • 143,271
  • 52
  • 317
  • 404
17
votes
1 answer

In celery, what would be the purpose of having multiple workers process the same queue?

In the documentation for celeryd-multi, we find this example: # Advanced example starting 10 workers in the background: # * Three of the workers processes the images and video queue # * Two of the workers processes the data queue with loglevel…
chaimp
  • 16,897
  • 16
  • 53
  • 86
16
votes
4 answers

Where do I register an rq-scheduler job in a Django app?

I'd like to use django_rq and rq-scheduler for offline tasks, but I'm unsure of where to call rq-scheduler's ability to schedule repeating tasks. Right now, I've added my scheduling to a tasks.py module in my app, and import that in __init__.py.…
Carson
  • 17,073
  • 19
  • 66
  • 87
1
2 3
…
50 51