RQ is a simple, lightweight, Python library for creating background jobs, and processing them.
RQ (Redis Queue) is a simple python library for queueing jobs and processing them in the background with workers. It is backed by redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.
Currently I'm working on python project that requires implement some background jobs (mostly for email sending and heavily database updates). I use Redis for task broker. So in this point I have two candidates: Celery and RQ. I had some experience…
We have a distributed architecture based on rabbitMQ and Celery.
We can launch in parallel multiple tasks without any issue. The scalability is good.
Now we need to control the task remotely: PAUSE, RESUME, CANCEL.
The only solution we found is to…
I'm trying to enqueue a basic job in redis using python-rq, But it throws this error
"ValueError: Functions from the main module cannot be processed by workers"
Here is my program:
import requests
def count_words_at_url(url):
resp =…
I'm using RQ, and I have a failed queue with thousands of items, and another test queue I created a while back for testing which is now empty and unused. I'm wondering how to remove all jobs from the failed queue, and delete the test queue…
I want to queue my ml predictions using rq. Example code (pesudo-ish):
predict.py:
import tensorflow as tf
def predict_stuff(foo):
model = tf.load_model()
result = model.predict(foo)
return result
app.py:
from rq import Queue
from…
We have recently forced to replace celery with RQ as it is simpler and celery was giving us too many problems. Now, we are not able to find a way to create multiple queues dynamically because we need to get multiple jobs done concurrently. So…
I'll start using django-rq in my project.
Django integration with RQ, a Redis based Python queuing library.
What is the best practice of testing django apps which is using RQ?
For example, if I want to test my app as a black box, after User makes…
How do I pass the result of a job to a job that depends on it?
What I currently do is passing id of the first job to the second,
first = queue.enqueue(firstJob)
second = queue.enqueue(secondJob, first.id, depends_on=first);
And inside secondJob…
Trying to use python-rq to support the back end to our Web application, but pushing new jobs takes very long - up to 12 seconds.
The performance hit happens when executing the enqueue_call function call, particularly when the number of worker…
I'm using the django-redis and django_rq frameworks to support both redis caching and redis background task processing for my Django application on Heroku. It's worked smoothly in the past, however now I keep getting a DatabaseError SSL error:…
I am using http://python-rq.org/ to queue and execute tasks on Heroku worker dynos. These are long-running tasks and occasionally I need to cancel them in mid-execution. How do I do that from Python?
from redis import Redis
from rq import Queue
from…
I'm using python-rq to manage Redis-based jobs and I want to determine which jobs are currently being processed by my workers.
python-rq offers a get_current_job function to find 'the current job' for a connection but:
I can't get this to work,…
I am trying to change the timeout on an rq job but nothing seems to work. I've got something to the effect of:
my_queue = Queue('my_task', connection=Redis())
job_args = (1, 2, 4)
my_queue.enqueue_call(
my_func,
…
I am trying python-rq and I do not see how to explicitely give priorities to the queues?
Does the priority come from the order they are defined when the worker is launched?
rqworker queueA queueB queueC
queueA is prioritized compare to queueB and…
While reading through the rq docs, I notice that there are some arguments that you can pass to rq worker when starting the worker
Example:
rq worker --worker-class 'foo.bar.MyWorker'
Argument list includes
--worker-class or -w: RQ Worker class to…
