24

I'm trying to enqueue a basic job in redis using python-rq, But it throws this error

"ValueError: Functions from the main module cannot be processed by workers"

Here is my program:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

from rq import Connection, Queue
from redis import Redis
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job
user3258973
  • 241
  • 2
  • 4

2 Answers2

30

Break the provided code to two files: count_words.py:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

and main.py (where you'll import the required function):

from rq import Connection, Queue
from redis import Redis
from count_words import count_words_at_url # added import!
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

I always separate the tasks from the logic running those tasks to different files. It's just better organization. Also note that you can define a class of tasks and import/schedule tasks from that class instead of the (over-simplified) structure I suggest above. This should get you going.. Also see here to confirm you're not the first to struggle with this example. RQ is great once you get the hang of it.

GG_Python
  • 3,436
  • 5
  • 34
  • 46
  • 1
    Is there any way around this? I find it makes my project fragmented when, for example, I want one task to create multiple other tasks and submit them. Now my two tasks, though logically related, have to be split into multiple files so that one can import the other. I noticed that the code in rq's Job class checks for instances of string_types, so it's not clear why you can't submit using 'this_filename.some_other_func', but it does't work. – sheridp Feb 22 '18 at 22:37
  • @sheridp, post a new question with details about your problem, will try to help. – GG_Python Feb 22 '18 at 23:51
  • Thanks @GG_Python, I just created a [new question](https://stackoverflow.com/questions/49078556/is-there-a-way-to-submit-functions-from-main-using-python-rq) – sheridp Mar 02 '18 at 22:49
  • did that but the output is some # of None – Sihat Afnan Dec 21 '22 at 21:36
7

Currently there is a bug in RQ, which leads to this error. You will not be able to pass functions in enqueue from the same file without explicitly importing it.

Just add from app import count_words_at_url above the enqueue function:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

from rq import Connection, Queue
from redis import Redis
redis_conn = Redis()
q = Queue(connection=redis_conn)

from app import count_words_at_url
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

The other way is to have the functions in a separate file and import them.

Ishwar
  • 338
  • 2
  • 11