1

I am having issues getting RQ-python to run. Like in the example of the documentation (https://python-rq.org) I have a function in an external file

def createAndSaveIndex(url_list, index_path):
    print("--------------------------------------started task------------------------------------")
    index = indexFromURLList(url_list=url_list)
    index.save_local(index_path)
    return "Im done!"

which I import into my main file and use in the queue:

from redis import Redis
conn = Redis()
q = Queue(connection=conn)

job = q.enqueue(f=createAndSaveIndex,
                args=(["amazon.com"], dirname+"/myIndex/"))   # how long to hold onto the result

print(job.id)

The job is created and I get the job Id, however not even the print statement is executed and job.is_finished always returns false. I am on MacOs and have redis installed thorugh homebrew. I called redis-server and using it through the terminal works too? Does anyone have an idea what I could have done wrong?enter image description here

Tried the different examples I found online, none of which worked in my case. Tried using the function inside and outside the file.

Edit I forgot to check and mention the worker, turns out it is actually throwing an error:

File "/Users/-/micromamba/envs/flask/lib/python3.11/site-packages/rq/utils.py", line 107, in import_attribute
    return __builtins__[name]
           ~~~~~~~~~~~~^^^^^^
KeyError: 'backend_index.createAndSaveIndex'

as well as:

File "/Users/-/micromamba/envs/flask/lib/python3.11/site-packages/rq/utils.py", line 109, in import_attribute
    raise ValueError('Invalid attribute name: %s' % name)
ValueError: Invalid attribute name: backend_index.createAndSaveIndex

backend_index is the file where my function is located and createAndSaveIndex the name of the function.

Edit I first had both of my python files in the root directory I was told that the function must come from a module and not just a file, so now this is my project structure, however nothing changed for me:

project/
├── main.py
└── indexCreation/
    ├── __init__.py
    └── createIndex.py (was named backend_index)

This is the content of my init.py

from .createIndex import createAndSaveIndex

Solution After trying to recreate the error on another computer I finally got it to work: Turns it out the issue was the kind of terminal I used. When creating the redis-server and initializing the rq worker I used my macOs Terminal and to run the code I used the VSCode built in terminal (Which I thought was the same since I was using the same venv). Now If you do everything in the integrated vscode terminal it works with no issue. Thank you very much @รยקคгรђשค for helping me find the issue.

Actual solution When working with rq again I encountered the error once again, but for another reason. Also make sure when calling rq worker, that your working directory is also the one where your imported module is located.

sc0urge
  • 11
  • 4
  • `rq worker --with-scheduler` have you setup the worker part ? – รยקคгรђשค Jul 01 '23 at 07:08
  • https://python-rq.org/#the-worker – รยקคгรђשค Jul 01 '23 at 07:08
  • **Refer:** https://python-rq.org/docs/#considerations-for-jobs – รยקคгรђשค Jul 02 '23 at 03:07
  • @รยקคгรђשค First of all thank you for trying to help me. Having that said I still am stuck. In the page you sent me they said that theres 3 rules, which I understood mean the following: function should be available for import (so not in __main__), both should have access to the same file, and that the function does not depend on any global variables (which my code shouldnt). To test this I tried doing it with a simple function that takes two numbers and adds them (and waits 10s) but that threw the same error. Have I understood the considerations correctly or am I wrong? Thanks again. – sc0urge Jul 02 '23 at 14:26
  • can you provide all the files in the directory, what seems is the directory is not a module. Also can you update your main file, it seems to be incomplete or you might have shortened your code. If there is no issue in code, then what you might just be missing is `__init__.py` file to make the folder a valid python module. – รยקคгรђשค Jul 02 '23 at 16:21
  • @รยקคгรђשค For testing puposes, the main file right now is just the code that I provided above. I though that the file I import from should just be a python file in the same directory. I now changed it to how you said by creating a folder in the root directory, putting the file with the function inside of there and creating an empty file called __init__.py and putting from .nameOfFile import nameOfFunction, however this didnt change anything but the path in the error. – sc0urge Jul 02 '23 at 17:25
  • Thanks for updating the question, can you confirm from where we are launching the worker ? I will try reproducing today and confirm. – รยקคгรђשค Jul 03 '23 at 10:27
  • @รยקคгรђשค If you mean the rq worker default command I just launch it in my terminal, just like the redis-server command. The code with the creation of the reddis queue is located in main.py. I am using a conda environment for both starting my worker and running my python file. Thanks again! – sc0urge Jul 03 '23 at 11:02
  • @รยקคгรђשค Ive fixed the error (see edit). Thank you for helping me!!! – sc0urge Jul 03 '23 at 14:55
  • oh that's strange, ideally the broker should be the only shared dependency. If I remember correctly RQ library doesn't require the publisher process to have access to the function, rather only the consumer or the scheduler process to be able to access the function getting queued in. So essentially the worker could be on an entirely different server with same code base and it would have still worked if the broker (redis instance) was common. – รยקคгรђשค Jul 04 '23 at 03:37
  • @รยקคгรђשค I have no Idea either. I don't know if it even makes sense, but perhaps it has something to do with the safety features of apples terminal? – sc0urge Jul 05 '23 at 11:26

1 Answers1

0

Solution: After trying to recreate the error on another computer I finally got it to work: Turns it out the issue was the kind of terminal I used. When creating the redis-server and initializing the rq worker I used my macOs Terminal and to run the code I used the VSCode built in terminal (Which I thought was the same since I was using the same venv). Now If you do everything in the integrated vscode terminal it works with no issue. Thank you very much @รยקคгรђשค for helping me find the issue.

sc0urge
  • 11
  • 4