I have a Flask application that schedules long running jobs to run using python-rq. One of my requirements is that the user can specify the number of jobs running at any given time.
The app doesn't need to kill any job in case the user use a smaller value than currently running jobs, but it needs to spawn another one in case the user increases the limit.
To run a job, the rq worker takes some time, but it doesn't need to babysit the job, it can safely run it and move on to the next one.
My issue is, sometimes the initial setup phase can take some time, so using only one worker might not be ideal. Another issue, which bugs me more, is that using this scheme my rq workers have to poll the database to know when they can go on and launch another job. Is there a better way to architect this?
