I'm using Celery and asyncio and I have some CPU bound function.
My issue is: When I'm running a single task, it takes X seconds. When I'm trying to run 10 tasks it takes approximately 10X seconds.
The current implementation looks similar to:
job = group(custom_func.s(param1, param2, param3, param4, param5) for param3, param4 in data)
job_result = job.apply_async()
job_result.get()
@celery_app.task
def custom_func(param1, param2, param3, param4, param5):
loop = asyncio.new_event_loop()
coro = doing_some_cpu_bound_things()
result = loop.run_until_complete(coro)
loop.close()
return result
After googling possible solutions, I've found this question: Celery parallel distributed task with multiprocessing
But I'm not sure if this is the root cause of my issue and how to use it properly in my case.
Any ideas about what I'm doing wrong?
