celery with redis: mixing workers with different concurrency

Question

I use the following scripts to distribute tasks to two workers on different nodes.

celery_call.py:

from celery_test import add
import time
results = []
for i in range(20):
    results.append(add.delay(i))
for result in results:
    timeStart = time.time()
    resultValue = result.get(timeout=10)
    timePassed = time.time() - timeStart
    print(timePassed, resultValue)

celery_test.py:

from celery import Celery
app = Celery('celery_test', backend='redis://ip', broker='redis://ip')

@app.task
def add(x):
    import time
    time.sleep(2)
    return x

I have two workers running on two different nodes. First node:

celery -A celery_test worker --concurrency 1 -l INFO

and second node:

celery -A celery_test worker --concurrency 10 -l INFO

The tasks are distributed and solved within few seconds. In 10 seconds on first node:

[2017-05-27 13:46:22,529: INFO/MainProcess] Received task: celery_test.add[2d9d592f-391f-4e1f-8dd2-5e50e5977c81]  
[2017-05-27 13:46:22,531: INFO/MainProcess] Received task: celery_test.add[e5c20ae9-92d7-4811-9b54-4efd3707bb5f]  
[2017-05-27 13:46:22,533: INFO/MainProcess] Received task: celery_test.add[bb9b50bc-1c5d-4ede-abfb-9591f53b3912]  
[2017-05-27 13:46:22,535: INFO/MainProcess] Received task: celery_test.add[b195fbb4-2683-461e-aee7-4715a1387eb6]  
[2017-05-27 13:46:22,537: INFO/MainProcess] Received task: celery_test.add[c8ce9d10-f03a-4585-9ff8-0d49cfd2e8b2]  
[2017-05-27 13:46:24,538: INFO/PoolWorker-1] Task celery_test.add[2d9d592f-391f-4e1f-8dd2-5e50e5977c81] succeeded in 2.0077112819999456s: 1
[2017-05-27 13:46:26,543: INFO/PoolWorker-1] Task celery_test.add[e5c20ae9-92d7-4811-9b54-4efd3707bb5f] succeeded in 2.0030374974012375s: 3
[2017-05-27 13:46:28,547: INFO/PoolWorker-1] Task celery_test.add[bb9b50bc-1c5d-4ede-abfb-9591f53b3912] succeeded in 2.0030434764921665s: 5
[2017-05-27 13:46:30,551: INFO/PoolWorker-1] Task celery_test.add[b195fbb4-2683-461e-aee7-4715a1387eb6] succeeded in 2.0029842611402273s: 7
[2017-05-27 13:46:32,555: INFO/PoolWorker-1] Task celery_test.add[c8ce9d10-f03a-4585-9ff8-0d49cfd2e8b2] succeeded in 2.0029691718518734s: 9

In 4 seconds on second node:

[2017-05-27 13:46:22,528: INFO/MainProcess] Received task: celery_test.add[402fb858-d9ec-4565-ab71-fbf4ec531787]  
[2017-05-27 13:46:22,530: INFO/MainProcess] Received task: celery_test.add[311fc7ed-e44a-4119-a0fa-c6849574723e]  
[2017-05-27 13:46:22,532: INFO/MainProcess] Received task: celery_test.add[af54e423-651b-4b01-a3d1-26ead5ae6af1]  
[2017-05-27 13:46:22,534: INFO/MainProcess] Received task: celery_test.add[29234f7f-f841-44c2-94fb-b491a074318b]  
[2017-05-27 13:46:22,537: INFO/MainProcess] Received task: celery_test.add[1a638710-810a-422d-8f2f-d554af3c5a92]  
[2017-05-27 13:46:22,538: INFO/MainProcess] Received task: celery_test.add[5b1a6863-1b62-4927-849b-e04e2d34ce7c]  
[2017-05-27 13:46:22,540: INFO/MainProcess] Received task: celery_test.add[f4cd393a-2f02-48dd-b27f-d73806e154da]  
[2017-05-27 13:46:22,543: INFO/MainProcess] Received task: celery_test.add[da8241bf-ce4e-4fe6-bd65-91350da8b163]  
[2017-05-27 13:46:22,544: INFO/MainProcess] Received task: celery_test.add[4892b31a-e488-4011-86e7-d55eb941cf1f]  
[2017-05-27 13:46:22,545: INFO/MainProcess] Received task: celery_test.add[c883b6ec-3842-4b50-bfab-35613f1724ed]  
[2017-05-27 13:46:22,548: INFO/MainProcess] Received task: celery_test.add[1b021f4c-b41d-46a8-8548-7016539e8a8b]  
[2017-05-27 13:46:22,549: INFO/MainProcess] Received task: celery_test.add[ae5d9b7d-0fa2-493b-aa0b-0b13b3764fdf]  
[2017-05-27 13:46:22,551: INFO/MainProcess] Received task: celery_test.add[af0d27fe-394f-4fbb-821e-95acbb99324c]  
[2017-05-27 13:46:22,552: INFO/MainProcess] Received task: celery_test.add[0b91cc9d-63c4-4a39-9d03-35ac474252bc]  
[2017-05-27 13:46:22,553: INFO/MainProcess] Received task: celery_test.add[1be5a881-064c-4cf8-8a18-10c2fe0400ed]  
[2017-05-27 13:46:24,538: INFO/PoolWorker-1] Task celery_test.add[402fb858-d9ec-4565-ab71-fbf4ec531787] succeeded in 2.0080740470439196s: 0
[2017-05-27 13:46:24,540: INFO/PoolWorker-3] Task celery_test.add[311fc7ed-e44a-4119-a0fa-c6849574723e] succeeded in 2.007692627608776s: 2
[2017-05-27 13:46:24,543: INFO/PoolWorker-6] Task celery_test.add[29234f7f-f841-44c2-94fb-b491a074318b] succeeded in 2.007782282307744s: 6
[2017-05-27 13:46:24,543: INFO/PoolWorker-5] Task celery_test.add[af54e423-651b-4b01-a3d1-26ead5ae6af1] succeeded in 2.007910629734397s: 4
[2017-05-27 13:46:24,546: INFO/PoolWorker-8] Task celery_test.add[1a638710-810a-422d-8f2f-d554af3c5a92] succeeded in 2.0075698774307966s: 8
[2017-05-27 13:46:24,548: INFO/PoolWorker-2] Task celery_test.add[f4cd393a-2f02-48dd-b27f-d73806e154da] succeeded in 2.0072230715304613s: 11
[2017-05-27 13:46:24,548: INFO/PoolWorker-10] Task celery_test.add[5b1a6863-1b62-4927-849b-e04e2d34ce7c] succeeded in 2.007256705313921s: 10
[2017-05-27 13:46:24,552: INFO/PoolWorker-7] Task celery_test.add[da8241bf-ce4e-4fe6-bd65-91350da8b163] succeeded in 2.0082139261066914s: 12
[2017-05-27 13:46:24,554: INFO/PoolWorker-9] Task celery_test.add[c883b6ec-3842-4b50-bfab-35613f1724ed] succeeded in 2.0077442210167646s: 14
[2017-05-27 13:46:24,554: INFO/PoolWorker-4] Task celery_test.add[4892b31a-e488-4011-86e7-d55eb941cf1f] succeeded in 2.007783567532897s: 13
[2017-05-27 13:46:26,542: INFO/PoolWorker-1] Task celery_test.add[1b021f4c-b41d-46a8-8548-7016539e8a8b] succeeded in 2.002950184047222s: 15
[2017-05-27 13:46:26,544: INFO/PoolWorker-3] Task celery_test.add[ae5d9b7d-0fa2-493b-aa0b-0b13b3764fdf] succeeded in 2.002891855314374s: 16
[2017-05-27 13:46:26,547: INFO/PoolWorker-5] Task celery_test.add[af0d27fe-394f-4fbb-821e-95acbb99324c] succeeded in 2.002899706363678s: 17
[2017-05-27 13:46:26,547: INFO/PoolWorker-6] Task celery_test.add[0b91cc9d-63c4-4a39-9d03-35ac474252bc] succeeded in 2.002899182960391s: 18
[2017-05-27 13:46:26,550: INFO/PoolWorker-8] Task celery_test.add[1be5a881-064c-4cf8-8a18-10c2fe0400ed] succeeded in 2.0029856264591217s: 19

However, the retrieval of the results is delayed and takes in total 20 seconds:

1.9911870956420898 0
0.0006098747253417969 1
0.0011210441589355469 2
2.003366231918335 3
0.0006439685821533203 4
2.0034918785095215 5
1.0012683868408203 6
1.00254487991333 7
1.001213788986206 8
1.002840518951416 9
1.0012362003326416 10
1.001204490661621 11
1.00126314163208 12
1.0012261867523193 13
1.0012249946594238 14
1.0012695789337158 15
1.0013458728790283 16
1.0013868808746338 17
1.0014445781707764 18
1.001399278640747 19

I have two questions:

How to setup celery for optimal distribution of jobs? E.g. a good solution would be to assign 1+1 tasks to the first node and 10+8 tasks to the second node.
IMHO the retrieval of results should take as long as the slowest worker, i.e. 10 seconds. Why does it take much longer? How to speed it up?
Turning off the first worker, the second worker needs 4 seconds for 10+10 tasks (ok), and retrieving the results takes 5 seconds. Why do I still loose a second?

score 0 · Accepted Answer · answered May 30 '17 at 22:12

Understanding celery task prefetching reduces the runtime to 4 or 5 seconds. Still sometimes 1 second too much, but 1. and 2. are solved.

Since it seems to be a different problem, I asked 3. again in a separate question: python celery - get() is delayed

celery with redis: mixing workers with different concurrency

1 Answers1