How can I submit another task_list after one task_list finished in asyncio module?

Question

I'm new to asyncio module， pls pardon me for my bad trial:

In my codes, I try to submit 100 tasks at first, then submit next 100 tasks after fore-100 tasks finished, then another next 100 tasks.

What should I do to make it work?

# len(ad_accounts) = 1000 for example 
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
task_list = [
            asyncio.ensure_future(_handle_account(account)) for ad_account in ad_accounts[:100]
        ]
#just submit 100 tasks here once
new_loop.run_until_complete(asyncio.wait(task_list))
# then can I continuely submit next 100 tasks?

You are doing correctly, but should use `get_event_loop` instead of new and set. What's the problem you have? — MT-FreeHK, Mar 25 '19 at 05:46
@MatrixTai , thanks for replying, I just want to run part of 1000 tasks (100 for example) at once , and run next 100 tasks after fore 100 tasks finished.. — jia Jimmy, Mar 25 '19 at 06:22
@MatrixTai Is this works ? `loop.run_until_complete(asyncio.wait(task_list_100)) loop.run_until_complete(asyncio.wait(task_list_200)) loop.run_until_complete(asyncio.wait(task_list_300))...` — jia Jimmy, Mar 25 '19 at 06:23

score 1 · Accepted Answer · answered Mar 25 '19 at 07:52

Few things to add,

get_event_loop will try to access any available event loop, if there is not, it will call new_event_loop with set_event_loop, [Doc].
Simply use asyncio.gather when you won't do any further actions individual task. Use gather provides you a way to stop whole task group as well, see this SO answer.
You may notice in the following, () generator is used instead of [] list comprehension, as you don't need the list actually be saved in memory, and you don't need it until iteration. Change back to [] bracket if it is not the case.

Whole working example will be:

# len(ad_accounts) = 1000 for example

chunk_size = 100
batched_tasks = (ad_accounts[i:i + chunk_size] for i in range(0, len(ad_accounts), chunk_size))
_loop = asyncio.get_event_loop()

for task_group in batched_tasks:
    task_list = [
        asyncio.ensure_future(_handle_account(account)) for ad_account in task_group
    ]
    #just submit 100 tasks here once
    _loop.run_until_complete(asyncio.gather(*task_list))
    # Or _loop.run_until_complete(asyncio.wait(task_list))

thanks for your patience, benefit a lot from your words. :) – jia Jimmy Mar 25 '19 at 09:10 — jia Jimmy, Mar 25 '19 at 09:10

How can I submit another task_list after one task_list finished in asyncio module?

1 Answers1