1

I have a solver doing some calculations,one takes 5-6 hours. I start it with subprocess.Popen(). I have written a class starting one calculation and managing the solution process (class is operating as long as calculation is in progress). I have 40 cores for the whole problem. I decided not to calculate them in series assigning all 40 cores for each, but asynchronously using AsyncIO, 4 tasks at the time assigning 10 coreas for each. After anyone of these 4 tasks gets solved I want to start another one out of the whole number of tasks, so each moment of time I want 4 tasks being calculated untill all of them done. I have moddeled the situation by a simple code.

Here I have a coroutine doing some work which is aimed to model calculation process. I create event loop assigning only 2 tasks. When both are finished the code finishes. But I want it not to finish but create another task with random (for example) argumen and add it to task list. I am too noobie to understand how...

import asyncio


async def counter(n):
    print(f'counter with argument {n} has been launched')
    for i in range(n):
        for j in range(n):
            for k in range(n):
                pass
    await asyncio.sleep(0)
    print(f'counter with argument {n} has FINISHED')


my_loop = asyncio.get_event_loop()
tasks = [my_loop.create_task(counter(100)), my_loop.create_task(counter(500))]
wait_tasks = asyncio.wait(tasks)
my_loop.run_until_complete(wait_tasks)
my_loop.close()
  • Does this answer your question? [how to add a coroutine to a running asyncio loop?](https://stackoverflow.com/questions/34499400/how-to-add-a-coroutine-to-a-running-asyncio-loop) – Ahmed AEK Nov 09 '22 at 07:58
  • Is that multiprocessing? –  Nov 09 '22 at 07:59
  • asyncio doesn't distribute work among cores, it only does everything in a single thread ... you should be using `multiprocessing` or `concurrent.futures.processpoolexecutor` ... but your question seems to be about both asyncio tasks and multiprocessing ... you might want to make that more clear ... like add an explain what portion of the code is distributed to 4 tasks and which is distributed to 10 cores. – Ahmed AEK Nov 09 '22 at 08:02
  • sorry, i didnt specified that. I assign numb of processes tp use for one task solving in bat file. When I Popen the bat file my OS launches calculation and assigns the specified proc numb in bat. So the code I am trying to write is just a kind of manager, managing one solution. But I want to create copies of this manager (operation in one core. like concurency) managing different tasks. –  Nov 09 '22 at 08:10
  • seems like you just want to run 4 tasks at the same time .... you can use the answer here to limit the number of tasks being executed, but as you are running an external bat file you should be using [asyncio subprocess](https://stackoverflow.com/a/63783037/15649230) also see [limit the number of tasks at the same time](https://stackoverflow.com/a/48486557/15649230) – Ahmed AEK Nov 09 '22 at 08:17
  • thanks a lot, i have found some usefull tips and now using Semafore. But here i have another issue arised: alright, say I have 10 tasks to be copleted with the condition that only 2 tasks can run simultaneously. Thats easy with semafore and ensure_future. But what if initially we had 10 tasks to do, started the main loop with semafore and during the main loop was running we have received extra 5 tasks to do? –  Nov 09 '22 at 11:36

0 Answers0