2

I am using fastapi with beanie as ODM and asyncio, 2 functions taking a lot of time so I decided to run these function in parallel, I tried thread but it's giving a function running in a different event loop error. I tried multiple solutions but they are not working. Some solutions suggested that threading should not be used with asyncio.

asyncio.to_thread(add_attack_shodan_results_to_scans_collection(cve_list_with_scan_id))
asyncio.to_thread(make_safe_dict_for_mongo_insertion_for_nmap(body.ip,inserted_id))

and other solutions are

asyncio.gather(
    add_attack_shodan_results_to_scans_collection(cve_list_with_scan_id),
    make_safe_dict_for_mongo_insertion_for_nmap(body.ip, inserted_id)
)

and

asyncio.run(add_attack_shodan_results_to_scans_collection(cve_list_with_scan_id))
asyncio.run(make_safe_dict_for_mongo_insertion_for_nmap(body.ip,inserted_id))

and

thrd1 = threading.Thread(target=make_thread_for_cve_calculation, args=(cve_list_with_scan_id))
thrd1.start()
thrd = threading.Thread(target=make_thread_for_nmap, args=(body.ip, inserted_id))
thrd.start()

and threading functions are

def make_thread_for_nmap(ip,scan_id):
    logger.info(f'thread id:{threading.current_thread().ident}')
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(make_safe_dict_for_mongo_insertion_for_nmap(ip,scan_id))
    loop.close()

def make_thread_for_cve_calculation(*calculate_cve):
    logger.info(f'thread id:{threading.current_thread().ident}')
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(add_attack_shodan_results_to_scans_collection(calculate_cve))
    loop.close()

What is the solution to run these function parallel or background? I also have checked the documentation but did not find any solution. Is there any way to run these function parallel without blocking other requests and what is the solution for "different event loop" error?

kggn
  • 73
  • 1
  • 8
lEgeL çOdēR
  • 21
  • 2
  • Why do you think you need to combine threads and async? – AKX Jul 30 '23 at 19:31
  • What are your functions doing ? Are they doing IO operations, or calculation ? It is very complicated to help you without knowing the functions, – filani Jul 30 '23 at 19:33
  • Use threading for synchronous functions that are not themselves doing asyncio. If your 2 functions are fully asyncio compliant, they are interleaving and there is no advantage to threads. Note that [`to_thread`](https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread) takes a function object and its parameters. In `asyncio.to_thread(add_attack_shodan_results_to_scans_collection(cve_list_with_scan_id))`, you called the function and then tried to "to_thread" its return value. Is that intended? – tdelaney Jul 30 '23 at 19:46
  • You really haven't explained your issue very well. But if you have long lived io-bound work, the normal solution is to use asyncio methods of access io and put the work in a task (i.e. asyncio.create_task). if these tasks are chewing up a lot of cpu themselves, then maybe they don't belong in the webserver, but hanging of a work queue like celery. – toppk Jul 30 '23 at 20:17
  • @filani my function is getting data from different resources like nmap and opencve and doing a little bit calculations. – lEgeL çOdēR Jul 30 '23 at 20:30
  • @AKX yes i want to run async functions as a thread – lEgeL çOdēR Jul 30 '23 at 20:32
  • @tdelaney these 2 functions are for getting data from api's and a little bit calculations, code is intended well in my IDE, i have tried asyncio.create_task() but it stuck on waiting for responce from api,that take a lot of time, and we not returning any value – lEgeL çOdēR Jul 30 '23 at 20:35
  • @toppk i have tried asyncio.create_task() but it start waiting for api responce, and at that time it no any other api work, whole system is stuck – lEgeL çOdēR Jul 30 '23 at 20:41
  • 2
    If you want to run them in threads, they likely shouldn't be `async` at all. – AKX Jul 31 '23 at 08:25
  • 1
    If you want two CPU-intensive calculations running in parallel, you don't want asyncio or threads, you want to use [multiprocessing](https://docs.python.org/3/library/multiprocessing.html). (C)Python still has a so called [Global Interpreter Lock](https://docs.python.org/3/glossary.html#term-global-interpreter-lock), which means that "threading" is not about parallel processing, it can only do parallel waiting (for I/O, and that's why it's not really beneficial to mix it with asyncio). – tevemadar Jul 31 '23 at 09:18
  • @lEgeLçOdēR when the entire system pauses it's one of two reasons, you are using sync io protocols or you are using lots of CPU for your work? Which of the two are you doing? Essentially, eaxmine all calls in the `asyncio.create_task` and decide which read/write to filesystem/database/network/etc, and are they using async protocols (i.e. using `await` to read/write). Like if I make a database call, via sync, I will block the entire webserver until I get a response, even though the database client is not doing any work. That is a simple fix, cpu bound needs a different solution entirely. – toppk Jul 31 '23 at 17:43
  • 1
    You might find [this answer](https://stackoverflow.com/a/71517830/17865804) and [this answer](https://stackoverflow.com/a/76148361/17865804) helpful – Chris Aug 01 '23 at 05:05

0 Answers0