Python running multiple independent pieces of code

Question

I am facing a small issue in my code. I have a main function that, given a certain condition arises, has to launch one or more different functions which deal with web scraping, in particular they use Selenium. The problem is that I would simply like to launch this web scraping "task", which is simply a python function, and not wait for it to terminate, rather let it go on independently from the rest of my code, so that I might be independently running 5 different instances of the same function, without waiting for them to terminate. Some pseudo code:

while True:
    condition = SomeComputation()
    if(condition):
        IndependentFunction( some_parameter )

Once IndependtFunction is called, I would like to not have to wait for it to end. I have looked at multiprocessing, but from what I understood I might not need such type of parallelisation. Thanks!

If you are network-bound, then pick multithreading. If cpu-bound, then pick multiprocessing. If you do not care about performance, I would pick multithreading to avoid dealing with pipes. There are lots of tutorials for both. — Razzle Shazl, Feb 25 '21 at 11:11
Might be useful: https://stackoverflow.com/a/2846697/2359945 — Razzle Shazl, Feb 25 '21 at 11:15
You may want to take a look at [async](https://realpython.com/async-io-python/) functions. — Jaime Etxebarria, Feb 25 '21 at 11:15
Thank you very much! Seems to be exactly what I was looking for! — Leo, Feb 25 '21 at 14:11

score 0 · Answer 1 · edited Feb 25 '21 at 12:20

0

If you're not depending on the output of that scraping then you could use threading

it would be like

mytask = threading.Thread(myfunction,args=(arg1,arg2,argn,))
mytask.start()

more details documentation: https://docs.python.org/3/library/threading.html

edited Feb 25 '21 at 12:20

DisappointedByUnaccountableMod

6,656
4
18
22

answered Feb 25 '21 at 11:14

Mohammed Ezzat

39
4

Thank you very much! Seems to be exactly what I was looking for! – Leo Feb 25 '21 at 14:10

score 0 · Accepted Answer · answered Feb 25 '21 at 11:16

0

You would need multithreading in order to do that. The basic usage of threading module on par with your independent function could be like this:

 import threading
 while True:
     condition = SomeComputation()
     if(condition):
         newThread = threading.Thread(target=IndependentFunction, args=(some_parameter,), daemon=True)
         newThread.start()

That daemon=True argument means that the thread will execute fully independently, and the main program will not wait for it to finish what it is doing before quitting the program entirely. Check this page for a more detailed tutorial.

answered Feb 25 '21 at 11:16

Nastor

638
4
15

1

Thank you very much! Seems to be exactly what I was looking for! – Leo Feb 25 '21 at 14:10
You can upvote my answer and/or accept is as default answer. Thanks. – Nastor Feb 25 '21 at 14:30

Python running multiple independent pieces of code

2 Answers2