Python multiprocessing with 5 arguments

Question

I currently have a function with 5 arguments that from a for loop it executes, from a list of countries and then it iterates from a list of sites (over 20). It performs some operations and then it runs from a bigger function

My code is as it follows:

sitelist= ['zzz', 'xxx', 'yyy']
for site in sitelist:
   startime, endtime = createtime(timer,site)
   excludedlistpaths = excludedlist(site)
   finallist = []
   bigfunc(site,startime,endtime,excludelistpaths)

Currently my list of sites is increasing so I'm looking into using multithreading/multiprocessing but I'm getting stuck as my function has multiple arguments that change every time. My first thoughts where to use something like:

 with ThreadPoolExecutor(max_workers=4) as executor:
      for site in sitelist:
               executor.submit(bigfunc, site,startime,endtime,excludelistpaths))

but it failed, also map seems to be the same in this case. Is there any way to create multiple processes/ threads to just concurrently do my function at the same time? Every instance of my function is totally separate from the others so the only thing I'm looking here is for speed.

"but it failed" isn't an adequate description of the problem. Please explain exactly what happened when you tried the code, and exactly how that is different from what is supposed to happen. If there were error messages, [include complete error messages](https://meta.stackoverflow.com/questions/359146/why-should-i-post-complete-errors-why-isnt-the-message-itself-enough), formatted as code. — Karl Knechtel, Apr 07 '21 at 10:46
What means by "my function has multiple arguments that change every time"? — Aaron, Apr 07 '21 at 10:46

score 0 · Answer 1 · answered Apr 09 '21 at 07:44

As your 5 arguments keep changing between executions, I would use a dictionary that is shared across all processes. That way, all processes always use the latest set of arguments.

This can be done using a shared dictionary from the multiprocess.manager in python. See this example: multiprocessing: How do I share a dict among multiple processes?

The dictionary gets locked, when the one process reads it. After the lock is released, another process can change the values again. Everything is handled by the manager.

Python multiprocessing with 5 arguments

1 Answers1