1

Let's presume I call a function for some input list and I get an output list. I want to do the same for 2 lists at a time (using 2 processes), wait for both to complete and then call another function at the end with both outputs (not combined). How can I achieve this with mpi4py? I looked into gather and reduce but I'm not sure..

Mikael Ken
  • 41
  • 1
  • 10
  • 1
    Probably this will answer your question: https://stackoverflow.com/questions/11968689/python-multithreading-wait-till-all-threads-finished – Aaron Dec 19 '19 at 07:15
  • I am using mpi4py – Mikael Ken Dec 19 '19 at 07:17
  • Any reason for not using classic threading? Im not used to mpi4py but I'd always prefer not using and external library – Aaron Dec 19 '19 at 07:18
  • 1
    It is a requirement :) – Mikael Ken Dec 19 '19 at 07:21
  • Alright, interesting question then, upvoted it. – Aaron Dec 19 '19 at 07:26
  • I switched back to multiprocessing library as you pointed me. Seemed easier. However i run p.join but how can I gather that data and use in another function ? ... @Aaron – Mikael Ken Dec 19 '19 at 08:42
  • Take a look at this question, if this doesnt help you, I'll help you with sample code! https://stackoverflow.com/questions/6893968/how-to-get-the-return-value-from-a-thread-in-python @Mikael Ken – Aaron Dec 19 '19 at 08:50
  • @Aaron this seems to be working. However, is it really that parallel? And how can I know how many threads it is using? – Mikael Ken Dec 19 '19 at 09:01
  • Depending on your implementation to check currently active threads you can use: threading.active_count() if you used a queue for example each script gets executed after each other. – Aaron Dec 19 '19 at 09:06
  • By using this: with concurrent.futures.ThreadPoolExecutor() as executor, first result is computed after 11 seconds and the second one after 2 seconds. But, second result won't show before, thus it means it's waiting for first to complete – Mikael Ken Dec 19 '19 at 09:09

0 Answers0