1

I am writing code for testing existence of bucket in different clouds. For parallel execution have created 3 pools and using imap to call and run all three in parallel. But once one pool is completed it does not wait for other pool to complete and the main program prints the result by executing remaining part of the program. The problem arises on Linux only It is working fine for windows using imap()

Have tried using map, imap and imap_unordered.

from multiprocessing.dummy import Pool

def cloud1(bucket_name):
    some code to fetch respective details
    return

def cloud2(bucket_name):
    some code to fetch respective details
    return

def cloud3(bucket_name):
    some code to fetch respective details
    return

bucket_names = ["bucket1","bucket2","bucket3"] # A list to store Bucket names

#Calling all function using pool
p1=Pool(15)
p2=Pool(15)
p3=Pool(50)
results1=p1.imap(cloud1, bucket_names)
results2=p2.imap(cloud2, bucket_names)
results3=p3.imap(cloud3, bucket_names)
p1.close()
p1.join()
p2.close()
p2.join()
p3.close()
p3.join()

#Printing output
print(results1)
print(results2)
print(results3)

The code should wait for each pool to finish the execution and then print the output like it is doing in windows.

ItsDg4u
  • 11
  • 2
  • does this code actually run? I found that `bucket_names[]` should be `bucket_names=[]` – MEdwin Jul 05 '19 at 09:35
  • Have updated the code. Thanks for pointing it out. – ItsDg4u Jul 05 '19 at 09:57
  • I see what you are trying to do. For GNU Linux, you can actually make use of `fork` to create different processes as in `from os import fork`. Thank me later. You can create 143 processes in Colab. – MEdwin Jul 05 '19 at 15:58
  • But what if i need the same code to be run-able on both windows and linux – ItsDg4u Jul 05 '19 at 16:48
  • you can create a check variable to assign code for windows and one for linux. There is a module in python called `plaform`. Check out and print `platform.platform()`. – MEdwin Jul 08 '19 at 08:17
  • Hey I am not able to find a way how can I use fork within the above code can you provide me some example. – ItsDg4u Jul 17 '19 at 13:30
  • find good examples here: https://stackoverflow.com/questions/33560802/pythonhow-os-fork-works – MEdwin Jul 17 '19 at 15:27

0 Answers0