1

I have a pair of functions defined, where the second is called inside the first, and the second has the associated pipe.recv() methods for each pipe.send() method in the first. However the first recv() method appears to be blocking despite information being in the pipe. Am I calling these in the wrong order? I don't quite understand what the problem is or why the recv() method would hang in this case. I know it's not a memory issue because the arrays involved are relatively small but I can't think of what the issue could be

def plots():

    #build axes and plots

    for i in range(num_sections):
        temp_arr = pipes_hist[1][0].recv()      #the KeyboardInterrupt traceback message points here
        arr_section += temp_arr       #in my actual code arr_section is predefined with the proper size

    #etc etc

def main_function(section_number):

     #lots of code goes here

     pipes_hist[section_number][1].send(array_section)
     pipes_hist[section_number][1].close()

     #etc etc

     plots()

#build list of pipes for each section

pipes_hist_send = [[] for i in range(num_sections)]
pipes_hist_recv = [[] for i in range(num_sections)]
pipes_hist = list(zip(pipes_hist_recv,pipes_hist_send))

for i in range(num_sections):
    pipes_hist[i] = list(pipes_hist[i])

for i in range(num_sections):         #these two loops can probably be combined
    pipes_hist[i][0],pipes_hist[i][1] = Pipe()

#begin multiprocessing

if __name__ == '__main__':
    pool = mp.Pool(num_sections)
    args = np.arange(num_sections)
    pool.map(main_function, args, chunksize=1)

    pool.close()
    pool.join()

Edit: The hanging also happens when I change map to map_async and call the plots function after the subsequent pool.map_async instead of inside main_function

tashton
  • 59
  • 7
  • May be it's this issue: https://stackoverflow.com/a/24914157/3254405 – boateng Nov 03 '17 at 17:54
  • Unfortunately it's not, I know for sure I'm calling the right ends at each `send()` and `recv()` method – tashton Nov 03 '17 at 18:00
  • I would try putting a for loop to sleep for .1 sec between opening more threads, may be it will solve the hanging issue.. – boateng Nov 03 '17 at 18:11
  • Could you elaborate? I'll try it but I'm just not sure where I would put that – tashton Nov 03 '17 at 18:16
  • May be put sleep functions in all your loops.. Threads and loops don't work together well.. – boateng Nov 03 '17 at 18:19
  • I tried adding time.sleep(0.1) in between every line of every loop and it's still hanging, which makes me relatively certain that each individual Pipe is safely defined and sequestered from the others – tashton Nov 03 '17 at 18:24
  • The other thing to check for is passing parameters into threads properly https://www.saltycrane.com/blog/2008/09/simplistic-python-thread-example/ – boateng Nov 03 '17 at 18:37
  • And since you want to pass the same variables you better switch to threads within the same process (where memory is shared) and not multiple processes with separate memory space.. https://stackoverflow.com/a/18114475/3254405 – boateng Nov 03 '17 at 18:48
  • 1
    Did you mean for the call to `recv()` inside the first function to always be using `pipes_hist[1]`? It seems like it should use `pipes_hist[i]`. – bnaecker Nov 03 '17 at 21:02
  • You're right, it should be `pipes_hist[i][0]`. However that's simply a typo here and not in my actual code – tashton Nov 04 '17 at 19:06
  • Too much of your code is missing to be certain, but it seems to me you are trying to `send()` data from a process to itself. This will cause the `send()` to block, if the data are large enough. Other (later in the queue) processes are probably waiting in `recv()` for those senders to get past their send-to-self, but they never do get past that. – torek Nov 05 '17 at 00:52
  • I've realized that embedding the `plots()` function inside the map may not be the best idea. I am attempting to break up a large process into several sections then stitch the outputs of each process together, then use that final product to generate the plots. Is there a better way to approach this idea? – tashton Nov 06 '17 at 02:35
  • The reason for multi processing is performance improvement.. https://stackoverflow.com/a/19718556/3254405 And sometimes it needs corresponding hardware too.. https://stackoverflow.com/questions/6457091/python-multiprocessing-between-amazon-cloud-instances – boateng Nov 08 '17 at 16:33

0 Answers0