Multiprocessor Numerical Adder

Question

So I am currently involved in a university project looking at thousands of samples of genetic data for cancer patients, might program was going to take too long to run so I used multiprocessing, it worked fine on an apple mac my friend borrowed me,but the moment I transferred it over to a university windows system it has failed and im unsure why the program doesn't work anymore.

I decided to strip my code as simply as possible to see the error,my program itself without the multiprocessing element to speed up the number of samples works fine. I believe the problem revolves around the code below. Instead of placing my very long program ive switched out it for a simple addition, and it still does not work, uses a very high cpu and I cannot see where I am going wrong. Kind Regards.

Expected result is instant 5,15,25,35 instantaneously, I have windows 10 on my computer Im currently using.

 import multiprocessing
 from multiprocessing import Pool
 import collections
 value=collections.namedtuple('value',['vectx','vecty'])
 Values=(value(vectx=0,vecty=5),value(vectx=5,vecty=10),value(vectx=10,vecty=15),value(vectx=15,vecty=20))
 print(1)
 def Alter(x):
   vectx=x.vectx
   vecty=x.vecty
   Z=(vectx+vecty)
   return(Z)
 if __name__ == '__main__':
     with Pool(2) as p:
          result=p.map(Alter, Values)
 print(2)
 new=[]
 for i in result:
     new.append(i)
 print(new)

I've attempted to put your code in your question but I may not have gotten all the indentation correct. Please take a look and fix if needed. — wwii, Sep 18 '19 at 19:53
[Formatting help](https://stackoverflow.com/help/formatting) ... [more Formatting](https://stackoverflow.com/editing-help) ... [Formatting sandbox](https://meta.stackexchange.com/questions/3122/formatting-sandbox) — wwii, Sep 18 '19 at 19:55
Your indentation is not correct - starting at `if __name__ == '__main__':` - please fix. — wwii, Sep 18 '19 at 20:24
I have resolved the small discrepancy , now please can I be provided a useful link to allow me to progress . — James Sharpe, Sep 18 '19 at 22:07
I would also recommend taking a look at numpy. You can handle large matrices easily, and even if you use multithreading and multiprocessing, it will still speed things up because your batches will be larger. — A T, Sep 19 '19 at 02:05

wwii · Accepted Answer · 2019-09-19T14:59:01.547

0

I don't know why exactly but this part

 print(2)
 new=[]
 for i in result:
     new.append(i)
 print(new)

needs to be in the suite of the if statement. Similar to the example in the documentation.

if __name__ == '__main__':
    with Pool(2) as p:
        result=p.map(Alter, Values)
    print(2)
    new=[]
    for i in result:
        new.append(i)
    print(new)

I suspect that - Compulsory usage of if __name__==“__main__” in windows while using multiprocessing - may be relevant.

If you run your original code from a command shell (like PowerShell or command prompt) with python -m mymodulename you will see all the stuff that is going on - Tracebacks from multiple spawned processes.

edited Sep 19 '19 at 14:59

answered Sep 19 '19 at 01:44

wwii

23,232
7
37
77

Thank you very much for your assistance, such a small tweak but with massive implication for if my program works, thank you. – James Sharpe Sep 19 '19 at 10:28
Hi again, okay so my program works independently, and the code tweak ive made allows multiprocessing to occur, why isn't the program working, it works on the mac but doesn't work on the windows. Should I post my entire program. – James Sharpe Sep 20 '19 at 12:27
That sounds like a different question. You'll need to include enough cose for people to easily reproduce the problem. - [mcve]. - `why isn't the program working, it works on the mac but doesn't work on the windows.` - you will need to be clear what you mean by this. The example code in this question with the indentation changes does work, it produces the expected result. – wwii Sep 20 '19 at 15:14

Multiprocessor Numerical Adder

1 Answers1