-2

I need to do parallelization in python and I am going to use multiprocessing as in between I have suffered one difficulty, I made 7 processes and start it but my cpu don't give full performance ..why? it didn't show any error but it runs slow and cpu performance not power up.. please give some suggestion. Thanks

 p1 =  Process(target=TestFor1,args = ())
 p2 =  Process(target=TestFor2,args = ())
 p3 =  Process(target=TestFor3,args = ())
 p4 =  Process(target=TestFor4,args = ())
 p5 =  Process(target=TestFor5,args = ())
 p6 =  Process(target=TestFor6,args = ())
 p7 =  Process(target=TestFor7,args = ())
 p1.start()
 p2.start()
 p3.start()
 p4.start()
 p5.start()
 p6.start()
 p7.start()

 p1.join()
 p2.join()
 p3.join()
 p4.join()
 p5.join()
 p6.join()
 p7.join()

This TestFor1 - 7, I made Seven different function which has individual for loops. and I wanted to run this 7 for loop (independent) parallel as I have 12 core cpu I read something so I put Process.init(self) also at beginning but still it doesn't give cpu power up.. suggestion appreciate

dbr
  • 11
  • 4
  • You should have a look at this thread : http://stackoverflow.com/questions/5784389/using-100-of-all-cores-with-python-multiprocessing – Jerome Sep 02 '14 at 15:56
  • i had read it but its not enough.. – dbr Sep 02 '14 at 16:57
  • You need to post a concrete example and the OS you are working on. I cooked one up and all my processes hit 99%. BTW, the accepted answer for @Jerk31's reference only makes sense for multistep pipelined workloads. Its not a general purpose solution. – tdelaney Sep 02 '14 at 17:02
  • when I am not using join() then it goes up means using all cpu.. but using join it wont..so I am strange. – dbr Sep 04 '14 at 10:50

1 Answers1

1

It all depends on what you are doing the parallel processes. If they are not doing cpu intensive tasks or are all accessing the same media (e.g., writing to the same disk), then you'll only get so much from mp.

This used all the cores on my machine:

import multiprocessing

def func():
    for i in xrange(10000000):
       sum(range(100)) 

procs = []
for i in xrange(multiprocessing.cpu_count()):
    p = multiprocessing.Process(target=func)
    p.start()
    procs.append(p)

print 'started', len(procs), 'processes'

for p in procs:
    p.join()
    print 'process done'

print 'all done'
tdelaney
  • 73,364
  • 6
  • 83
  • 116
  • thanks for answer.. this program also use all cores in my pc too. but in my for loop its not power up the all cpu.. I did it same way as you mentioned.. but I don't know why it's not using all cores.. please share possibility if you have some idea.. – dbr Sep 04 '14 at 10:08
  • @dbr - it all depends on what your for loops are doing. Can you post a small example of your for loop that shows the problem or add some details on what it does? If your for loop is accessing some external resource like the hard drive, that will slow it down. – tdelaney Sep 04 '14 at 15:23
  • thanks for the answer .. but I did it .. but without using of join() it runs well. so is it necessary to use join() after start()? – dbr Sep 05 '14 at 13:41
  • join() causes the parent to wait until the child is finished. If you don't call join(), it will be called anyway at process exit (the parent still waits for the child to exit). I think calling it is a good idea - explicit is better than implicit. You can do `p.daemon = True` before start to have the parent kill the child before exit. – tdelaney Sep 05 '14 at 16:45
  • in above processes.. I did this seven process p1 to p7 parallel but how can I get return value in same processes?? – dbr Sep 11 '14 at 11:49
  • @dbr - you want the result back in the parent? There are several ways to do this. The simplest is the multiprocessing.Pool. The mp doc is pretty long, but its got several options. – tdelaney Sep 11 '14 at 15:36
  • @dbr. There are plenty of examples and tutorials out there. Start with the examples in the mp doc and move out from there. – tdelaney Sep 11 '14 at 15:49