1

Although there are is a lot of information on multiprocessing, I can't figure out the right way for my problem:

I have a function that calculates some data - takes about 5 seconds each time. I need to run this function between 2 and 100 times. The function can be run independently, so if one call takes 20 seconds and another only 3, that does not matter for the end result. This case is more than perfect for a multiprocessing setup. But starting 100 processes in parallel for 100 function-calls does not make sense to me.

What is the keyword I'm looking for? Queue, Pool, Manager? I want to calculate always x cases at the time, depending on the number of cores a system has. Once one of those calculations is done and the CPU has a free core, it should take the next case out of a list and hit the transistors again. Once all of them are done, I would like to access the outputs.

Thanks in advance!

lyvic
  • 197
  • 1
  • 11
  • I suspect you're looking for a [thread pool](http://en.wikipedia.org/wiki/Thread_pool_pattern), with a number of threads equal to the number of cores. – Aya May 01 '13 at 11:20

1 Answers1

0

You can use the multiprocessing library. Or, if you want to move outside the standard library and get more features, see celery.

Also, read this article, because it goes over some of the differences between threads and processes. Specifically, threads are a performance hack, and unless you really need that boost, it is easier to let the kernel handle the details of IPC and shared time.

Spencer Rathbun
  • 14,510
  • 6
  • 54
  • 73
  • 1
    Thank you Spencer, your link was perfect of course and Aya gave me the right keyword just above. Unfortunately I ran into every possbile mistake at once. Here are two things one might consider when working with multiprocessing: [name=main](http://stackoverflow.com/questions/12465270/pythons-multiprocessing-map-async-generates-error-on-windows?rq=1) and [picklable](http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma). The rest ist easy. – lyvic May 03 '13 at 09:07