0

I'm building a small Python framework to run some experiments; I want to be able to launch many (>50) applications concurrently. However, I also want to be able to simulate different applications arriving at different times.

The quick and dirty way to do this would be to spawn a new Process object every time a new application arrives. The more refined answer is to use Process Pools (and cut down on the number of objects created). I've used both in the past, so I understand the basics of each.

What I'm unsure of is how to expand a Pool dynamically, such that if a Process object is unavailable, a new one is spawned and the Pool expands; but if a Process object is available, it's reused appropriately.

Is there a data structure -- or an object -- in Python's multiprocessing library to accomplish this?

tonysdg
  • 1,335
  • 11
  • 32
  • Is there are specific reason you'd want to expand the number of workers beyond the starting size of your `Pool`? Usually you'd just set the size of the pool to match the number of cores your CPU has, since that's going to be the most efficient number of parallel processes for CPU-limited jobs (any more and they'd just waste time fighting with each other for CPU time). – Blckknght Feb 01 '18 at 22:03
  • @Blckknght: The short answer is "yes". Without getting too far into the specifics, I'm running on an experimental system where the number of cores isn't just limited to the my current CPU -- think cluster-like setups. I may be launching all of the jobs on the first CPU, but after that they can migrate to other CPUs in the system. – tonysdg Feb 01 '18 at 22:06
  • @Blckknght: Now that I write that out, I'm realizing that what I'm describing is a rather specific scenario, and a custom solution would be needed anyway :) – tonysdg Feb 01 '18 at 22:07
  • Have you looked at https://stackoverflow.com/a/5182131/2398354 – TomDotTom Feb 02 '18 at 06:45
  • @TomDotTom: I haven't, but that's intriguing -- thank you!! – tonysdg Feb 02 '18 at 19:57

0 Answers0