0

I'm working on a problem that requires me to run X (10, 20 or 30) processes at once, at the same time. Is that possible? My understanding of computer architecture is weak, sorry about that.

I've looked into Python 3's multiprocessing module, but I couldn't get it to work as I intend. For example:

def test(number):
    for i in range(4):
        print("process {}, iteration {}".format(number, i))

pool = ThreadPool(4)
processes = [i for i in range(10)]
pool.map(test, processes)

Output is something like this:

process 0, iteration 0
process 0, iteration 1
process 0, iteration 2
process 0, iteration 3
process 1, iteration 0
process 1, iteration 1
process 1, iteration 2
process 1, iteration 3
...

What I'd want is:

process 0, iteration 0
process 1, iteration 0
process 2, iteration 0
process 3, iteration 0
process 4, iteration 0
...
process 9, iteration 0
process 0, iteration 1
...

It doesn't really matter to me the order that the iterations occur, just that all 10 processes start at the same time. Python's pool method seems to be start N processes, then waits for them to finish, then starts the next N processes, etc. I'd like to start all processes at once.

Is that possible? Thanks!

CalculusNerd
  • 29
  • 4
  • 1
    Perhaps [this](https://stackoverflow.com/a/9786225/7315159) or [this](https://pythonhosted.org/joblib/parallel.html) might help. – Niayesh Isky Mar 24 '18 at 06:24
  • Why do you need them to execute at the same time? Do they have some internal dependencies or is it just a matter of ordering the output in a reasonable order? – JohanL Mar 24 '18 at 08:03
  • Ordering of output doesn't matter as long as the processes start simultaneously. Basically I'm writing a genetic algorithm where I have "N" workers, each of which start at the same time and the way that the workers evolve would depend on the other workers. That is, the quality of the results would depend on how many other workers are running along with it. So what Python's pool method seems to be doing, is starting only a subset of the workers, then WAITING for those to finish, before starting the next batch of workers. That's not good for my algorithm. – CalculusNerd Mar 24 '18 at 14:29
  • I was able to get this working using Multiprocessing.Process() and start() .... Essentially I called Process() N number of times, storing the Process() objects in a list. Then subsequently looped through them and called start(). It also turns out that this runs as expected on CMD, but messes up on Git Bash. (CMD starts these simultaneously, whereas Git Bash starts Process 1, waits for it to finish, before starting Process 2...) – CalculusNerd Mar 24 '18 at 18:27

0 Answers0