0

I have a list of 80 objects in a list that are iterated through with a while loop. There are many operations that have to be run for each object, the script works great, no problem, but it takes 7 hours to complete. Is there a way to spawn a new process for each object in the list instead of going through the list serially? I've looked at subprocesses and other techniques, would like some input from here, though.

So, assume this example;

i = 0
while i < len(listName):
    print("blah" + str(i))
    i += 1

When this processes, it does through each list item in order, one at a time. I would like to know how I can go about spawning a different process for each object in listName.

Kimomaru
  • 993
  • 4
  • 14
  • 30
  • do you mean `multiprocessing` and not `subprocess`? – eagle Apr 12 '18 at 19:30
  • But here the next loop depends on the previous loop? – Willem Van Onsem Apr 12 '18 at 19:30
  • 2
    Read about [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) and process pools specifically. Also, stating the _real_ problem more clearly should help. What is the nature of these "many operations"? It can significantly affect the answer. – 9000 Apr 12 '18 at 19:31
  • @9000, the actual code is a bit long, I didn't want to confuse the issue. The actual operations themselves run fine, I just need them to be run simultaneously. – Kimomaru Apr 12 '18 at 19:33
  • 1
    @Kimomaru: I see. But is the code CPU-bound (lots of numpy / pandas)? I/O-bound (lots of disk or network access)? How much memory does it consume? Do parts depend on one another? Etc. – 9000 Apr 12 '18 at 19:34
  • @9000, it uses the paramiko module to ssh into remote hosts, execute commands, grab output, uses regex against the output to get a value, then write the result to a file. Each host connected to has their own file, and when all of the hosts are complete I will combine the content from all the files to a super file that will comprise the report. Not sure if this answers you question. – Kimomaru Apr 12 '18 at 19:37
  • 1
    @Kimomaru: (1) It does! (2) for the task, coroutines (`async` stuff or [just `select`](https://stackoverflow.com/a/14888341/223424)) would be more economical if you have great many hosts, and (3) have you considered [Fabric](http://www.fabfile.org/)? It does have provisions for running commands on multiple hosts [in parallel](http://docs.fabfile.org/en/1.14/usage/parallel.html). – 9000 Apr 12 '18 at 19:50
  • @9000 , thank you! I will take a look at these. Much appreciated! – Kimomaru Apr 12 '18 at 20:00

0 Answers0