Parallel Python for loop

Question

I work primarily with arcgis and pci flavours of python 2.7. I have a number of processes that I've created that run outside of these programs but use these libraries. They are run via .bat files through cmd.

Currently, they run the processes in a series of for loops. And each for loop processes sequentially. I was wondering if there was a way to run the processing within the for loop for each object in the list at the same time. That is in parallel. The only way I can think of this is opening a cmd for each object in the list, and running the processing separately.

Is what I am asking even possible? Where should I look for solutions?

Refer to this [link](https://stackoverflow.com/questions/4081217/how-to-modify-list-entries-during-for-loop) hope it helps man, good luck I believe it is possible... not by cmd'ing each object in the list... Perl?? — Boschko, Dec 08 '17 at 22:56
This doesn't seem related to what I'm asking. I have to do it in python. — NorthLand, Dec 08 '17 at 22:59
wait so are you trying to call an external command in python and you want them to compute at the same time, non-sequentially? — Boschko, Dec 08 '17 at 23:02
I am running python and python scripts via a cmd batch file. I have some processes that require each vector shape within a shapefile to go through the same process but separate from each other. So each vector shape gets split from the original shapefile. So I have a list of shapefiles. These intermediary files get processed. Currently this processing runs sequentially. I want to parallelize it. — NorthLand, Dec 08 '17 at 23:07
Subprocess would allow you to call external commands and connect them to their input/output/error pipes (stdin, stdout, and stderr). Along those lines envoy is used as a convenience wrapper around the subprocess module.Thats what I'd look into — Boschko, Dec 08 '17 at 23:08
You could write `for` loops in Python that did what's in the .bat files, or you could launch the .bat files from Python. Which of those do you want to accomplish? — martineau, Dec 08 '17 at 23:27
@martineau the for loops are in Python. The .bat file just tells cmd which version of python to grab and what .py file to run in that file. I'm trying to make the for loops that are in Python run in parallel as opposed to sequentially. — NorthLand, Dec 08 '17 at 23:35
In that case it sounds to me like you would want to use the [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing) module which will allow you to run multiple Python processes in parallel. — martineau, Dec 08 '17 at 23:38
Well, the answer you've chosen won't allow much concurrent processing to occur. However, it not possible to supply you with anything more that a suggestion to look at `multiprocessing` given the complete lack of code in your question. — martineau, Dec 08 '17 at 23:52

score 1 · Accepted Answer · edited Dec 12 '17 at 18:05

Look into Subprocess So youd want a new commandline window created in the background where test.bat runs in parallel.. and in your case you don't want to wait for the command to complete before you continue your program, so use subprocess.Popen instead (may be something to look into as well)

subprocess.call

Run the command described by args. Wait for command to complete, then return the returncode attribute.

If you want to start an external program from your python script pass the program's filename to subprocess.Popen() on Ubuntu Linux you would enter something like

>>>import subprocess 
>>>subprocess.Popen('/usr/bin/gnome-...')
<subprocess.Popen Object at 0x7f2bcf93b20

The Return value is a Popen object which has two useful methods : poll() & wait()

poll() is like asking your friend if he has finished running the code you gave him. wait() is like waiting for your friend to finish working on his code before you keep working on yours.(something you might want to look into)

Parallel Python for loop

1 Answers1