0

sorry for this question because there are several examples in Stackoverflow. I am writing in order to clarify some of my doubts because I am quite new in Python language.

i wrote a function:

def clipmyfile(inFile,poly,outFile):
... # doing something with inFile and poly and return outFile

Normally I do this:

clipmyfile(inFile="File1.txt",poly="poly1.shp",outFile="res1.txt")
clipmyfile(inFile="File2.txt",poly="poly2.shp",outFile="res2.txt")
clipmyfile(inFile="File3.txt",poly="poly3.shp",outFile="res3.txt")
......
clipmyfile(inFile="File21.txt",poly="poly21.shp",outFile="res21.txt")

I had read in this example Run several python programs at the same time and i can use (but probably i wrong)

from multiprocessing import Pool
p = Pool(21)  # like in your example, running 21 separate processes

to run the function in the same time and speed my analysis

I am really honest to say that I didn't understand the next step.

Thanks in advance for help and suggestion Gianni

Community
  • 1
  • 1
Gianni Spear
  • 7,033
  • 22
  • 82
  • 131

1 Answers1

1

The map that is used in the example you provided only works for functions that recieve one argument. You can see a solution to this here: Python multiprocessing pool.map for multiple arguments

In your case what you would do is (assuming you have 3 arrays with files, polies, outs):

def expand_args(f_p_o):
    clipmyfile(*f_p_o)  

files = ["file1.txt", "file2.txt"] 
polis = ["poli1.txt", "poly2.txt"]
outis = ["out1.txt", "out2.txt"]
len_f = len(files)
p = Pool()
p.map(expand_args, [(files[i], polis[i], outis[i]) for i in xrange(len_f)])
Community
  • 1
  • 1
Bogdan
  • 8,017
  • 6
  • 48
  • 64
  • Thanks!. p=Pool() need to be empty inside ()? – Gianni Spear Oct 12 '12 at 10:59
  • If I refer to the documentation, it can be empty, or you can define the number of process you want. The call is of the form multiprocessing.Pool([processes[, initializer[, initargs[, maxtasksperchild]]]])¶ – Gcmalloc Oct 12 '12 at 11:37