4

There is list of files which I am processing one by one with some function:

list_of_files = [file_1, file_2, file_3, .... file_n]

# each file we are processing in function function_x(file)

for file in list_of_files:
   function_x(file)

But it takes too long to process one file so I would like to process 4 files in paralel, when any of them is finished, to continue with next form list_of_files

user3292147
  • 369
  • 2
  • 4
  • 11
  • See [this example](http://stackoverflow.com/a/2846697/2296458) for how you could use threading. Similarly you could use [mulitprocessing](https://stackoverflow.com/questions/8753306/python-multiprocessing-for-parallel-processes) – Cory Kramer Dec 30 '15 at 15:40
  • well, take a look at [Concurrency documentation](https://docs.python.org/3/library/concurrency.html) on the python site, try it yourself, and when you encounter an issue, then come back here with your code, then you'll get help. – saljuama Dec 30 '15 at 15:43

1 Answers1

5

Try to use parallel map:

import multiprocessing
pool = multiprocessing.Pool()
pool.map(function_x, list_of_files)
Dima Kudosh
  • 7,126
  • 4
  • 36
  • 46
  • How to make sure it it waits till processing one of four files is finished before continuing with next one. Code above would process all N files at same time and my machine would crash – user3292147 Dec 30 '15 at 15:45
  • 2
    @user3292147 If you read the documentation for `Pool` it says *"If processes is `None` then the number returned by `cpu_count()` is used"* So it will not create a process for `N` files, it will create up to `cpu_count` number of processes unless you tell it otherwise. – Cory Kramer Dec 30 '15 at 15:47