80

I have the following function:

def copy_file(source_file, target_dir):
    pass

Now I would like to use multiprocessing to execute this function at once:

p = Pool(12)
p.map(lambda x: copy_file(x,target_dir), file_list)

The problem is, lambda's can't be pickled, so this fails. What is the most neat (pythonic) way to fix this?

Nicolás Ozimica
  • 9,481
  • 5
  • 38
  • 51
Peter Smit
  • 27,696
  • 33
  • 111
  • 170

5 Answers5

77

Use a function object:

class Copier(object):
    def __init__(self, tgtdir):
        self.target_dir = tgtdir
    def __call__(self, src):
        copy_file(src, self.target_dir)

To run your Pool.map:

p.map(Copier(target_dir), file_list)
Fred Foo
  • 355,277
  • 75
  • 744
  • 836
70

For Python2.7+ or Python3, you could use functools.partial:

import functools
copier = functools.partial(copy_file, target_dir=target_dir)
p.map(copier, file_list)
unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677
  • 1
    This one even looks cleaner... I will decide later which one to make my answer – Peter Smit Jan 28 '11 at 13:30
  • @Peter Smit: Oops -- you saw my post before I deleted it... I'm undeleting this just to announce it doesn't work due to a bug in Python2. – unutbu Jan 28 '11 at 13:38
  • Ah, ok. I was looking in the docs and didn't see a comment about the pickability, so I assumed they were... I'll accept the other answer then, as that seems to be the way to go. – Peter Smit Jan 28 '11 at 13:52
  • 1
    Still, a +1 for this answer since it's shorter (in Python 3, that is ;) – Fred Foo Jan 28 '11 at 14:47
  • 9
    Landing here much later, as an update `functools.partial` is also picklable in python 2.7. – pythonic metaphor Jul 18 '13 at 22:42
  • This solution has the problem that only the second and higher arguments can be "curried". The function object solution is much more flexible. – Igor Rivin Jul 21 '18 at 00:18
  • 1
    Used this to fix a parallel search for non-isomorphic graphs. It runs 15x faster than the Fred Foo's solution – Alice Schwarze Oct 23 '19 at 02:27
11

Question is a bit old but if you are still use Python 2 my answer can be useful.

Trick is to use part of pathos project: multiprocess fork of multiprocessing. It get rid of annoying limitation of original multiprocess.

Installation: pip install multiprocess

Usage:

>>> from multiprocess import Pool
>>> p = Pool(4)
>>> print p.map(lambda x: (lambda y:y**2)(x) + x, xrange(10))
[0, 2, 6, 12, 20, 30, 42, 56, 72, 90]
petRUShka
  • 9,812
  • 12
  • 61
  • 95
1

From this answer, pathos let's you run your lambda p.map(lambda x: copy_file(x,target_dir), file_list) directly, saving all the workarounds / hacks

Community
  • 1
  • 1
Rufus
  • 5,111
  • 4
  • 28
  • 45
1

You can use starmap() to solve this problem with pooling.

Given that you have a list of files, say in your working directory, and you have a location you would like to copy those files to, then you can import os and use os.system() to run terminal commands in python. This will allow you to move the files over with ease.

However, before you start you will need to create a variable res = [(file, target_dir) for file in file_list] that will house each file with the target directory.

It will look like...

[('test1.pdf', '/home/mcurie/files/pdfs/'), ('test2.pdf', '/home/mcurie/files/pdfs/'), ('test3.pdf', '/home/mcurie/files/pdfs/'), ('test4.pdf', '/home/mcurie/files/pdfs/')]

Obviously, for this use case you can simplify this process by storing each file and target directory in one string to begin with, but that would reduce the insight of using this method.

The idea is that starmap() is going to take each component of res and place it into the function copy_file(source_file, target_dir) and execute them synchronously (this is limited by the core quantity of your cpu).

Therefore, the first operational thread will look like

copy_file('test1.pdf', '/home/mcurie/files/pdfs/')

I hope this helps. The full code is below.

from multiprocessing.pool import Pool
import os

file_list = ["test1.pdf", "test2.pdf", "test3.pdf", "test4.pdf"]
target_dir = "/home/mcurie/files/pdfs/"


def copy_file(source_file, target_dir):
    os.system(f"cp {source_file} {target_dir + source_file}")
    
if __name__ == '__main__':
    with Pool() as p:
        res = [(file, target_dir) for file in file_list]
        for results in p.starmap(copy_file, res):
            pass
Derek
  • 11
  • 1