Pickling error on concurrent.Futures.PoolProcessExecutor - How to Lock?

Question

I have been messing around with Multiprocessing on and off for months now trying to figure out an elegant repeatable solution to my issue of wanting multiple processes to write to the same file without messing each other up.

I have used the Multiprocessing Producer/Consumer relationship to overcome these hurdles in the past. Good articles and posts I've found include:

• Processing single file from multiple processes in python

• http://sebastianraschka.com/Articles/2014_multiprocessing_intro.html

I've tried implementing a function similar to a shared counter described here:

• http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing

I have become a big fan of the simplicity of the concurrent.Futures ProcessPoolExecutor and using map on each executor as described here:

• http://www.dalkescientific.com/writings/diary/archive/2012/01/19/concurrent.futures.html

• https://docs.python.org/3/library/concurrent.futures.html

Tonight, I thought I had found the answer to my search with discovering a module called fasteners for readwrite locks, but apparently this approach only works on threading.

• http://fasteners.readthedocs.org/en/latest/examples.html#reader-writer-shared-locks

QUESTION: IS there an elegant, simple solution to sharing a lock so that all Processes from ProcessPoolExecutor do not overwrite eachother when writing to a file?

NOTE: I'm writing about 800M rows of ~200 fields to one file using csv.DictWriter. Other recommendations are welcome.

Answer 1

You are looking at the solution from the wrong angle. Instead of sharing a lock to protect the access over a file, give file access to a single process. The other processes will just tell to it what to write.

From that perspective, there are plenty of questions similar to your on stackoverflow.

Python multiprocessing safely writing to a file

Writing to a file with multiprocessing

Python Multiprocessing using Queue to write to same file