14

I am accessing a very large Pandas dataframe as a global variable. This variable is accessed in parallel via joblib.

Eg.

df = db.query("select id, a_lot_of_data from table")

def process(id):
    temp_df = df.loc[id]
    temp_df.apply(another_function)

Parallel(n_jobs=8)(delayed(process)(id) for id in df['id'].to_list())

Accessing the original df in this manner seems to copy the data across processes. This is unexpected since the original df isnt being altered in any of the subprocesses? (or is it?)

autodidacticon
  • 1,310
  • 2
  • 14
  • 33

2 Answers2

13

The entire DataFrame needs to be pickled and unpickled for each process created by joblib. In practice, this is very slow and also requires many times the memory of each.

One solution is to store your data in HDF (df.to_hdf) using the table format. You can then use select to select subsets of data for further processing. In practice this will be too slow for interactive use. It is also very complex, and your workers will need to store their work so that it can be consolidated in the final step.

An alternative would be to explore numba.vectorize with target='parallel'. This would require the use of NumPy arrays not Pandas objects, so it also has some complexity costs.

In the long run, dask is hoped to bring parallel execution to Pandas, but this is not something to expect soon.

Kevin S
  • 2,595
  • 16
  • 22
  • 1
    I had assumed from http://stackoverflow.com/questions/10721915/shared-memory-objects-in-python-multiprocessing that subprocesses wouldnt receive a full copy unless the original object was altered. Does joblib break with copy-on-write semantics? – autodidacticon Nov 11 '15 at 17:40
  • 1
    Only a small number of types can be passed using shared memory. Pandas objects are not in this list. joblib automatically handles memory sharing for numpy arrays depending on the size of the array using the keyword argument `max_nbytes` when invoking `Parallel`. See [joblib's site](https://github.com/joblib/joblib/blob/master/doc/parallel_numpy.rst). Also see [this answer](http://stackoverflow.com/a/22487898/2551705). You can of course use NumPy arrays in place of Pandas and you might see speedups. – Kevin S Nov 12 '15 at 02:01
3

Python multiprocessing is typically done using separate processes, as you noted, meaning that the processes don't share memory. There's a potential workaround if you can get things to work with np.memmap as mentioned a little farther down the joblib docs, though dumping to disk will obviously add some overhead of its own: https://pythonhosted.org/joblib/parallel.html#working-with-numerical-data-in-shared-memory-memmaping

Randy
  • 14,349
  • 2
  • 36
  • 42