Shipping Python modules in pyspark to other nodes

Question

How can I ship C compiled modules (for example, python-Levenshtein) to each node in a Spark cluster?

I know that I can ship Python files in Spark using a standalone Python script (example code below):

from pyspark import SparkContext
sc = SparkContext("local", "App Name", pyFiles=['MyFile.py', 'MyOtherFile.py'])

But in situations where there is no '.py', how do I ship the module?

Related to [Easiest way to install Python dependencies on Spark executor nodes?](https://stackoverflow.com/questions/29495435/easiest-way-to-install-python-dependencies-on-spark-executor-nodes) — zero323, May 02 '18 at 16:44

score 49 · Accepted Answer · edited Nov 22 '16 at 01:49

49

If you can package your module into a .egg or .zip file, you should be able to list it in pyFiles when constructing your SparkContext (or you can add it later through sc.addPyFile).

For Python libraries that use setuptools, you can run python setup.py bdist_egg to build an egg distribution.

Another option is to install the library cluster-wide, either by using pip/easy_install on each machine or by sharing a Python installation over a cluster-wide filesystem (like NFS).

edited Nov 22 '16 at 01:49

EmmaOnThursday

167
4
9

answered Jul 10 '14 at 21:35

Josh Rosen

13,511
6
58
70

This worked! I was able to create the egg file for the module using your command above, and then the process was as simple as adding the egg file in the pyFiles parameter. Cool stuff! – mgoldwasser Jul 11 '14 at 14:18
Any suggestions in getting PySpark to install .egg dependencies? Similar to http://stackoverflow.com/questions/32119225/databricks-spark-egg-dependencies-not-installed-automatically – ramhiser Jul 06 '16 at 16:30
4

if my *.egg file depends on other 3rd party libraries, will this work as well? – guilhermecgs Oct 27 '16 at 19:36
@mgoldwasser how to do it in pyspark 2.0? – stackit Nov 08 '16 at 06:56
could you let me know how can we ship the .egg_info directory ? Is it the same ? – TopCoder Jul 28 '17 at 01:22
@Josh_Rosen yes, what is the egg_info directory for? – rjurney Mar 14 '21 at 18:47

score 4 · Answer 2 · edited Apr 29 '18 at 16:53

4

There are two main options here:

If it's a single file or a .zip/.egg, pass it to SparkContext.addPyFile.
Insert pip install into a bootstrap code for the cluster's machines.
- Some cloud platforms (DataBricks in this case) have UI to make this easier.

People also suggest using python shell to test if the module is present on the cluster.

edited Apr 29 '18 at 16:53

Peter Mortensen

30,738
21
105
131

answered Apr 23 '18 at 00:22

ivan_pozdeev

33,874
19
107
152

Shipping Python modules in pyspark to other nodes

2 Answers2

Linked

Related