Cannot import pandas on Spark worker

Question

I have been able to set up a virtualenv and install my required libraries in all of the worker nodes of my Spark cluster, but I still cannot import pandas:

Traceback (most recent call last):
  File "/scratch/nas/2/larissa/ExperimentsML/app/experimenter/sklearn/sklearn-spark-tests.py", line 209, in <module>
    main()
  File "/scratch/nas/2/larissa/ExperimentsML/app/experimenter/sklearn/sklearn-spark-tests.py", line 202, in main
    print sc.parallelize(experiments).map(lambda experiment: run_experiment(df, input_dict, experiment)).collect()
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py", line 808, in collect
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 79 in stage 0.0 failed 4 times, most recent failure: Lost task 79.3 in stage 0.0 (TID 82, 172.18.8.2, executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 174, in main
    process()
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 169, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/scratch/nas/2/larissa/ExperimentsML/app/experimenter/sklearn/sklearn-spark-tests.py", line 202, in <lambda>
    print sc.parallelize(experiments).map(lambda experiment: run_experiment(df, input_dict, experiment)).collect()
  File "/scratch/nas/2/larissa/ExperimentsML/app/experimenter/sklearn/sklearn-spark-tests.py", line 61, in run_experiment
    import pandas
ImportError: No module named pandas

I have this code:

sc.parallelize(experiments).map(lambda experiment: run_experiment(df, input_dict, experiment)).collect()

And inside run_experiment I have the following imports:

def run_experiment(df, input_dict, experiment):
    from sklearn.pipeline import Pipeline
    from sklearn.svm import SVC
    from sklearn import datasets, linear_model, tree
    from sklearn.preprocessing import *
    from sklearn.tree import DecisionTreeClassifier
    from sklearn.linear_model import LinearRegression, LogisticRegression
    from sklearn.neighbors import KNeighborsClassifier
    from sklearn.naive_bayes import GaussianNB
    from sklearn.ensemble import RandomForestClassifier
    from sklearn import metrics
    from sklearn.metrics import *

    import pandas

    ...

I only get the error with pandas, so scikit-learn is installed successfully.

In my install_virtualenv.sh script I have the following libraries, and from the logs they are all installed correctly.

pip install --upgrade pip
pip install urllib3[secure]
pip install pyopenssl ndg-httpsclient pyasn1
pip install cloudpickle==0.2.2
pip install distributed==1.16.1
pip install joblib==0.11
pip install scipy==0.18.1
pip install numpy==1.12.0
#pip install pandas==0.19.2
#pip install pandas --upgrade
pip install pickleshare==0.7.4
pip install py4j==0.10.4
pip install python-dateutil==2.6.0
pip install --upgrade requests
pip install scikit-learn==0.18.1
pip install sklearn==0.0
pip install sklearn-pandas==1.3.0
pip install spark-sklearn==0.2.0
easy_install pandas

With pip list I get:

asn1crypto (0.22.0)
backports-abc (0.5)
backports.ssl-match-hostname (3.5.0.1)
certifi (2017.4.17)
cffi (1.10.0)
chardet (3.0.4)
click (6.7)
cloudpickle (0.2.2)
cryptography (1.9)
dask (0.14.1)
dask-searchcv (0.0.2)
distributed (1.16.1)
enum34 (1.1.6)
futures (3.1.1)
HeapDict (1.0.0)
idna (2.5)
ipaddress (1.0.18)
joblib (0.11)
msgpack-python (0.4.8)
ndg-httpsclient (0.4.2)
numpy (1.12.0)
pandas (0.19.2)
pathlib2 (2.2.1)
pickleshare (0.7.4)
pip (9.0.1)
psutil (5.2.2)
py4j (0.10.4)
pyasn1 (0.2.3)
pycparser (2.17)
pyOpenSSL (17.0.0)
python-dateutil (2.6.0)
pytz (2017.2)
requests (2.17.3)
scandir (1.5)
scikit-learn (0.18.1)
scipy (0.18.1)
setuptools (2.2)
singledispatch (3.4.0.3)
six (1.10.0)
sklearn (0.0)
sklearn-pandas (1.3.0)
sortedcollections (0.5.3)
sortedcontainers (1.5.7)
spark-sklearn (0.2.0)
tblib (1.3.2)
toolz (0.8.2)
tornado (4.5.1)
urllib3 (1.21.1)
zict (0.1.2)

As you can see, I have even tried with easy_install, but still cannot import pandas. Any idea as to why this is happening?

Thanks!

All my python packages installed correctly on ubuntu with: sudo apt-get -y install python-numpy python-scipy python-matplotlib ipython ipython-notebook python-pandas python-sympy python-nose — flyingmeatball, Jun 08 '17 at 20:30
@flyingmeatball I do not have sudo access on the cluster, but what is the difference from installing with pip or easy_install? — Larissa Leite, Jun 08 '17 at 21:13
https://stackoverflow.com/questions/3220404/why-use-pip-over-easy-install — flyingmeatball, Jun 08 '17 at 21:15
@flyingmeatball thanks. yeah I don't normally use easy_install, only pip, but I gave it a try and still did not work — Larissa Leite, Jun 08 '17 at 21:16
pip list shows that it is installed, but you import is not working so issue is not with installing it. — oshaiken, Jun 08 '17 at 21:28

score 2 · Answer 1 · answered Jun 13 '17 at 22:02

2

It sounds like your Spark workers are pointing to the default/system installation of python (e.g. /usr/bin/python) rather than your virtualenv. You can tell Spark to use your virtualenv by setting the PYSPARK_PYTHON environment variable, e.g. export PYSPARK_PYTHON=/path/to/my/python.

answered Jun 13 '17 at 22:02

santon

4,395
1
24
43

Thank you for your answer. Do I set this in my `install_virtualenv.sh` script or somewhere else? – Larissa Leite Jun 13 '17 at 22:03
It shouldn't depend much on the virtaulenv. You should set the environment variable before you initializ your SparkContext. For example, if you're using the shell binary `pyspark` then you set it before calling `pyspark`. If you're starting your context in a Jupyter notebook, you can call something like `os.environ['PYSPARK_PYTHON'] = '...'`. – santon Jun 13 '17 at 22:08
Do I need install pandas on all spark worker , or just zeppellin ? – Mithril Aug 31 '18 at 10:32
You will need pandas installed on all spark worker machines in the same path. That’s also true of any other python package your code imports. – santon Sep 01 '18 at 14:22

Cannot import pandas on Spark worker

1 Answers1