unable to add spark to PYTHONPATH

Question

I am struggling to add spark to my python path:

(myenv)me@me /home/me$ set SPARK_HOME="/home/me/spark-1.2.1-bin-hadoop2.4"
(myenv)me@me /home/me$ set PYTHONPATH=$PYTHONPATH:$SPARK_HOME:$SPARK_HOME/python:$SPARK_HOME/python/build:$SPARK_HOME/bin

(myenv)me@me /home/me$ python -c 'import sys; print(sys.path)'
['', '/home/me/.virtualenvs/default/lib/python2.7', '/home/me/.virtualenvs/default/lib/python2.7/plat-x86_64-linux-gnu', '/home/me/.virtualenvs/default/lib/python2.7/lib-tk', '/home/me/.virtualenvs/default/lib/python2.7/lib-old', '/home/me/.virtualenvs/default/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/home/me/.virtualenvs/default/local/lib/python2.7/site-packages', '/home/me/.virtualenvs/default/lib/python2.7/site-packages']

(myenv)me@me /home/me$ python -c 'import pyspark'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named pyspark

Possible duplicate of [How do I install pyspark for use in standalone scripts?](http://stackoverflow.com/questions/25205264/how-do-i-install-pyspark-for-use-in-standalone-scripts) — Kamil Sindi, Dec 31 '16 at 15:18

score 7 · Answer 1 · edited May 23 '17 at 10:30

7

I've got the same problem, but this helped.

Just add the following command in your .bashrc

export SPARK_HOME=/path/to/your/spark-1.4.1-bin-hadoop2.6
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/build:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH

edited May 23 '17 at 10:30

Community

1
1

answered Aug 04 '15 at 06:27

ppk28

367
4
6

https://stackoverflow.com/questions/40808064/pyspark-append-executor-environment-variable – mathtick Feb 26 '18 at 19:05

score 0 · Answer 2 · edited May 23 '17 at 12:02

0

I think you mixed up PYTHONPATH and sys.path. But are you sure you need to modify PYTHONPATH if you have pyspark installed properly?

EDIT:

I haven't used pyspark, but would this help? importing pyspark in python shell

edited May 23 '17 at 12:02

Community

1
1

answered Mar 03 '15 at 10:52

laike9m

18,344
20
107
140

pyspark is installed. I can run it in shell mode but I can't debug the code using an IDE - I get the ImportError. – thecheech Mar 03 '15 at 10:55
@cheeech That's because you didn't set correct python interpreter in your IDE. Apparently you're using env, then you should set your virtualenv as your project interpreter. – laike9m Mar 03 '15 at 10:57
the env interpreter is set in the IDE. The issue is external to the IDE (I am running the python commands on the command line). – thecheech Mar 03 '15 at 10:59
How did you install pyspark? – laike9m Mar 03 '15 at 11:12
downloaded and extracted http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.1/spark-1.2.1.tgz – thecheech Mar 03 '15 at 11:15
I tried it both with source code package and pre-built for hadoop – thecheech Mar 03 '15 at 11:20

unable to add spark to PYTHONPATH

2 Answers2

Linked