unable to load pyspark under anaconda

Asked Jul 24 '18 at 17:04

Active Jul 24 '18 at 17:34

Viewed 53 times

I have installed pyspark (pyspark-2.3.1-py37_1.tar). when I try to import, I get this error:

[bin]$ ./conda list | grep -i spark
    pyspark                   2.3.1                    py37_1    <unknown>

[bin]$ ./python
Python 2.7.15 |Anaconda, Inc.| (default, May  1 2018, 23:32:55)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyspark
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pyspark

Any ideas?

edited Jul 24 '18 at 17:08

OneCricketeer

179,855
19
132
245

asked Jul 24 '18 at 17:04

user1471980

10,127
48
136
235

I would suggest downloading the actual Apache Spark tarball package, then run `pyspark` out of that. – OneCricketeer Jul 24 '18 at 17:11
I do have export SPARK_HOME="/usr/local/spark-2.3.1-bin-hadoop2.7" set up, still not working – user1471980 Jul 24 '18 at 17:13
Have you tried running `$SPARK_HOME/bin/pyspark` ? – OneCricketeer Jul 24 '18 at 17:16
I installed pyspark under anaconda distribution. The package is there, when I try to import it in junyper notebook, import error. – user1471980 Jul 24 '18 at 17:24
1

I have never used Anaconda, but it sounds like your Jupyter Kernel is pointing at the wrong Python version where you have installed Spark into – OneCricketeer Jul 24 '18 at 18:28
As @cricket_007 has suggested, before trying with Jupyter, see first if you can run it in the command line with `$SPARK_HOME/bin/pyspark` – desertnaut Jul 24 '18 at 23:58
1

SPARK_HOME directory was set wrong in my path. I corrected the path in bashrc file and now it works fine. Thanks everybody for their input. – user1471980 Jul 25 '18 at 02:39

unable to load pyspark under anaconda

0 Answers0