setup pySpark on intellij

Question

How to set up pySpark on intellij. Even after setting the environment variables spark_home and pythonpath, import pySpark is giving error - Import error : No module named pySpark

Have a look at `findSpark`. The problem is that PySpark isn't on sys.path by default You can address this by either symlinking pyspark into your site-packages, or adding pyspark to sys.path at runtime. findspark does the latter, see https://github.com/minrk/findspark — Alex, May 05 '17 at 08:47
you can try this https://medium.com/@gauravmshah/pyspark-on-intellij-with-packages-auto-complete-5e3208504707 — Gaurav Shah, Dec 13 '18 at 18:17

score 1 · Answer 1 · answered Jun 25 '17 at 23:50

1

The solution in this post worked for me.

There are 2 changes:
1. At the interpreter level - add $SPARK_HOME/python path for code completion
2. For the run configuration - add $SPARK_HOME and the Py4j-*.zip to PYTHONPATH for submitting jobs

answered Jun 25 '17 at 23:50

ganeiy

298
2
9

score 0 · Answer 2 · answered May 10 '17 at 09:04

0

Go to File -> Settings
Look for Project Structure
Click on Add Content Root and add $SPARK_HOME/python

After this, your editor will look in the Spark's python directory for source files.

answered May 10 '17 at 09:04

novic3

106
5

score 0 · Answer 3 · edited Jan 11 '21 at 23:58

0

Click on edit configuration
Click on environment variables
Add these variables

PYTHONPATH = %SPARK_HOME%\python;%SPARK_HOME%\python\build;%PYTHONPATH%

PYSPARK_SUBMIT_ARGS = --master local[2] pyspark-shell

SPARK_HOME = <spark home path>

SPARK_CONF_DIR = %SPARK_HOME\conf

SPARK_LOCAL_IP = 127.0.0.1

edited Jan 11 '21 at 23:58

OneCricketeer

179,855
19
132
245

answered Dec 22 '17 at 06:12

Siddhi Parkar

1
2

score 0 · Answer 4 · edited Apr 05 '21 at 16:02

I followed steps as per https://www.youtube.com/watch?v=j8AcYWQuv-M and it helped me connect successfully, with modifications as below.

Ensuring Plugin for Python is installed (I used Python 3.9)
Downloading Spark 3.1.1 from https://spark.apache.org/downloads.html. Entering details of python and py4j paths from here.
Setting the JAVA_HOME correctly - lower jdk 1.8 (JDK home path /Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home)
The extra additional step that I did was adding JAVA_HOME same as above as Environment Variable under Run/Debug Configurations option in IntelliJ.

setup pySpark on intellij

4 Answers4