0

I am trying to install Pyspark on windows since yesterday but I am constantly getting this error. It's been more then 48 hours, I tried everything to resolve the problem. Reinstalled Pyspark from scratch numerous times but still could not get it to work.

Whenever I am running -

spark = SparkSession.builder.getOrCreate()

I am getting this error -

RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_20592/2335384691.py in <module>
      1 # create a spark session
----> 2 spark = SparkSession.builder.getOrCreate()

c:\users\bhola\appdata\local\programs\python\python38\lib\site-packages\pyspark\sql\session.py in getOrCreate(self)
    226                             sparkConf.set(key, value)
    227                         # This SparkContext may be an existing one.
--> 228                         sc = SparkContext.getOrCreate(sparkConf)
    229                     # Do not update `SparkConf` for existing `SparkContext`, as it's shared
    230                     # by all sessions.

c:\users\bhola\appdata\local\programs\python\python38\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
    390         with SparkContext._lock:
    391             if SparkContext._active_spark_context is None:
--> 392                 SparkContext(conf=conf or SparkConf())
    393             return SparkContext._active_spark_context
    394 

c:\users\bhola\appdata\local\programs\python\python38\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    142                 " is not allowed as it is a security risk.")
    143 
--> 144         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    145         try:
    146             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

c:\users\bhola\appdata\local\programs\python\python38\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
    337         with SparkContext._lock:
    338             if not SparkContext._gateway:
--> 339                 SparkContext._gateway = gateway or launch_gateway(conf)
    340                 SparkContext._jvm = SparkContext._gateway.jvm
    341 

c:\users\bhola\appdata\local\programs\python\python38\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf, popen_kwargs)
    106 
    107             if not os.path.isfile(conn_info_file):
--> 108                 raise RuntimeError("Java gateway process exited before sending its port number")
    109 
    110             with open(conn_info_file, "rb") as info:

RuntimeError: Java gateway process exited before sending its port number

I Tried the solution given in this stackoveflow post and in this stackoverflow2 post.

export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell"

In my windows system I used variable name = PYSPARK_SUBMIT_ARGS and variable value = "--master local[2] pyspark-shell"

But it's not working.

Other system variables that is set on my machine are during installations are-

SPARK_HOME = D:\spark\spark-3.2.0-bin-hadoop3.2

HADOOP_HOME = D:\spark\spark-3.2.0-bin-hadoop3.2

Path = D:\spark\spark-3.2.0-bin-hadoop3.2\bin

PYSPARK_DRIVER_PYTHON = jupyter

PYSPARK_DRIVER_PYTHON_OPTS = jupyter

JAVA_HOME = C:\Program Files\Java\jdk1.8.0_301

Can anyone help me with this?

bhola prasad
  • 675
  • 7
  • 22

1 Answers1

0

Did you download the winutils.exe from https://github.com/kontext-tech/winutils? You'll need to put that in \Hadoop\bin and add paths, etc.

M. Fury
  • 1