Pyspark couldn't initialize spark context

Question

I am trying to run a spark program using this spark-submit server2.py --master local[2]. Then I got this error:

ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: '<pyspark.conf.SparkConf object at 0x7f69bb067710>'
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2924)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:548)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:238)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:745)

This is the code I am running :

import networkx as nx
TCP_IP = "192.168.1.136"
TCP_PORT = 5000
from pyspark import SparkConf,SparkContext
from pyspark.streaming import StreamingContext

# Create a Grid graph using networkx library
G = nx.grid_2d_graph(5, 5)  # 5x5 grid

# Creating a Spark Configuration
conf=SparkConf()
conf.setAppName('ShortestPathApp')

sc= SparkContext(conf)
ssc= StreamingContext(sc,1)

def shortestPath(line):
    # get the values from rdd
    vehicleId = line[0]
    source = line[1]
    destination = line[2]
    deadline = line[3]

    # find shortest path
    shortest = nx.dijkstra_path(G, source, destination)
    print(shortest)


# receive from Socket
dataStream =ssc.socketTextStream(TCP_IP,TCP_PORT)
vehicle_data = dataStream.map(lambda line: line.split(" "))
vehicle_data.foreachRDD(lambda rdd: rdd.foreach(shortestPath))
ssc.start()
ssc.awaitTermination()

Can anyone please help me what I am doing wrong. I tried solution from this post Couldn't initialize spark context but it is not working.

Can you see what is your `$SPARK_HOME` and `pyspark.__version__`? — pissall, Apr 27 '20 at 05:33

score 2 · Accepted Answer · edited Apr 27 '20 at 05:42

2

Reorder this

spark-submit server2.py --master local[2]

To

spark-submit --master local[2] server2.py

edited Apr 27 '20 at 05:42

ernest_k

44,416
5
53
99

answered Apr 27 '20 at 05:34

lone_ranger

75
4

can you please tell me how socketTextStream works. Do we need to add socket.connect to connect to the server to receive data @ernest_k – BhavinT Apr 27 '20 at 05:42
Please answer the above comment @lone_ranger – BhavinT Apr 27 '20 at 05:43

Pyspark couldn't initialize spark context

1 Answers1