Cannot run multiple SparkContexts at once

Question

I am at the beginner stage of learning spark. I have Just started coding using python using pyspark.while going through basic code I got this error on Jupyter notebook. Well I have installed spark on my PC which in working condition. My problem is when I enter "pyspark" on my Ubuntu terminal it directly goes to webUI of jupyter. It doesn't go in Interactive shell. I dont know why?

2nd when I run following code I got error ..

from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName('appName').setMaster('local')
sc = SparkContext(conf=conf)
data = range(10)
dist_data = sc.parallelize(data)
print(dist_data.reduce(lambda a, b: a+b))

error of above code is...

Error Message

ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=local[*]) created by at /home/trojan/.local/lib/python3.6/site-packages/IPython/utils/py3compat.py:186

What does that mean?? Please tell me what could be the error! sorry for error image I couldn't paste it clearly so I pasted screen shot of error Hope it work!

What does it mean? I think that error message is pretty clear. — pault, Jan 18 '18 at 19:31
Possibly related: https://stackoverflow.com/a/47870277/5858851 — pault, Jan 18 '18 at 23:25

score 4 · Accepted Answer · answered Jan 18 '18 at 19:27

4

You can run only one spark context for one python kernel (notebook). If you need another spark context you can open another notebook, otherwise, there are no reason for multiple spark contexts on the same notebook, you can use it multiple times, depends on your problem.

answered Jan 18 '18 at 19:27

maksim

88
5

I didnt get it what you said!! well am I missed something while Installation. is my spark installation gone wrong??? – Shriniwas Jan 19 '18 at 15:04
1

I resolved my problem by reinstalling jupyter and it works fine! I am able to run my code using 'spark-submit'. my Pyspark shell and Jupyter notebook is working fine now! – Shriniwas Jan 20 '18 at 11:04
I closed the notebook, but the error message remained. – Soerendip Sep 13 '18 at 23:07

score 2 · Answer 2 · answered Jul 09 '20 at 15:40

2

Please try this code-

from pyspark import SparkContext
sc = SparkContext.getOrCreate();

answered Jul 09 '20 at 15:40

Sanjay Nandakumar

383
2
10

score 0 · Answer 3 · answered Jul 12 '18 at 17:12

0

Check whether you have called SparkContext() more than once. Mke it as one in

answered Jul 12 '18 at 17:12

VjyAnnd

1
3

Cannot run multiple SparkContexts at once

3 Answers3