7

Why do I get this error on my browser screen,

: Java gateway process exited before sending the driver its port number args = ('Java gateway process exited before sending the driver its port number',) message = 'Java gateway process exited before sending the driver its port number'

for,

#!/Python27/python
print "Content-type: text/html; charset=utf-8"
print

# enable debugging
import cgitb
cgitb.enable()

import os
import sys

# Path for spark source folder
os.environ['SPARK_HOME'] = "C:\Apache\spark-1.4.1"

# Append pyspark to Python Path
sys.path.append("C:\Apache\spark-1.4.1\python")

from pyspark import SparkContext
from pyspark import SparkConf

print ("Successfully imported Spark Modules")

# Initialize SparkContext
sc = SparkContext('local')
words = sc.parallelize(["scala","java","hadoop","spark","akka"])
print words.count()

I followed this example.

Any ideas how I can fix it?

Run
  • 54,938
  • 169
  • 450
  • 748
  • 1
    That might be helpful http://stackoverflow.com/a/30851037/296549 – Bacon Aug 06 '15 at 02:37
  • 1
    Nope... sorry. answers are poorly written. I don't understand them. – Run Aug 06 '15 at 03:30
  • I want to run the python scrip on browsers - not on a terminal or windows command prompt, by the way. – Run Aug 06 '15 at 03:39
  • my problem is not related to ipython notebook as well. – Run Aug 06 '15 at 04:08
  • 1
    Python+Spark on Windows is not very solid. I'm working with some guys from Microsoft and they aren't able to get around this kind of issue. Your problem is a little different than mine however. Doublecheck your paths? – Zach Garner Oct 05 '15 at 20:58
  • @ZachGarner I fixed that problem. Just run the script on wsgi. **never user cgi** for python! – Run Oct 07 '15 at 03:54

4 Answers4

2

Check if there are any extra information before the Error line that says:

Error: Could not create the Java Virtual Machine.

In my case it was an invalid option that I had set in the conf file. Memory (initial heap size) is not allowed to have a comma: 3.5g is for example not acceptable whereas 3500m is.

architectonic
  • 2,871
  • 2
  • 21
  • 35
2

I had a similar issue to this, and eventually when I looked at the my test's output there were error messages from $SPARK_HOME/bin/spark-class, with line numbers.

After tracking down what was going on on the affected lines it turned out that there were single quotes around the $JAVA_HOME value in my environmental variables, which was causing issues with path expansion (it was assumed for some reason to be relative to my home directory, rather than an absolute path)

While this may not be your exact issue, it is worth examining the start of your output for extra information to help in narrowing down the root cause.

theheadofabroom
  • 20,639
  • 5
  • 33
  • 65
1

My friend has met with the same problem as yours. I checked her computer and found that she had two versions of Java in it. I uninstalled the older one and rewrote the $JAVA_HOME value. The problem was solved.

user7210421
  • 11
  • 1
1

I've been trying to solve this problem since yesterday. I've just solved it.

I did exactly what this link (https://sparkbyexamples.com/pyspark/pyspark-exception-java-gateway-process-exited-before-sending-the-driver-its-port-number/) tells us:

(1) Set PYSPARK_SUBMIT_ARGS;

(2) Install Open JDK;

(3) Set JAVA_HOME Environment Variable.

I won't try to explain these steps to you because the link can do it much better than me (you just need to follow the steps).

By the way, it is a good resource in case you're learning pyspark.

Gus Cunha
  • 11
  • 3