19

Same problem as Failed to start master for spark in windows 10 which is also not solved.

My spark is working well by testing pyspark.cmd and spark-shell.cmd

After runing .\sbin\start-master.sh I got:

ps: unknown option -- o
Try 'ps --help' for more information.
starting org.apache.spark.deploy.master.Master, logging to C:\spark-1.6.1-bin-hadoop2.6/logs/spark--org.apache.spark.deploy.master.Master-1-%MY_USER_NAME%-PC.out
ps: unknown option -- o
Try 'ps --help' for more information.
failed to launch org.apache.spark.deploy.master.Master:
  ========================================
  Picked up _JAVA_OPTIONS: -Xmx512M -Xms512M
full log in C:\spark-1.6.1-bin-hadoop2.6/logs/spark--org.apache.spark.deploy.master.Master-1-%MY_USER_NAME%-PC.out

I tried to visit web UI, while the localhost:4040 is working the localhost:8080 cannot be reached.

And I found there is the .log file created at the folder of %SPARK_HOME%/logs . They contains same content:

Spark Command: 

C:\Program Files\Java\jdk1.7.0_79\bin\java -cp C:\spark-1.6.1-bin-hadoop2.6/conf\;C:\spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar;C:\spark-1.6.1-bin-hadoop2.6\lib\datanucleus-api-jdo-3.2.6.jar;C:\spark-1.6.1-bin-hadoop2.6\lib\datanucleus-core-3.2.10.jar;C:\spark-1.6.1-bin-hadoop2.6\lib\datanucleus-rdbms-3.2.9.jar -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --ip hahaha-PC --port 7077 --webui-port 8080

========================================
Picked up _JAVA_OPTIONS: -Xmx512M -Xms512M

Working environment: Spark: 1.6.1 Windows 10

Looking forward to your reply and thanks for your time so much!

Community
  • 1
  • 1
Spider
  • 1,380
  • 4
  • 22
  • 42
  • do you have java 7+ on your java home & class path? run command 'echo %java_home%' – Nahum Apr 13 '16 at 09:06
  • you are trying to run a shell command in powershell, do you think it's possible ? – eliasah Apr 13 '16 at 09:09
  • @NahumLitvin Yeah. After input the echo %JAVA_HOME%, I got C:\Program Files\Java\jdk1.7.0_79. – Spider Apr 13 '16 at 09:13
  • @eliasah I got this result by running ".\sbin\start-master.sh" in git bash. Is there anything wrong? Thank you so much! – Spider Apr 13 '16 at 09:15

5 Answers5

74

The launch scripts located at %SPARK_HOME%\sbin do not support Windows. You need to manually run the master and worker as outlined below.

  1. Go to %SPARK_HOME%\bin folder in a command prompt

  2. Run spark-class org.apache.spark.deploy.master.Master to run the master. This will give you a URL of the form spark://ip:port

  3. Run spark-class org.apache.spark.deploy.worker.Worker spark://ip:port to run the worker. Make sure you use the URL you obtained in step 2.

  4. Run spark-shell --master spark://ip:port to connect an application to the newly created cluster.

Kenny John Jacob
  • 1,188
  • 8
  • 21
Abhishek Chaurasia
  • 816
  • 6
  • 4
  • I tried step 1) using cygwin on windows. After running the command, it doesn't return anything. So I don't have the url. Do I have to run anything before that? – kavya Nov 18 '16 at 11:23
  • It works perfectly. The session is there till cmd is open. As soon as I close the cmd, it gets terminated. Is there something missing here to maintain its session. – Utkarsh Saraf Oct 30 '17 at 07:38
  • Works for me in WindowsPowerShell with RunAsAdministrator – RazvanParautiu Oct 05 '18 at 08:36
  • 1
    Want to add that I had trouble with step 1. Ensure that your JAVA_HOME variable has no spaces in it, whether by reinstall or DOS-type pathing, and that it points to the actual Java folder, and not the /bin in your Java install. – KGVT Oct 15 '18 at 18:58
  • Works perfectly on windows 10, Spark 2.4.0 and Hadoop 2.7, just make sure that all environment variables are correct. – Damir Varevac Mar 03 '19 at 11:05
10

If you are looking to start the master worker and slaves this should work for you. it works for me

  1. To start master worker open windows command prompt on the spark/bin directory then copy and paste this command and hit enter
spark-class org.apache.spark.deploy.master.Master

You will need to point your browser to http://localhost:8080/. If you get an error message "server not found" refresh page. From this page you will get your unique url. looks like this URL: spark://192.xxx.xx.xxx:7077

  1. open a new terminal and go the %SPARK_HOME%/bin,copy and paste this line of code and hit enter.
spark-class org.apache.spark.deploy.worker.Worker spark://ip:port

This part 

spark://ip:port
is the URL obtained from step 1. Refresh the browser tab opened in step one to see if the worker has started.

NOTE: JDK 1.9 is not supported

Kaimenyi
  • 121
  • 1
  • 2
7

Just found answer here: https://spark.apache.org/docs/1.2.0/spark-standalone.html

"Note: The launch scripts do not currently support Windows. To run a Spark cluster on Windows, start the master and workers by hand."

Spider
  • 1,380
  • 4
  • 22
  • 42
1

After executing spark-class org.apache.spark.deploy.master.Master, just goto http://localhost:8080 to get ip:port. And then open another command shell to execute spark-class org.apache.spark.deploy.worker.Worker spark://IP:PORT

Ashish Garg
  • 11
  • 1
0

a little trick should help. I changed JAVA_HOME path to the DOS version: c:\Progra~1\Java\jre1.8.0_131 for instance then rebooted. After this i was able to run spark-class org.apache... command mentioned above.

yImI
  • 121
  • 1
  • 5