0

I have installed hadoop on ubuntu on virtual box(host os Windows 7).I have also installed Apache spark, configured SPARK_HOME in .bashrc and added HADOOP_CONF_DIR to spark-env.sh. Now when I start the spark-shell it throws error and do not initialize spark context, sql context. Am I missing something in installation and also I would want to run it on a cluster (hadoop 3 node cluster is set up).

CDspace
  • 2,639
  • 18
  • 30
  • 36
Nakshatra
  • 1
  • 1
  • Any examples of what you've tried and what errors you're seeing would be appreciated. – Will Jul 25 '16 at 22:33
  • Welcome to Stack Overflow. [Read here](http://stackoverflow.com/help/mcve) for more information about how to create a Minimal, Complete and Verifiable question. – Toby Jul 25 '16 at 23:27
  • add detailed error you get, and how are you invoking `spark-shell`? did you source the `.bashrc` file after adding new env variables? – Ronak Patel Jul 25 '16 at 23:41
  • Thank you for the help. I am now able to install and run spark without any errors. There was a compatibility issue I suppose between spark 1.6.2 and hadoop 2.6. Now I would like to ask should I be making any master slave setting in the hadoop configuration files too? It would be really helpful if you could provide me some links – Nakshatra Jul 26 '16 at 11:54

1 Answers1

0

I have the same issue when trying to install Spark local with Windows 7. Please make sure the below paths is correct and I am sure I will work with you. I answer same question in this link So, you can follow the below and it will work.

  • Create JAVA_HOME variable: C:\Program Files\Java\jdk1.8.0_181\bin
  • Add the following part to your path: ;%JAVA_HOME%\bin
  • Create SPARK_HOME variable: C:\spark-2.3.0-bin-hadoop2.7\bin
  • Add the following part to your path: ;%SPARK_HOME%\bin
  • The most important part Hadoop path should include bin file before winutils.ee as the following: C:\Hadoop\bin Sure you will locate winutils.exe inside this path.
  • Create HADOOP_HOME Variable: C:\Hadoop
  • Add the following part to your path: ;%HADOOP_HOME%\bin

Now you can run the cmd and write spark-shell it will work.

Moustafa Mahmoud
  • 1,540
  • 13
  • 35