3

I need to install Apache Spark on a Windows machine. According to the documentation I should have sbt installed on my machine and also override its default options to use a maximum of 2G of RAM.

After trying a lot I decided to choose maven instead (I overrided the default options to use 512MB) and I could run mvn compile that started to download some dependencies until it reached Zinc server. It said Zinc server didn`t run on port 8080 and maven failed the build.

Do I need to install the Zinc server locally and add it to Windows path? Can I compile my project without the server?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
Mr.Q
  • 4,316
  • 3
  • 43
  • 40
  • You do not need zinc at all to build Spark. It just speeds up incremental builds. – Sean Owen Dec 31 '14 at 18:45
  • See also more up-to-date advice at [How to set up Spark on Windows? - Stack Overflow](http://stackoverflow.com/questions/25481325/how-to-set-up-spark-on-windows) – nealmcb May 01 '15 at 00:03
  • Do you really have to build Spark from source? You can install pre-built packages, too. – stholzm Jan 08 '16 at 06:53

1 Answers1

1

I've just started learning Apache Spark so I might be far from the valid answer. It worked for me on Mac OS X so your millage may vary.

The build process is described in Building:

Spark uses Simple Build Tool, which is bundled with it. To compile the code, go into the top-level Spark directory and run

sbt/sbt assembly

I've not changed the default build settings to do the assembly.

Yesterday's build from the sources:

jacek:~/oss/spark
$ sbt/sbt assembly
...
[success] Total time: 3632 s, completed May 6, 2014 2:39:07 AM

Today's build from the sources with some environment variables that add few other modules to the final assembly:

jacek:~/oss/spark
$ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true sbt/sbt assembly
...
[success] Total time: 4564 s, completed May 7, 2014 1:49:56 AM
Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
  • i guess you could run it on mac OS because that mac machines have high config (e.g. high RAM) but as i said before i get the jvm memory error and i guess it is because of my ram (i have 2Gig RAM installed on my machine) but i colulden`t find any documents about minimum RAM requirments of SPARK. – Mr.Q May 07 '14 at 05:26
  • Use sbt and in the cmd do `set SBT_OPTS=-Xmx1g` and then execute `sbt/sbt assembly`. Add the stacktrace/error to your question when it happens. Stay away from maven from the moment as I can help more - I know it better than maven for Scala builds. – Jacek Laskowski May 07 '14 at 06:31