2

The newest version of Spark (1.4) now comes with SparkR. Does anyone know how to go about installing the SparkR implementation on Windows? The sparkR.R script is currently located in C:/spark-1.4.0/R/pkgs/R/

This appears to be a step in the right direction, but the instructions don't work for Windows as there is no sparkR directory as it relates to.

Optimus
  • 1,354
  • 1
  • 21
  • 40
  • 1
    Take a look [here](http://sgsong.blogspot.co.il/2015/06/how-to-use-sparkr-within-rstudio.html) maybe – David Arenburg Jun 16 '15 at 13:10
  • Thanks for the link @DavidArenburg. It seems like it's the right direction, but there is no SparkR directory in the spark-1.4.0 folder so it doesn't exactly provide a solution. – Optimus Jun 16 '15 at 13:27
  • You can find my solution to a similar question here: http://stackoverflow.com/a/31252999/3719822 – Emaasit Jul 06 '15 at 18:49

3 Answers3

3

@DavidArenburg put me on the right track.

Following the Windows documentation in the C:\spark-1.4.0\R\WINDOWS.md, I installed RTools and added R.exe and RTools to my computers PATH.

Then, I ran install-dev.bat in C:\spark-1.4.0\R This added the lib\SparkR\ installation that I was missing.

Then, from the command prompt, I ran

mklink /D "C:\Program Files\R\R-3.1.3\library\SparkR" "C:\spark-1.4.0\R\lib\SparkR"

This added a link in my R packages directory to the installation in the spark folder.

library(SparkR) # this should run now.
Optimus
  • 1,354
  • 1
  • 21
  • 40
0

Frustrated with finding a solution myself, I put together a quick write up here, which addresses the install on Mac, but it shouldn't be far from implementation in Windows. Hope this helps.

Mark S
  • 471
  • 3
  • 9
  • Thanks @MarkS, That's not too different than [this](http://sgsong.blogspot.co.il/2015/06/how-to-use-sparkr-within-rstudio.html), but certainly still helpful. I've outlined what else is necessary in my answer for Windows. – Optimus Jun 17 '15 at 19:59
  • 1
    Agreed - just wanted to show the end-to-end install as well, since I stumbled upon this thread while searching for an answer for getting started. – Mark S Jun 17 '15 at 20:28
0

For some reason I could not get it to work with the mklink.

So I very simply copy-pasted the SparkR folder from under my Spark installation (on my computer it's found at: C:\spark\spark\R\lib) to my R library folder (on my computer it's found at: C:\Program Files\R\R-3.1.1\library.

And now it's working!

ElinaJ
  • 791
  • 1
  • 6
  • 18