2

R is not detecting the latest Java version (1.8) installed for the usage of sparklyr package. 

config <- spark_config()
config$sparklyr.gateway.port = 10000
config$sparklyr.gateway.connect.timeout = 1  
config$sparklyr.gateway.start.wait = 1000
library(sparklyr)
library(dplyr) 
library(DBI)
sc <- spark_connect(master="yarn-client", method = c("shell"),config=config, app_name = "sparklyr",version="2.0.0", extensions = sparklyr::registered_extensions())

Error in validate_java_version(spark_home) : 
Java version1.6.0.65 detected but 1.7+ is required. Please download and 
install Java from https://www.java.com/en/

I updated the Java version and confirmed in Mac terminal. Then I restarted the Rsession:

/Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java -
version 
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

But I'm still getting the same error outdated Java version:

Error in validate_java_version(spark_home) : 
Java version1.6.0.65 detected but 1.7+ is required. Please download and 
install Java from https://www.java.com/en/

Any idea how to fix this?

zx8754
  • 52,746
  • 12
  • 114
  • 209
albit paoli
  • 161
  • 2
  • 11

2 Answers2

4

Currently (beginning 2019, version 0.9.4) sparklyr requires java v1.7-1.8 and the default version installed on macOSX is usually 1.6.

You can check your version in the terminal with 

java -version

If you get something like 

java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-468-11M4833)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-468, mixed mode)

you need to update it. If you have brew installed, simply do:

brew tap caskroom/versions
brew cask install java8

Once you do that you can check the installation on your local machine with 

ls -1 /Library/Java/JavaVirtualMachines/

Usually, after restarting the R session, sparklyr should recognise the right version (1.8). If that would NOT be the case, you can set the JAVA_HOME environment variable in R with: 

Sys.setenv(JAVA_HOME = "/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home")

This works for me when using the RStudio recommended guidelines for connecting in local mode.

I have another java version, what do I do?

It's important to type java8 when calling the brew cask install otherwise the latest version of java would be installed. I did this mistake and installed openjdk version "11.0.2" 2019-01-15.

If you have a similar situation do brew cask remove java to uninstall java and get the right version required.

This info is comes from an issue on the sparklyr GitHub Repo and another stackOverflow question.

Ni-Ar
  • 146
  • 14
  • 1
    The brew command needs to be updated because of recent changes to Oracle's licensing requirements. I'd recommend `$ brew cask install homebrew/cask-versions/adoptopenjdk8`. Then, from R: `Sys.setenv(JAVA_HOME = "/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home")`. – Grant Mar 09 '20 at 17:48
0

I was able to fix this by installing the latest JAVA version from the following website:

http://www.oracle.com/technetwork/java/javase/downloads/index.html

You could further check if the version was updated successfully by running the following command on mac terminal:

java -version

Now I'm able to use the sparklyr package.

albit paoli
  • 161
  • 2
  • 11