Take a look here: R, Java, rJava and macOS adventures and here: R 3.4, rJava, macOS and even more mess ;)
R, Java and proper settings that make every piece of software happy can be a struggle ;)
You need to play with JAVA_HOME, R CMD javareconf, and sometimes even compilation of rJava from sources.
If you need a specific version of Java just for R, you can always play with something like this.
First, get the list of JVMs
/usr/libexec/java_home -V
Matching Java Virtual Machines (4):
9, x86_64: "Java SE 9" /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home
1.8.0_144, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
1.8.0_111, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home
1.7.0_80, x86_64: "Java SE 7" /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
Then, pick one just before starting R
-- start_R_with_Java_8.sh --
#!/bin/bash
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8.0_144)
export PATH=$JAVA_HOME/bin:${PATH}
R
This way, in system you will have Java 9, while for R, you will have 1.8.