0

I am trying to run java code in IntelliJ (community 2019.2) on macbook pro. It is a very simple java code, I need to test if I can access first() from JavaPairRDD. My java version:

  12.0.2

My java code: 

    import org.apache.spark.SparkContext;
     import org.apache.spark.SparkConf;
     import org.apache.spark.api.java.JavaDoubleRDD;
     import org.apache.spark.api.java.JavaRDD;
      import org.apache.spark.api.java.JavaPairRDD;
      import org.apache.spark.api.java.JavaSparkContext;
      import java.io.IOException;
      import java.net.URL;
      import java.util.ArrayList;
      import java.util.Arrays;
      import java.util.Collections;
            import java.util.List;
            import java.util.concurrent.TimeUnit;
            import java.io.File;

     public class test_java_pair_rdd_from_mvn {

     public static void main(String[] args) {

          SparkConf conf = new SparkConf().setAppName("test").setMaster("local");
          JavaSparkContext sc = new JavaSparkContext(conf);

    JavaDoubleRDD doubleRDD = sc.parallelizeDoubles(Arrays.asList(1.0, 2.0, 3.0, 5.0, 8.0));
    JavaRDD<String> stringRDD = sc.parallelize(Arrays.asList("Hello", "World"));
    JavaPairRDD<String, Double> cartesian = stringRDD.cartesian(doubleRDD);
    cartesian.first();

    cartesian.collect().forEach(test -> {
        System.out.println(test._1);
        System.out.println(test._2);
    });


   }
}

My pom.xml:

 <?xml version="1.0" encoding="UTF-8"?>
 <project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
    http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.my_test</groupId>
<artifactId>test_java_rdd</artifactId>
<version>1.0-SNAPSHOT</version>

<properties>
    <scala.version>2.10</scala.version>
    <scala.binary.version>2.10</scala.binary.version>
    <java.version>12</java.version>
    <maven.javadoc.skip>true</maven.javadoc.skip>
</properties>

I got compile error:

 Error:java: error: release version 5 not supported

Why it complains this? I am using java 12. (language level 12)

thanks

user3448011
  • 1,469
  • 1
  • 17
  • 39
  • This is actually Maven's doing. The default target level for a Maven build (if the POM doesn't specify anything) is Java 5. You have a "java.version" property in the POM file, but that is not a property name that Maven understands. See http://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-source-and-target.html for the correct ways to specify the version in your POM file. – Stephen C Mar 30 '20 at 02:47
  • Note that some parents, most notably Spring Boot Parent, _do_ apply `java.version` in the appropriate configuration places. – chrylis -cautiouslyoptimistic- Mar 30 '20 at 03:32

1 Answers1

1

The JDK version or language level would have been differently setup in one of the settings. Right click on project, open Module Settings, choose the correct language level for each module from the dropdown.

If the above does not work for you, here is a detailed procedure that might help

  1. Goto Go to IntelliJ IDE File Menu Item -> Preferences -> Build, Execution, Deployment -> Java Compiler Make sure you have chosen the right compiler version.

  2. Right click on project, open Module Settings a) Ensure each module has the correct java compiler version b) Choose Platform > SDK's . Ensure correct SDK is setup and appropriate language level is setup

  3. Add the jdk version to your pom.xml

   <properties> 
      <maven.compiler.source>12</maven.compiler.source> 
      <maven.compiler.target>12</maven.compiler.target> 
   </properties>
  1. Right click on the project, there should be a item called Maven which has a sub menu item Reimport

That should work!

Deepak
  • 1,347
  • 7
  • 20