1

When I create a Scala SBT project in IntelliJ, everything works fine. I set up the jar dependencies and I can run the examples.

But if I create a Java project and try to run:

Error:(29, 28) java: C:\workspace-java\SparkExamples\src\org\apache\spark\examples\sql\JavaSparkSQL.java:29: package org.apache.spark.sql does not exist

I added that jar to module dependencies which contains the class in the error message. In the IDE everything seems fine, I can navigate into the JavaSparkSQL class. But when I make, it's doesn't find that class.

I added the jar to CLASSPATH, and also copy to JAVA_HOME/jdk/jre/lib/ext.

What do you think what do I wrong? What should I check?

UPDATE: The problem is only with org.apache.spark.sql package. There are other packages what I can use. (org.apache.spark.SparkConf; org.apache.spark.api.java.JavaRDD; etc.)

UPDATE 2.: I gave a try to Eclipse Mars, and it was completed successfully the first time. So it is an IntelliJ specific problem.

UPDATE 3.:

Something wrong with the jar file which comes with Apache Spark (spark-assembly-1.4.1-hadoop2.6.0.jar).

Eclipse has no problem with this jar.

I found that when I delete an irrelevant file from jar ("about.html") then the problem is gone. IntelliJ see every package from the modified jar.

How is this possible?

Ákos Kiszely
  • 47
  • 1
  • 1
  • 10
  • do you have spark-sql jar in your classpath when you are running the jar? ex :http://central.maven.org/maven2/org/apache/spark/spark-sql_2.10/1.4.1/spark-sql_2.10-1.4.1.jar – Jishnu Prathap Jul 23 '15 at 08:13
  • 1
    Thank you for your great comment Jishnu Prathap. Based on your comment I have made a workaround for this problem. After I added spark-sql.jar then the original error gone but another appeared: class file for org.apache.spark.sql.types.StructType not found The original jar (spark-assembly-1.4.1-hadoop2.6.0.jar) contains everything and more than spark-sql.jar so I copied every class into spark-sql.jar from the assembly jar. And it worked! But I don't understand the core of the problem. Why everything works in Eclipse without that spark-sql.jar and why not in IntelliJ. – Ákos Kiszely Jul 23 '15 at 08:45
  • What is the scope of dependency you have added? by default its compile scope.If you make it runtime i think it should work in Intellij too.http://stackoverflow.com/questions/7070570/can-a-program-depend-on-a-library-during-compilation-but-not-runtime – Jishnu Prathap Jul 23 '15 at 08:55
  • The scope is compile but the error comes when IntelliJ compile so it should be good. (I tried with runtime without success). – Ákos Kiszely Jul 23 '15 at 09:01
  • One more fun thing: I copy spark-assembly-1.4.1-hadoop2.6.0.jar and rename to spark-assembly-cut.jar and in this new jar I delete everything but the sql package. When I add this new jar eveything works fine! So it seems IntelliJ cannot read well this assembly.jar. Maybe there are too much content in it? I don't know. – Ákos Kiszely Jul 23 '15 at 09:04

0 Answers0