4

I am building my spark environment, refering to http://spark.apache.org/docs/latest/building-spark.html#spark-tests-in-maven. But when i use this command: "mvn -Pyarn -Phadoop-2.3 -DskipTests -Phive -Phive-thriftserver clean package", I met some errors.

[error] bad symbolic reference. A signature in WebUI.class refers to term eclipse
[error] in package org which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling WebUI.class.
[error] bad symbolic reference. A signature in WebUI.class refers to term jetty
[error] in value org.eclipse which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling WebUI.class.
[error]
[error]      while compiling: /download_wlh/spark-1.6.0/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
[error]         during phase: erasure
[error]      library version: version 2.10.5 
[error]     compiler version: version 2.10.5

There is somebody ask a question which is similar to this. Refering to strange error message: bad symbolic reference. A signature in package.class refers to term apache in package org which is not available

But there are no solutions mentioned.

Community
  • 1
  • 1
hitwlh
  • 71
  • 9
  • in which phase do you get these errors? It would also be helpful if you cod provide the WebUI signature that is refereed and the place where it is called. – hotzst Jan 17 '16 at 16:50

1 Answers1

5

I ran into the same problem as you trying to make a Spark distribution with Hadoop 2.4 using Java 8 and Scala 2.11.

My original attempt was: ./make-distribution.sh --name hadoop-2.4-custom --tgz -Phadoop-2.4 -Dscala-2.11

This produced the error described in Spark SQL 1.5 build failure

Following the advice from that post, I ran ./dev/change-version-to-2.11.sh and then omitted the -Dscala-2.11 as one of the comments suggested. This yielded your exact error. When I added the scala flag back in, the build passed. In summary, I fixed all my problems with:

./dev/change-version-to-2.11.sh ./make-distribution.sh --name hadoop-2.4-custom --tgz -Phadoop-2.4 -Dscala-2.11

Community
  • 1
  • 1
Tim K
  • 71
  • 1
  • 4
  • That worked for me. Here is the command I used `build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.5 -Dscala-2.11 -DskipTests clean package` – user1933178 Nov 16 '16 at 14:49