1

, Spark code compile operation fails on few machines whereas the same source code passes on few other machines. Please check the error on Centos (4.10.12-1.el7.elrepo.x86_64) ./build/mvn -X -DskipTests -Dscala.lib.directory=/usr/share/scala -pl core compile

INFO] Building Spark Project Core 2.2.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for org.apache.spark:spark-launcher_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-network-common_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-network-shuffle_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-unsafe_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-tags_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-tags_2.11:jar:tests:2.2.2-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.804 s
[INFO] Finished at: 2017-12-04T23:18:58-08:00
[INFO] Final Memory: 43M/1963M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:2.2.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-launcher_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-network-common_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-unsafe_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-tags_2.11:jar:2.2.2-SNAPSHOT, org.apache.spark:spark-tags_2.11:jar:tests:2.2.2-SNAPSHOT: Failure to find org.apache.spark:spark-launcher_2.11:jar:2.2.2-SNAPSHOT in http://artifact.eng.stellus.in:8081/artifactory/libs-snapshot was cached in the local repository, resolution will not be reattempted until the update interval of snapshots has elapsed or updates are forced -> [Help 1]

Note: The same source code passes on another CentOS machine(3.10.0-514.el7.x86_64) ./build/mvn -DskipTests -Dscala.lib.directory=/usr/share/scala -pl core compile

[INFO] — maven-compiler-plugin:3.7.0:compile (default-compile) @ spark-core_2.11 —
[INFO] Not compiling main sources
[INFO]
[INFO] — scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-core_2.11 —
[INFO] Using zinc server for incremental compilation
[info] Compile success at Dec 4, 2017 11:17:34 PM [0.331s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.663 s
[INFO] Finished at: 2017-12-04T23:17:34-08:00
[INFO] Final Memory: 52M/1297M
[INFO] ------------------------------------------------------------------------
Puja
  • 11
  • 2

2 Answers2

0

If the build works only in specific machine it means that you already have the missing dependency on your .m2 local repository. If you will remove all your directories under .m2 (on the centos machine) the build will fail.

You should add all your missing dependencies.

[WARNING] The POM for org.apache.spark:spark-launcher_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-network-common_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-network-shuffle_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-unsafe_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-tags_2.11:jar:2.2.2-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.spark:spark-tags_2.11:jar:tests:2.2.2-SNAPSHOT is missing, no dependency information available
Gal Shaboodi
  • 744
  • 1
  • 7
  • 25
  • Thanks for the reply. In our case the repository was not empty but still the build was not working. Hence I copied the .m2/repository from the machine where the build had passed clean to the server on which it failed. After performing compile operation the build ran successfully after copying the .m2/repository. Now my concern is what if I did not have any machine where the build had passed clean before and it as a very first attempt to perform compile operation on a newly installed server and I do not have any backup of .m2/repo from other servers how would I resolve this problem. – Puja Dec 15 '17 at 09:06
0

The following patch fix the compilation bug 

diff --git a/pom.xml b/pom.xml
index cc48ee794e..ef0b01bec8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2518,6 +2518,13 @@
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-jar-plugin</artifactId>
           <version>3.1.2</version>
+                 <executions>
+                   <execution>
+                     <goals>
+                       <goal>test-jar</goal>
+                     </goals>
+                   </execution>
+                 </executions>
         </plugin>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>

See here stackoverflow.com/questions/4786881 why is "test-jar" dependency required for "mvn compile"

Another possibility: you have a maven profile that does <maven.test.skip>true</maven.test.skip> (which skips both test compilation and execution)
instead of <skipTests>true</skipTests> (which skip execution, but compiles test)

Arnaud Nauwynck
  • 11
  • 2