2

There appears to be something misconfigured with my project that's causing maven to miss the dependencies in my base pom.xml. When I try to build a module with intellij's build options, or build my project with mvn clean package, I get an error that 'object apache is not a member of package org'. My pom file has the following:

<dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
        <exclusions>
          ...
        </exclusions>
      </dependency>

I have confirmed that the spark-sql_2.11 (my version) jar exists in my local .m2 repository, and that my settings are configured to import maven projects automatically.

When I remove references to org.apache._ the jar builds successfully with both mvn clean package and intellij's build module menu option, however the two jars do not function the same. The jar created from building the module through intellij (/out/artifacts/[project]_jar/[project].jar) works fine with spark-submit, however using the jar created from mvn clean package (/target/[project]-test-SNAPSHOT.jar) throws an error 'No main class set in jar' despite the fact that it appears there is, 'jar tvf ' shows this class, and MANIFEST.MF has it correctly listed as the main class.

It almost seems like there's a disconnect between my scala class and the maven project, but I can't understand where. Any tips on how to diagnose why this isn't working would be greatly appreciated, and I can absolutely provide additional details if necessary. Thanks in advance!

Edit: adding full contents of pom.xml

    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <artifactId>basepom</artifactId>
    <groupId>com.cdtk.maven.basepom</groupId>
    <version>1.0-SNAPSHOT</version>
    <relativePath />
  </parent>

  <groupId>com.[company].testproject2</groupId>
  <artifactId>swtest2</artifactId>
  <version>test-SNAPSHOT</version>
  <packaging>jar</packaging>

  <description>
    This parent pom holds all the necessary version
    and configuration information for all submodules.
  </description>

  <properties>
    <required.maven.version>3.5.2</required.maven.version>
    <upper.maven.version>3.5.999</upper.maven.version>
    <required.java.update>20</required.java.update>

    <scala.major.version>2</scala.major.version>
    <scala.minor.version>11</scala.minor.version>
    <scala.patch.version>0</scala.patch.version>

    <spark.version>2.3.0</spark.version>
  </properties>

  <dependencyManagement>
    <dependencies>
      <!-- =================================== -->
      <!--        Core Spark BOM               -->
      <!-- =================================== -->
      <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
        <exclusions>
          <exclusion>
            <artifactId>commons-beanutils-core</artifactId>
            <groupId>commons-beanutils</groupId>
          </exclusion>
          <exclusion>
            <artifactId>jersey-server</artifactId>
            <groupId>com.sun.jersey</groupId>
          </exclusion>
          <exclusion>
            <artifactId>javax.inject</artifactId>
            <groupId>javax.inject</groupId>
          </exclusion>
          <exclusion>
            <artifactId>javax.ws.rs-api</artifactId>
            <groupId>javax.ws.rs</groupId>
          </exclusion>
          <exclusion>
            <artifactId>stax-api</artifactId>
            <groupId>javax.xml.stream</groupId>
          </exclusion>
          <exclusion>
            <artifactId>aopalliance</artifactId>
            <groupId>aopalliance</groupId>
          </exclusion>
          <exclusion>
            <artifactId>commons-logging</artifactId>
            <groupId>commons-logging</groupId>
          </exclusion>
          <exclusion>
            <artifactId>paranamer</artifactId>
            <groupId>com.thoughtworks.paranamer</groupId>
          </exclusion>
        </exclusions>
      </dependency>
      <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcprov-jdk15on</artifactId>
        <version>1.60</version>
      </dependency>
      <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-core</artifactId>
        <version>2.6.5</version>
      </dependency>
      <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.major.version}.${scala.minor.version}.${scala.patch.version}</version>
      </dependency>
      <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.7.7</version>
      </dependency>
      <dependency>
        <groupId>org.xerial.snappy</groupId>
        <artifactId>snappy-java</artifactId>
        <version>1.1.2.6</version>
      </dependency>
      <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-math3</artifactId>
        <version>3.4.1</version>
      </dependency>
      <dependency>
        <groupId>commons-lang</groupId>
        <artifactId>commons-lang</artifactId>
        <version>2.6</version>
      </dependency>
      <dependency>
        <groupId>com.google.code.findbugs</groupId>
        <artifactId>jsr305</artifactId>
        <version>3.0.2</version>
      </dependency>
      <dependency>
        <groupId>org.scala-lang.modules</groupId>
        <artifactId>scala-parser-combinators_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>1.0.4</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>commons-compiler</artifactId>
        <version>3.0.8</version>
      </dependency>
      <dependency>
        <groupId>org.glassfish.hk2.external</groupId>
        <artifactId>javax.inject</artifactId>
        <version>2.5.0-b32</version>
      </dependency>
      <dependency>
        <groupId>org.glassfish.jersey.core</groupId>
        <artifactId>jersey-common</artifactId>
        <version>2.25.1</version>
      </dependency>
      <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty</artifactId>
        <version>3.9.9.Final</version>
      </dependency>
      <dependency>
        <groupId>javax.activation</groupId>
        <artifactId>activation</artifactId>
        <version>1.1.1</version>
      </dependency>
      <dependency>
        <groupId>org.glassfish.jersey.core</groupId>
        <artifactId>jersey-client</artifactId>
        <version>2.25.1</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.9.13</version>
      </dependency>
      <dependency>
        <groupId>commons-net</groupId>
        <artifactId>commons-net</artifactId>
        <version>3.1</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-xc</artifactId>
        <version>1.9.13</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-jaxrs</artifactId>
        <version>1.9.13</version>
      </dependency>
      <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-core-asl</artifactId>
        <version>1.9.13</version>
      </dependency>
      <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-reflect</artifactId>
        <version>${scala.major.version}.${scala.minor.version}.${scala.patch.version}</version>
      </dependency>

      <!-- =================================== -->
      <!--        Top level dependencies       -->
      <!-- =================================== -->

      <dependency>
        <groupId>com.typesafe</groupId>
        <artifactId>config</artifactId>
        <version>1.3.0</version>
      </dependency>
      <dependency>
        <groupId>com.oracle</groupId>
        <artifactId>ojdbc6</artifactId>
        <version>11.2.0.3</version>
      </dependency>

      <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>spark-avro_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>4.0.0</version>
      </dependency>

      <!-- =================================== -->
      <!--        Test dependencies            -->
      <!-- =================================== -->
      <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>3.0.5</version>
        <scope>test</scope>
      </dependency>
      <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest-maven-plugin</artifactId>
        <version>2.0.0</version>
        <scope>test</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>org.scalatest</groupId>
      <artifactId>scalatest_${scala.major.version}.${scala.minor.version}</artifactId>
      <exclusions>
        <exclusion>
          <artifactId>scala-library</artifactId>
          <groupId>org.scala-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>scala-xml_${scala.major.version}.${scala.minor.version}</artifactId>
          <groupId>org.scala-lang.modules</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>net.alchim31.maven</groupId>
          <artifactId>scala-maven-plugin</artifactId>
          <version>3.2.2</version>
        </plugin>
        <plugin>
          <groupId>org.scalatest</groupId>
          <artifactId>scalatest-maven-plugin</artifactId>
          <version>2.0.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>
      <plugin>
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <scalaCompatVersion>${scala.major.version}.${scala.minor.version}</scalaCompatVersion>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.basepom.maven</groupId>
        <artifactId>duplicate-finder-maven-plugin</artifactId>
        <configuration>
          <ignoredClassPatterns combine.self="append">
            <ignoredClassPattern>org.apache.spark.unused.UnusedStubClass</ignoredClassPattern>
          </ignoredClassPatterns>
          <ignoredResourcePatterns combine.self="append">
            <!-- mercury internal file (should provide link to code that proves its ok to have more than one on the classpath) -->
            <ignoredResource>overrides.conf</ignoredResource>
            <ignoredResource>git.properties</ignoredResource>
          </ignoredResourcePatterns>
        </configuration>
      </plugin>
    </plugins>
  </build>

</project>
Sassy Dubs
  • 23
  • 4
  • 1
    can you paste pom file here ? – koiralo Jun 18 '19 at 13:03
  • 1
    are your variables defined anywhere? – steven35 Jun 18 '19 at 13:15
  • @ShankarKoirala I've added the full contents of the pom file above. Thanks! – Sassy Dubs Jun 18 '19 at 14:27
  • @steven35 which variables are you referring to? I believe the that necessary dependencies for the import statements that are causing the build to fail are present in the pom.xml (i.e import org.apache.spark.sql.SparkSession) – Sassy Dubs Jun 18 '19 at 14:29
  • Did you add spark dependencies inside dependencies ? – koiralo Jun 18 '19 at 14:54
  • @ShankarKoirala I had not, thank you for your response and for the useful link, that was exactly the info I needed. Any ideas about the second problem I am facing, where the main class is only recognized within the jar that was not created through maven CLI? – Sassy Dubs Jun 18 '19 at 16:58

2 Answers2

0

You should add the spark-core and spark-sql dependencies in your child pom or in current pom, if not add it as below 

<dependencies>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.major.version}.${scala.minor.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>

</dependencies>

Take a look at Difference between dependencies and dependency management

For the main class not found Add the maven-assembly-plugin plugin to the child pom as

koiralo
  • 22,594
  • 6
  • 51
  • 72
  • This was precisely what I needed, thank you! I took the structure of my base pom from a project that employed multiple modules with their own child poms and failed to realize the difference between dependencies and dependency management. Adding the above to the dependencies section of the base pom resolved my issue. – Sassy Dubs Jun 18 '19 at 16:52
  • I am still having the issue where the jar created with mvn clean package throws an error that there is no main class set in the jar, despite the fact that there is a 'main' definition in my App class and it is present in the jar when I view the contents with jar tvf. Any idea what the issue could be? I have tried a variety of different spark submit syntaxes (passing the --class parameter before the jar as seen in other threads, etc) but they all seem to encounter this error. – Sassy Dubs Jun 18 '19 at 16:56
  • @SassyDubs add maven-assembly-plugin to the child pom file with mainClass defined – koiralo Jun 19 '19 at 09:06
  • now it is failing with "[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-single) on project swtest2: Error reading assemblies: No assembly descr iptors found. " I tried adding the assembly.xml from another working project under src/tests/scala/assembly.xml and pointing the configuration to this file, but it doesn't seem to be working. What am I missing? – Sassy Dubs Jun 19 '19 at 20:34
0

You can add maven assembly jar to build a fat jar. Add the below plugin and update the name of your main class.

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.6</version>
                <executions>
                    <execution>
                        <id>jar-with-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                            <archive>
                                <manifest>
                                    <mainClass>***com.abc.mainclass***</mainClass>
                                </manifest>
                            </archive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
SantoshK
  • 139
  • 7
  • Adding this and trying to build with mvn clean package fails with the following: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-single) on project swtest2: Error reading assemblies: No assembly descriptors found. – Sassy Dubs Jun 19 '19 at 20:30