Maven build [WARNING] we have a duplicate class

Question

Anybody has any idea what happened to my maven build? I am getting a lot of duplicate warnings.

[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/NoOpLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/Jdk14Logger.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar

I've looked into my local m2 repo, I have two classes there in commons-logging-api jar, LogFactoryImpl.class and LogFactoryImpl$1.class. Same as all the classes mentioned in the warnings.

One thing to mention is that I am using shade plugin in my pom.xml.

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>1.4</version>
            <configuration>
                <createDependencyReducedPom>true</createDependencyReducedPom>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.~~~~black out my own main class here~~~~~</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>

I noticed that the dependency tree looks like as below

[INFO] +- org.apache.cxf:cxf-bundle-jaxrs:jar:2.5.1:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] \- org.apache.hadoop.hive:hive-jdbc:jar:0.7.1-cdh3u3:compile
[INFO]    \- org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile
[INFO]       \- commons-logging:commons-logging-api:jar:1.0.4:compile

and commons-logging.jar and commons-logging-api.jar both have org/apache/commons/logging/LogFactory.class.

somehow Shad plugin is trying to squeeze them in to a big fat jar at the end. then the warning is showing up. It's been said this is ignorable warning. But I am a bit worried, How does the application know what is the exact class should be used if there are two duplicated class with the same name?

LogFactoryImpl.class and LogFactoryImpl$1.class the class with $1 in the name is local class inside LogFactoryImpl. — khmarbaise, Mar 21 '12 at 11:54

score 14 · Answer 1 · edited Sep 15 '18 at 17:13

You might also have run into a limitation of maven-shader-plugin. It replaces the default jar artifact (created by maven-jar-plugin). This works fine on a clean build, but on a rebuild where the jar is not regenerated, the shader runs again on the jar it created last time, which already contains copies of all the class dependencies. That produces a lot of warnings about duplicates.

This issue is still unaddressed as of maven-shader-plugin 2.0: https://issues.apache.org/jira/browse/MSHADE-126

One workaround is to add the maven-jar-plugin explicitly to your pom.xml and add the configuration setting <forceCreation>true</forceCreation>.

This is what caused it for me. You can solve it by following the solution here: http://stackoverflow.com/questions/8880361/superfluous-warnings-when-using-maven-shade-plugin — stantonk, Aug 28 '13 at 16:56

score 13 · Accepted Answer · answered Mar 21 '12 at 15:34

13

Take a look at the "Dependency Exclusions" section in the Maven doc.

In your provided example, I'll exclude the commons-logging:commons-logging-api:jar:1.0.4:compile dependency from org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile. In your pom.xml :

    <dependency>
        <groupId>org.apache.hadoop.hive</groupId>
        <artifactId>hive-common:jar</artifactId>
        <version>0.7.1-cdh3u3</version>
        <exclusions>
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

answered Mar 21 '12 at 15:34

ndeverge

21,378
4
56
85

Thanks a lot, that make sense. so what's the best way to manage the dependencies like this. let's say, you have projectA.jar, projectB.jar, they have the same dependencies in different versions. eg. dependency.1.0, dependency.2.0. It has risks to just use one of them. Is there a way to let ProjectA keeps using depdency.1.0 while ProjectB still uses dependency.2.0? – Shengjie Mar 21 '12 at 20:09
1

Nope, there is no way (maybe with OSGI but I'm not an expert in this area). In this case, it is the class loader which selects the class to use (depending on the OS, JVM etc..., it is often the first jar referenced in the classpath that is used). Your best bet is to take the latest version (exclude other versions), hoping that it is backward compatible... But it is better than letting the classloader choosing for you. – ndeverge Mar 21 '12 at 20:37

score 5 · Answer 3 · answered Oct 02 '13 at 13:51

In my case, my parent pom was including commons-beanutils and my child module (which is the only thing I wanted to compile) was including commons-io.

The shade plug in complained about duplicates since commons-io and commons-beansutil shared some common classes. Note that beansutiul was being included even though it was not needed, and was not used.

I solve this by minimizing the jar by adding this to the configuration:

<minimizeJar>true</minimizeJar>

Now the shade plugin did not add unused resources.

Warning went away.

score 3 · Answer 4 · answered Mar 06 '17 at 14:38

3

All above (about reviewing dependencies tree and excluding) is correct in the most of cases, but in my case (i didn't have overlapping in my dependencies) preliminary clean helped (don't know why though):

mvn clean package

answered Mar 06 '17 at 14:38

radistao

14,889
11
66
92

score 2 · Answer 5 · edited Oct 29 '22 at 10:57

You can exclude the jar you don't want (the ones that are giving the duplicate warnings using the following tags under the shade plugin -

<configuration>
    <artifactSet>
        <excludes>
            <exclude>commons-logging:commons-logging</exclude>
        </excludes>
    </artifactSet>
    <minimizeJar>true</minimizeJar>
</configuration>

More details can be found at http://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html

score 0 · Answer 6 · answered Dec 09 '14 at 21:43

0

I saw this happen in eclipse when I updated my parent project's dependencies.

I deleted all the files in my target directory and it fixed the issues.

answered Dec 09 '14 at 21:43

Ben

2,771
6
33
45

score 0 · Answer 7 · answered Feb 07 '18 at 02:13

In my case I was relying on a package that also creates a shaded jar.

Shaded jars are meant for deployment, not installing as a dependency.

Creating a reduced dependency POM during the build process of the dependency, instructs maven on which dependencies can be left out.

In the maven-shade-plugin configuration:

<configuration>
  <createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>

For more details see this post:

What is the maven-shade-plugin used for, and why would you want to relocate java packages?

The error I was getting from maven:

WARNING: x.jar, y.jar contain overlapping classes

score -3 · Answer 8 · answered Mar 21 '12 at 11:58

-3

You have dependencies in your pom which contain duplicate classes but without the appropriate pom i couldn't say a word about it.

answered Mar 21 '12 at 11:58

khmarbaise

92,914
28
189
235

added more info, can you have a look at it again? Thanks – Shengjie Mar 21 '12 at 15:28
Have you checked which of the dependencies contains the duplicate classes? Based on your output the commons-logging-api-1.0.4.jar which seemed to be coming from using a different api version (duplicated) or you are using an other logging provider (slf4j ?). – khmarbaise Mar 21 '12 at 15:36

Maven build [WARNING] we have a duplicate class

8 Answers8

Linked

Related