1

I'm using maven shade plugin, and shaded jar is used to submit my ETL job. I want to use 1.7.7 and 1.9.1 versions of Apache avro as transitive dependency. But getting an error java.lang.NoClassDefFoundError: org/apache/avro/message/BinaryMessageEncoder. In the logs, we can see lower version is getting used. It was all fine before I set HADOOP_USER_CLASSPATH_FIRST=true when running the jar. What are ways I can have both versions on the classpath? Is it possible using shade plugin?

mazaneicha
  • 8,794
  • 4
  • 33
  • 52
Phaneendra H
  • 11
  • 2
  • 1
    Simple answer to this: That is not possible. You can not have two different version of the same artifact. – khmarbaise Jun 21 '21 at 11:47
  • 1
    It's doable, but tricky, because of transitive dependencies and runtime class losing. The process involves doing roughly what shade does--rewriting FQNs throughout the bytecode. It is quite brittle and error-prone. Why do you think you need both versions? – Dave Newton Jun 21 '21 at 13:03
  • _was all fine before I set_ - Then sounds like a separate problem where you should not be including Hadoop **provided** libraries in your own code unless you are sure they are backwards compatible – OneCricketeer Jun 22 '21 at 19:48
  • Normally **this should NEVER be done**. Check more on [multiple versions of the same dependency in maven / java](https://stackoverflow.com/questions/24962607/multiple-versions-of-the-same-dependency-in-maven). It is questionable why are you trying to do this. Try for a different solution, or you may want to explain why would you need 2 versions of avro in the same code. Do you have a scenario where. – azbarcea Jun 22 '21 at 19:53
  • This happens quite often actually, if your application code requires new features/bug fixes from newer version of a common utility/tool than the one used by the platform itself. – mazaneicha Jun 22 '21 at 21:21
  • avro models directly used in my app are built using 1.9.1 with that I get below error java.lang.NoSuchMethodError: org.apache.avro.Schema.getJsonProp(Ljava/lang/String;)Lorg/codehaus/jackson/JsonNode; at org.kitesdk.data.spi.ColumnMappingParser.hasEmbeddedColumnMapping(ColumnMappingParser.java:104) ~ – Phaneendra H Jun 23 '21 at 17:30

1 Answers1

0

Found that org.apache.avro - 1.8.2 is compatible with both (1.7.7 an 1.9.1) that worked for me.

Phaneendra H
  • 11
  • 2