0

I am trying to run a map reduce job which takes an avro file as input and does some processing. I followed the sample program apache has given us here

http://avro.apache.org/docs/1.7.6/mr.html

But I keep on running into this exception

java.lang.Exception: java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)

Caused by: java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;

Any idea on what I may be doing wrong? I have specified my pom configs in the bottom. Also I am using MapR version 4.

<repositories>
    <repository>
        <id>MapR</id>
        <url>http://repository.mapr.com/maven/.</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.2.0</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.7.6</version>
    </dependency>
    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-mapred</artifactId>
        <version>1.7.6</version>
        <classifier>hadoop2</classifier>
    </dependency>
</dependencies>
anonymous123
  • 1,271
  • 6
  • 19
  • 43

2 Answers2

0

Common cause of such errors is this:

  • Your software was compiled against 1.7.6 version of avro, but in runtime, classes from older version were probably loaded.
  • Make sure that 1.7.6 is the actual version of your avro artifacts in your runtime classpath. Print out the classpath at the start of your mapper. If you're using oozie, the classpath jars are listed in launcher job output.
  • The first avro jar you see in the classpath is the one that will be used to load the classes, so if it isn't 1.7.6, that's the problem.
  • You can force your classpath artifacts to come first in the task's classpath by setting mapreduce.job.user.classpath.first configuration property to true.

Also you have another error in your pom that may very well cause you problems, maybe the very ones you're seeing. You are using avro-mapred artifact compiled for hadoop2 while the hadoop artifact you're depending on is that of hadoop1. These should not be compatible. If you're using hadoop1, loose the hadoop2 classifier on avro-mapred, and if you're using hadoop2, remove hadoop-core and put hadoop-mapreduce-client-core instead.

miljanm
  • 906
  • 7
  • 20
0

I have solved this by injecting the right Avro jar in bootstrap action, as described here:

https://stackoverflow.com/a/40235289/3487888

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
user3487888
  • 858
  • 7
  • 6