Hadoop giving error on avro tools concat

Question

I'm trying to concatenate multiple avro files in an HDFS directory using avro tools. I'm following the approach as this question. However, I'm getting the following error:

Exception in thread "main" java.io.FileNotFoundException: /user/myuser/output/output.avro (No such file or directory)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
    at org.apache.avro.tool.Util.fileOrStdout(Util.java:79)
    at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:58)
    at org.apache.avro.tool.Main.run(Main.java:80)
    at org.apache.avro.tool.Main.main(Main.java:69)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

• The directory /user/myuser/output/ does exist.

• The command I'm using is hadoop jar /home/myuser/avro-tools-1.7.4.jar concat ${IN} /user/myuser/output/output.avro

• $IN has the value /user/hive/warehouse/somedatabase.db/some_table/on_date=2016-05-01/for_date=2016-05-08/confidence=0.8/000000_0

• The hadoop version is Hadoop 2.0.0-cdh4.7.0.

Theoretically, it should work, however it does not. I feel as if avro-tools is checking the local filesystem for the file(directory?) and that is causing the error. Any pointers?

Answer 1

Try using Avro 1.7.5 or later or prefix your argument with "hdfs://".

See https://issues.apache.org/jira/browse/AVRO-867 for more details.

Basically, AVRO-867 rewrote the utility methods used by avro-tools to deal with the file system. They now work as you expect, ie. no longer require the hdfs:// prefix and support any file system.