Avro - java.io.IOException: Not a data file

Question

I am using https://github.com/allegro/json-avro-converter to convert my json message into an avro file. After calling the convertToAvro method I get a byte array: byte[] byteArrayJson. Then I am using the commons library from Apache:

FileUtils.writeByteArrayToFile(myFile.avro, byteArrayJson);

The file is created. When I try to reconvert it to json, using:

java -jar avro-tools-1.8.1.jar tojson myFile.avro > testCheck.json


Exception in thread "main" java.io.IOException: Not a data file.
    at 
org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105)
    at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:84)
    at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:71)
    at org.apache.avro.tool.Main.run(Main.java:87)
    at org.apache.avro.tool.Main.main(Main.java:76)

I have created a Junit test and used convertToJson method (from the previous link) and assert the strings and it is everything ok. But with the jar it is not working. Am I doing something wrong? I am using the cmd, not powerShell, because I saw in a SO post that this can change the encoding. I think that the problem is with encoding, but I have no idea where to look. (I am using windows as OS)

I don't understand avro but check this out and tell me if you're following the same steps? https://community.hortonworks.com/questions/32034/avro-javaioioexception-javaioioexception-not-a-dat.html — Justin Tamblyn, Jun 12 '19 at 06:56
you can also do like this https://avro.apache.org/docs/1.8.2/gettingstartedjava.html — Mak, Jun 12 '19 at 07:00
I had a look at both this links before posting, I am using the library from github because is really easy to play with it. You can convertToJson or convertToAvro very easy, but the only think is I want to store the byte array into an avro file. And when I store it and try to convert it using avro-tools is not working. I think I have some wrong bytes, as it is here: https://stackoverflow.com/questions/51148486/java-io-ioexception-not-a-data-file-after-converting-json-to-avro-with-avro-tool — agata, Jun 12 '19 at 07:04
The Windows 10 PowerShell transforms the binary stream into a UTF8 stream. Changing the encoding changes the magic bytes, which (correctly) causes the exception to be thrown. Side note: the PowerShell app can be forced not to change the encoding by using a pipe instead of greater-than like so: can you try like this from powershell java -jar .\avro-tools-1.7.7.jar fromjson --schema-file .\data.avsc .\data.json | .\data.avro — Mak, Jun 12 '19 at 07:06
Yes, is the answer from the link I posted. But as I mentioned in the post, I am using cmd, so no changes should happen. But I think that when the file is created something is going wrong, but IDK what — agata, Jun 12 '19 at 07:08
@Agata may i ask why you are using code to write the file and tools JAR to read the JSON back ? why not code ? — TechFree, Jun 12 '19 at 08:10

score 2 · Accepted Answer · edited Jul 29 '22 at 09:46

The reason is that the avro file do not contain same data when produced from these 2 different ways and this is expected behavior.

As a test, use this command to generate the avro file

java -jar avro-tools-1.8.2.jar fromjson  --schema-file avroschema.json
testCheck.json > myFile2.auro

Now read this and print in Java, and notice that it doesnt contain ONLY AVRO RECORD It contains the scme as well ( at least ) -see the String converted data below. This means the data in AVRO files is different when generated using acro tools and when using avro converter

bjavro.schemaœ{"type":"record","name":"Acme","fields":[{"name":"username","type":"string"}]}avro.c

The validation within tools API "fails" when you try to read an avro file generated from converter with tojson command.

Now the correct command to use to read the "json" using avro tools when the file is generated using converter is fragtojson. See that we are really reading only JSON fragment ( an avro record here )

java -jar avro-tools-1.8.2.jar fragtojson --schema-file avroschema.json myFile.avro > myFile21.json

Another thought here is avoid using AVRO tools altogether and create your own executable jar with converter as dependency, and use it read AVRO JSON records.

score 0 · Answer 2 · answered Jun 12 '19 at 08:34

After I had a look at the avro files, I saw that the one created using avro-tools has the schema in it and the one created by the library from github doesn't. So I don't use the method from the library convertToAvro, but I am using convertToGenericDataRecord and then create my own dataFileWriter and set the record (which was obtained after calling the convertToGenericDataRecord method).

Avro - java.io.IOException: Not a data file

2 Answers2

Linked