Validate JSON before further processing

Question

I have the following csv file (In production the number of records can range from 20k-100k and has many fields )

id,firstname,lastname,email,profession
100,Betta,Wandie,Betta.Wandie@gmail.com,developer
101,Janey,Firmin,Janey.Firmin@gmail.com,doctor

I need to convert this to json and do further processing. CSV->JSON->PROCESS FURTHER

.I am able to convert it to JSON directly using the code given here directly convert CSV file to JSON file using the Jackson library

But i want do validations for json like if lastname has null value then ignore that record or id is missing then ignore that record.

How can i handle the validation?I am using Java 8 and spring boot latest version

Your code would be much simpler if you modeled your data using a Java record class (`Person`). You can read one record at a time with an iterator or stream. — chrylis -cautiouslyoptimistic-, Apr 02 '20 at 22:18
i was thinking that too.Can you share the code snippet for stream — Pale Blue Dot, Apr 02 '20 at 22:28

score 0 · Answer 1 · answered Apr 02 '20 at 22:34

I have done something similar by using JavaScript (Nashorn). Yes, that is nasty, but it works, and it is astonishingly fast!

Unfortunately, I do not have the source code at hand …

Why I did it that way had the same reasons as @chrylis-on strike implies in their comment: the validation is much easier if you have an object for each JSON record. But as I was lazy, and there was definitely no need for the Java object I had to create for this, I had the idea with the JavaScript script inside my Java program.

Basically, a JSON String is the source for a JavaScript program; you can assign it directly to a JavaScript variable and then you can access the records as array elements and the fields by their name. So your JavaScript code walks through the records and drops those that do not match your validation rules.

Now the Java part: the keyword here is JSR223; it is an API that allows you to execute scripts inside your Java environment. In your case, you have to provide the converted JSON in the context, then you start the script that writes the modified JSON back to the context.

If the converted JSON is too large, you can even use the same technique to check record by record; if you compile the script, it is nearly as fast as native Java.

You can even omit Jackson and let JavaScript do the conversion …

Sorry that I cannot provide code samples; I will try to get hold on them and add them if I get them.

This seems a whole lot more complicated than a 7-line Java record class. — chrylis -cautiouslyoptimistic-, Apr 03 '20 at 00:12
@chrylis-onstrike-: The OP said that "In production the number of records can range from 20k-100k and *has many fields*" – but for the sample data, you are right! — tquadrat, Apr 03 '20 at 08:46

Validate JSON before further processing

1 Answers1