Spring Batch - Remove Certain Types of Lines From Reader & JSON Parsing

Question

What I'm attempting to do here is only read/parse the JSON portion of a file that is structured like this:

From: blahblah:blahblah:3:120319123
2015-02-03 20:05:48.4070783443 +0000 UTC
{"id":"daj;flkaj;dl", "body":"dafjelkaf", "morestuff": 
"flakejfl;ka}flkajfl;daf{fad"}

From: blahblah:blahblah:2:120319123
2016-04-03 20:05:48.4070783443 +0000 UTC
{"id":"daj;flkaj;dl", "body":"dafjelkaf", "morestuff": 
"flakejfl;ka}flkajfl;daf{fad"}

I was originally planning to use FlatFileItemReader with the JsonLineMapper, but I would first need to discard the useless lines before parsing the Json line.

How can I skip the line that starts with From:, the line with the date, and blank line when reading?

Tip: using `BufferedReader` to read file line by line, then check if line start with `{` then read it into a string, and find the line end with `}` to end your json. — TuyenNTA, Jun 14 '17 at 22:45
Whoops that was a mis-click on suggestions.My bad, looks like it was already corrected. Thanks for pointing it out. :) — Novacane, Jun 16 '17 at 00:29

score 1 · Accepted Answer · answered Jun 15 '17 at 06:51

1

Extend the FlatFileItemReader and call

setRecordSeparatorPolicy(yourPolicyTOSkipLines);

Where yourPolicyTOSkipLines is your own implementation of SimpleRecordSeparatorPolicy

See the blank lines skipping code How to skip blank lines in CSV using FlatFileItemReader and chunks

answered Jun 15 '17 at 06:51

StanislavL

56,971
9
68
98

This looks like it's what I need, I'll accept the answer as soon as I get a chance to implement. Thanks! – Novacane Jun 16 '17 at 00:30
This worked. Created a custom implementation that would return false for end of record if the beginning of the line didn't start with "{" and then in the pre-processor I would remove that line. – Novacane Jun 28 '17 at 16:11

score 0 · Answer 2 · answered Apr 12 '18 at 09:15

0

Another slightly more hacky kind of approach would be to use the setComments() method of the FlatFileItemReader. The method requires a String array with comment prefixes. If you add "From:" to that array, it will be skipped.

answered Apr 12 '18 at 09:15

TomRaaff

83
1
9

Spring Batch - Remove Certain Types of Lines From Reader & JSON Parsing

2 Answers2