How to parse a text file with separated json objects in Java?

Question

I have a text file which gets updated in every 15-16 minute with some json data. These json data are separated by #### lines in between. Snippet of the file is :

[{"accountId":"abc","items":[{"serviceName":"XYZ","dataCenter":"TG","startTimeUtc":"2017-04-05T19:57:33.509+0000","endTimeUtc":"2017-04-05T19:57:33.509+0000","usage":[{"resourceName":"XYZ_EE_PAAS_GATEWAYS","quantity":7,"units":"number"}]}]},{"accountId":"XYZp1cm9mbe","items":[{"serviceName":"XYZ","dataCenter":"TG","startTimeUtc":"2017-04-05T19:57:33.509+0000","endTimeUtc":"2017-04-05T19:57:33.509+0000","usage":[{"resourceName":"XYZ_EE_PAAS_GATEWAYS","quantity":6,"units":"number"}]}]}]
######################
[{"accountId":"abc","items":[{"serviceName":"XYZ","dataCenter":"TG","startTimeUtc":"2017-04-05T19:59:33.523+0000","endTimeUtc":"2017-04-05T19:59:33.523+0000","usage":[{"resourceName":"XYZ_EE_PAAS_GATEWAYS","quantity":7,"units":"number"}]}]},{"accountId":"XYZp1cm9mbe","items":[{"serviceName":"XYZ","dataCenter":"TG","startTimeUtc":"2017-04-05T19:59:33.523+0000","endTimeUtc":"2017-04-05T19:59:33.523+0000","usage":[{"resourceName":"XYZ_EE_PAAS_GATEWAYS","quantity":6,"units":"number"}]}]}]
######################
[{"accountId":"abc","items":[{"serviceName":"XYZ","dataCenter":"TG","startTimeUtc":"2017-04-05T20:01:33.531+0000","endTimeUtc":"2017-04-05T20:01:33.531+0000","usage":[{"resourceName":"XYZ_EE_PAAS_GATEWAYS","quantity":7,"units":"number"}]}]},{"accountId":"XYZp1cm9mbe","items":[{"serviceName":"XYZ","dataCenter":"TG","startTimeUtc":"2017-04-05T20:01:33.531+0000","endTimeUtc":"2017-04-05T20:01:33.531+0000","usage":[{"resourceName":"XYZ_EE_PAAS_GATEWAYS","quantity":6,"units":"number"}]}]}]
######################

This file gets updated every 15-16 minute with new entry. I want to read the file and store the latest entry excluding the #### line in a json object. How to do it in java ? I don't want to use the 15 min interval as it is not constant.

My simple requirement is at any point of time I would read the file and want to retrieve the last json in above the ### line.

Two questions: 1st: Will there be only one line added in each update? 2nd: How large will your file grow? The answers to these question will have great impact on the solution you should choose: If the file is relatively small it might be no great overhead to reparse the complete file on each update. If only one line is added on each update you could use the update as trigger and only parse the last line. — mschenk74, Jun 22 '17 at 08:16
@mschenk74 1st: Will there be only one line added in each update? It depends, it may come as a single line entry or may come in as a formatted json entry in multi lines. 2nd: How large will your file grow? The file will have max 15 entries. — saurav, Jun 22 '17 at 08:20
Given this additional information of you I would say that there is no need for optimization in your case. So I would start with the answer of Sebastion Kruse below. Do not optimize a solution unless it is too slow. Beware: When parsing the complete file on each update you will have to code some logic to detect wich lines you have already parsed in previous updates. — mschenk74, Jun 22 '17 at 08:25

Sebastian Kruse · Answer 1 · 2017-06-22T11:31:10.557

With Java 8, you can do it like this:

public JsonObject retrieveLastEntry(Path path) throws IOException {
  String[] jsonLines = Files.lines(path)
    .filter(line -> !line.equals("######################")
    .toArray();
  String lastJsonLine = jsonLines[jsonLines.length - 1];
  return MyFavoriteJsonParser.parse(lastJsonLine);
}

MyFavoriteJsonParser refers to whatever JSON library you want to use (maybe have a look at this question). There might be few performance considerations here. If your file is very large (considerably more than a few MB), then the .toArray() call maybe not right for you. In fact, if performance is extremely crucial, you might even need to consider parsing the file backwards. But the golden rule for performance optimization is to go with a simple solution first and see if (and where) it might be not performant enough.

If your JSON goes across lines, however, the Stream API is not the best choice. In that case, a regular iteration comes to the rescue:

public JsonObject retrieveLastEntry(File file) throws IOException {
  String lastJson = "";
  StringBuffer sb = new StringBuffer();
  try (BufferedReader reader = new BufferedReader(new InputStreamReader(new FileReader(file), "UTF-8")))) {
    String line;
    while ((line = reader.readLine()) != null) {
      if (line.equals("######################") {
        lastJson = sb.toString(); sb.setLength(0);
      } else {
        sb.append(line).append('\n');
      }
   }
   return MyFavoriteJsonParser.parse(lastJsonLine);
}

The basic idea is to aggregate lines between the ###... and put them into a variable whenever a new separator is reached. You still might want to consider the case of having no entry at all and handle IOExceptions properly.

I think this is pretty much the idiomatic way of doing it.

Thanks for answer. Its very much helpful. My only concern is what to do if the whole json is not in a single line, instead a multi line formatted json ? ###################### is the only separator and marks the end of an entry. — saurav, Jun 22 '17 at 09:36
That's a good point. Your example was not giving that away, but I update for that. Speaking of which, further considerations are: consecutive separators, will there always be a terminal separator? — Sebastian Kruse, Jun 22 '17 at 11:32
Yes, I have updated the question too. The separator will be there after each entry. I.e lets say the file is empty now and 1 entry added, so it will look like this : This is the first entry ######################### After the second entry : This is the first entry ######################### This is thes econd entry ######################### In comment I am not able to format it properly. — saurav, Jun 22 '17 at 15:29

How to parse a text file with separated json objects in Java?

1 Answers1