0

I have a method that is receiving an InputStream of data in JSON format. Using Jackson's ObjectMapper, I am able to convert the InputStream into a JsonNode that I can edit, like so:

JsonNode revisions = mapper.readTree(data);

From there, I am able to iterate through each element and make my changes. In doing so, though, I am storing all the elements in a list and then converting the list to a Stream. I would prefer to operate on each element one at a time from the InputStream, that way I don't have to store it all in memory.

Here's what I have:

public Stream<Revision> jsonToRevisionObjects(InputStream allData) throws IOException {
    // convert the InputStream to a JsonNode
    JsonNode revisions = mapper.readTree(allData);

    List<Revision> newRevisions = new ArrayList<>();
    for (JsonNode revision : revisions.get("results")) {
        // create Revision objects and add them to newRevisions
    }
    return newRevisions.stream();
}

This essentially defies the point of even using Stream since I'm storing all the new Revision objects into memory. Instead, I'd like to read one element at a time and send it off to the stream before loading in the next element. Is there a way of doing this? Based on surrounding code, the input parameter will always be an InputStream (there lies the problem) and the return type will always be Stream.

this might be possible if I was able to convert an InputStream into a Stream and do the following:

return allDataStream.map(rev -> {
    // create Revision object
       });

but I'm not sure how to get to that point if it's a possibility.

user2869231
  • 1,431
  • 5
  • 24
  • 53
  • I'm assuming that `allData` is a list of JSON objects, is that correct? I am wondering, in that case, if it is acceptable to read each object into memory one at a time, rather than the whole list – one stevy boi Apr 29 '16 at 17:05
  • yes, which is what I'd like to do instead. It would be easy if the data was coming in as a stream (then I could just use map), but I'm not sure what to do since it's coming in as an InputStream – user2869231 Apr 29 '16 at 17:06
  • I found the following response to a similar question, which may be helpful http://stackoverflow.com/a/31275144/3529744 – one stevy boi Apr 29 '16 at 17:07
  • "without storing data into memory" that's not possible, AFAIK you have to read a stream to know what's "inside" and "edit" it –  Apr 29 '16 at 17:13
  • ok sorry, not storing everything in memory. In other words, 1 element at a time. I will update – user2869231 Apr 29 '16 at 17:42

1 Answers1

0

To use streaming reads, you must use JsonParser, either directly, or by passing it to ObjectMapper/ObjectReader. If so, you may read sub-trees as JsonNode if you want to.

To construct a JsonParser from InputStream is simple:

JsonParser p = mapper.getFactory().createParser(inputStream);

but operation after this varies; you can either read token-stream directly from JsonParser, or ask ObjectMapper or ObjectReader to read next "value" from stream. And then the structure of JSON data matters; you may need to advance parser's stream (nextToken()) if you want to avoid reading all the contents.

StaxMan
  • 113,358
  • 34
  • 211
  • 239
  • hmm, it'll take a bit of work but I'll try some things. Do you know if there's a way to get chunks of json at a time? As in.. I have a list of JSON objects and want to grab a full object at a time in that list (so let's say 3 key:value pairs). I can keep iterating one at a time doing nextField and nextValue, but I'd prefer to just create an entire map at a time – user2869231 Apr 29 '16 at 22:33
  • @user2869231 not quite sure I understand your question here. Jackson has to decode parts of underlying JSON content to know what is what, and this is exposed as a `JsonToken` stream, traversable using `JsonParser.nextToken()` (and similar methods). With that you can do anything you want. At higher level, `ObjectReader`/`ObjectMapper` your choices are more limited, as they read sub-trees. – StaxMan May 03 '16 at 06:15