Is there an event-driven JSON REST client API for Java?

Question

I have a Java application which uses Spring's RestTemplate API to write concise, readable consumers of JSON REST services:

In essence:

 RestTemplate rest = new RestTemplate(clientHttpRequestFactory);
 ResponseEntity<ItemList> response = rest.exchange(url,
            HttpMethod.GET,     
            requestEntity,
            ItemList.class);

 for(Item item : response.getBody().getItems()) {
        handler.onItem(item);
 }

The JSON response contains a list of items, and as you can see, I have have an event-driven design in my own code to handle each item in turn. However, the entire list is in memory as part of response, which RestTemplate.exchange() produces.

I would like the application to be able to handle responses containing large numbers of items - say 50,000, and in this case there are two issues with the implementation as it stands:

1. Not a single item is handled until the entire HTTP response has been transferred - adding unwanted latency.
2. The huge response object sits in memory and can't be GC'd until the last item has been handled.

Is there a reasonably mature Java JSON/REST client API out there that consumes responses in an event-driven manner?

I imagine it would let you do something like:

 RestStreamer rest = new RestStreamer(clientHttpRequestFactory);

 // Tell the RestStreamer "when, while parsing a response, you encounter a JSON
 // element matching JSONPath "$.items[*]" pass it to "handler" for processing.
 rest.onJsonPath("$.items[*]").handle(handler);

 // Tell the RestStreamer to make an HTTP request, parse it as a stream.
 // We expect "handler" to get passed an object each time the parser encounters
 // an item.
 rest.execute(url, HttpMethod.GET, requestEntity);

I appreciate I could roll my own implementation of this behaviour with streaming JSON APIs from Jackson, GSON etc. -- but I'd love to be told there was something out there that does it reliably with a concise, expressive API, integrated with the HTTP aspect.

Answer 1

A couple of months later; back to answer my own question.

I didn't find an expressive API to do what I want, but I was able to achieve the desired behaviour by getting the HTTP body as a stream, and consuming it with a Jackson JsonParser:

  ClientHttpRequest request = 
        clientHttpRequestFactory.createRequest(uri, HttpMethod.GET);
  ClientHttpResponse response = request.execute();

  return handleJsonStream(response.getBody(), handler);

... with handleJsonStream designed to handle JSON that looks like this:

 { items: [ 
      { field: value; ... }, 
      { field: value, ... },
      ... thousands more ... 
 ] }

... it validates the tokens leading up to the start of the array; it creates an Item object each time it encounters an array element, and gives it to the handler.

 // important that the JsonFactory comes from an ObjectMapper, or it won't be
 // able to do readValueAs()
 static JsonFactory jsonFactory = new ObjectMapper().getFactory();

 public static int handleJsonStream(InputStream stream, ItemHandler handler) throws IOException {

     JsonParser parser = jsonFactory.createJsonParser(stream);

     verify(parser.nextToken(), START_OBJECT, parser);
     verify(parser.nextToken(), FIELD_NAME, parser);
     verify(parser.getCurrentName(), "items", parser);
     verify(parser.nextToken(), START_ARRAY, parser);
     int count = 0;
     while(parser.nextToken() != END_ARRAY) {
        verify(parser.getCurrentToken(), START_OBJECT, parser);
        Item item = parser.readValueAs(Item.class);
        handler.onItem(item);
        count++;
     }
     parser.close(); // hope it's OK to ignore remaining closing tokens.
     return count;
 }

verify() is just a private static method which throws an exception if the first two arguments aren't equal.

The key thing about this method is that no matter how many items there are in the stream, this method only every has a reference to one Item.

Answer 2

Is there no way to break up the request? It sounds like you should use paging. Make it so that you can request the first 100 results, the next 100 results, so on. The request should take a starting index and a count number. That's very common behavior for REST services and it sounds like the solution to your problem.

The whole point of REST is that it is stateless, it sounds like you're trying to make it stateful. That's anathema to REST, so you're not going to find any libraries written that way.

The transactional nature of REST is very much intentional by design and so you won't get around that easily. You'll fighting against the grain if you try.

Answer 3

you can try JsonSurfer which is designed to process json stream in event-driven style.

JsonSurfer surfer = JsonSurfer.jackson();
Builder builder = config();
builder.bind("$.items[*]", new JsonPathListener() {
        @Override
        public void onValue(Object value, ParsingContext context) throws Exception {
            // handle the value
        }
    });
surfer.surf(new InputStreamReader(response.getBody()), builder.build());

Answer 4

From what I've seen, wrapping frameworks (like you are using) make things easy by deserializing the response into an object. In your case, a collection of objects.

However, to use things in a streaming fashion, you may need to get at the underlying HTTP response stream. I am most familiar with Jersey, which exposes https://jersey.java.net/nonav/apidocs/1.5/jersey/com/sun/jersey/api/client/ClientResponse.html#getEntityInputStream()

It would be used by invoking

Client client = Client.create();
WebResource webResource = client.resource("http://...");
ClientResponse response = webResource.accept("application/json")
               .get(ClientResponse.class);
InputStream is = response.getEntityInputStream();

This provides you with the stream of data coming in. The next step is to write the streaming part. Given that you are using JSON, there are options at various levels, including http://wiki.fasterxml.com/JacksonStreamingApi or http://argo.sourceforge.net/documentation.html. They can consume the InputStream.

These don't really make good use of the full deserialization that can be done, but you could use them to parse out an element of a json array, and pass that item to a typical JSON object mapper, (like Jackson, GSON, etc). This becomes the event handling logic. You could spawn new threads for this, or do whatever your use case needs.

Answer 5

I won't claim to know all the rest frameworks out there (or even half) but I'm going to go with the answer

Probably Not

As noted by others this is not the way REST normally thinks of it's interactions. REST is a great Hammer but if you need streaming, you are (IMHO) in screwdriver territory, and the hammer might still be made to work, but it is likely to make a mess. One can argue that it is or is not consistent with REST all day long, but in the end I'd be very surprised to find a framework that implemented this feature. I'd be even more surprised if the feature is mature (even if the framework is) because with respect to REST your use case is an uncommon corner case at best.

If someone does come up with one I'll be happy to stand corrected and learn something new though :)

Perhaps it would be best to be thinking in terms of comet or websockets for this particular operation. This question may be helpful since you already have spring. (websockets are not really viable if you need to support IE < 10, which most commercial apps still require... sadly, I've got one client with a key customer still on IE 7 in my personal work)

Answer 6

You may consider Restlet.

http://restlet.org/discover/features

Supports asynchronous request processing, decoupled from IO operations. Unlike the Servlet API, the Restlet applications don't have a direct control on the outputstream, they only provide output representation to be written by the server connector.

Answer 7

The best way to achieve this is to use another streaming Runtime for JVM that allows reading response off websockets and i am aware of one called atmostphere

This way your large dataset is both sent and received in chunks on both side and read in the same manner in realtime withou waiting for the whole response.

This has a good POC on this: http://keaplogik.blogspot.in/2012/05/atmosphere-websockets-comet-with-spring.html

Server:

    @RequestMapping(value="/twitter/concurrency")
@ResponseBody
public void twitterAsync(AtmosphereResource atmosphereResource){
    final ObjectMapper mapper = new ObjectMapper();

    this.suspend(atmosphereResource);

    final Broadcaster bc = atmosphereResource.getBroadcaster();

    logger.info("Atmo Resource Size: " + bc.getAtmosphereResources().size());

    bc.scheduleFixedBroadcast(new Callable<String>() {

        //@Override
        public String call() throws Exception {

            //Auth using keaplogik application springMVC-atmosphere-comet-webso key
            final TwitterTemplate twitterTemplate = 
                new TwitterTemplate("WnLeyhTMjysXbNUd7DLcg",
                        "BhtMjwcDi8noxMc6zWSTtzPqq8AFV170fn9ivNGrc", 
                        "537308114-5ByNH4nsTqejcg5b2HNeyuBb3khaQLeNnKDgl8",
                        "7aRrt3MUrnARVvypaSn3ZOKbRhJ5SiFoneahEp2SE");

            final SearchParameters parameters = new SearchParameters("world").count(5).sinceId(sinceId).maxId(0);
            final SearchResults results = twitterTemplate.searchOperations().search(parameters);

            sinceId = results.getSearchMetadata().getMax_id();

            List<TwitterMessage> twitterMessages = new ArrayList<TwitterMessage>();

            for (Tweet tweet : results.getTweets()) {
                twitterMessages.add(new TwitterMessage(tweet.getId(),
                                                       tweet.getCreatedAt(),
                                                       tweet.getText(),
                                                       tweet.getFromUser(),
                                                       tweet.getProfileImageUrl()));
            }

            return mapper.writeValueAsString(twitterMessages);
        }

    }, 10, TimeUnit.SECONDS);
}

Client: Atmosphere has it's own javascript file to handle the different Comet/Websocket transport types and requests. By using this, you can set the Spring URL Controller method endpoint to the request. Once subscribed to the controller, you will receive dispatches, which can be handled by adding a request.onMessage method. Here is an example request with transport of websockets.

       var request = new $.atmosphere.AtmosphereRequest();
   request.transport = 'websocket';
   request.url = "<c:url value='/twitter/concurrency'/>";
   request.contentType = "application/json";
   request.fallbackTransport = 'streaming';

   request.onMessage = function(response){
       buildTemplate(response);
   };

   var subSocket = socket.subscribe(request);

   function buildTemplate(response){

     if(response.state = "messageReceived"){

          var data = response.responseBody;

        if (data) {

            try {
                var result =  $.parseJSON(data);

                $( "#template" ).tmpl( result ).hide().prependTo( "#twitterMessages").fadeIn();

            } catch (error) {
                console.log("An error ocurred: " + error);
            }
        } else {
            console.log("response.responseBody is null - ignoring.");
        }
    }
   }

It has support on all major browsers and native mobile clients Apple being pioneers of this technology:

As mentioned here excellent support for deployment environments on web and enterprise JEE containers:

http://jfarcand.wordpress.com/2012/04/19/websockets-or-comet-or-both-whats-supported-in-the-java-ee-land/