1

When using a RestTemplate to talk to an external service, I've seen more than once OutOfMemory errors on our application because the service streams gigs of data (due to a bad implementation on their side, in case of errors they they were sending back big stacktraces in each element of the array, which usually contains few thousand). It was ending in about 6gb of data, serialized by jackson in our app and totally exploding the Xmx of the jvm.

I've looked around but there don't seem to be any way to protect against this kind of even, i.e. aborting the request when the streamed response exceed a given size.

Is there a solution to this? We are using apache's httpcomponents httpclient 4.5.5, but any other underlying implementation would be acceptable.

Besides RestTemplates, a solution for Spring's reactive WebClient would also be welcome.

petronius
  • 451
  • 3
  • 11
  • I'm aware that in httpclient 4.3 there was an option [http.connection.max-line-length](https://hc.apache.org/httpcomponents-client-4.2.x/tutorial/html/connmgmt.html) but it's deprecated and I could find the replacement. – petronius Apr 24 '20 at 09:35

2 Answers2

1

This has to be enforced at the underlying HTTP client library (spring supports different ones like JDK client, apache client, okHTTP..)

Here you talking about apache-httpcomponent , did you check this HttpEntity.getContent() ? It actually returns an InputStream that you can read by yourself and determine when the size has been exceeded..

https://hc.apache.org/httpcomponents-core-4.4.x/httpcore/apidocs/org/apache/http/HttpEntity.html

CodeScale
  • 3,046
  • 1
  • 12
  • 20
  • Very interesting: "The content is received from a stream, or generated on the fly." That could definitely do it, I'm gonna give it a shot. – petronius Apr 24 '20 at 09:37
  • Great let me know – CodeScale Apr 24 '20 at 15:29
  • Well, so i'm guessing this solution works, but it's way too cumbersome and i gave up. It's not possible to use spring's ```RestTemplate``` to get the ```HttpEntity```, so all the current beans/tests/logic needed to be reworked. I've actually found the new ```max-line-length``` for http components 4.3+: it's in ```org.apache.http.config.MessageConstraints``` to be used in ```org.apache.http.config.ConnectionConfig```. It *does* work, but with a totally unusable behaviour: no exception is thrown if the response size exceeds the maximum set, instead the restTemplate returns an empty element. – petronius Apr 27 '20 at 16:16
  • But you can load an inputstream with `RestTemplate` . Look at this https://riptutorial.com/spring/example/20722/downloading-a-large-file – CodeScale Apr 27 '20 at 18:20
  • most important part is `ResponseExtractor` – CodeScale Apr 27 '20 at 18:21
0

For the records, here is the final solution. The problematic was to load a list of objects that can be very big, using pagination (through elastic search scroll api).

ResponseExtractor<Car[]> responseExtractor = responseEntity -> {
    long pageContentLengthInBytes = responseEntity.getHeaders().getContentLength();
    long presumableFreeMemoryInBytes = this.getAvailableFreeMemoryAmount();

    if (presumableFreeMemoryInBytes - TWENTY_MEGABYTES < pageContentLengthInBytes) {
        log.error("Not enough memory to store the page ({} avaiable, content-length={}, trashing it", presumableFreeMemoryInBytes, pageContentLengthInBytes);
        responseEntity.close();
        return null;
    }
    return objectMapper.readValue(responseEntity.getBody(), Car[].class);
};

Car[] responseEntities = this.restTemplate.execute(uri, HttpMethod.GET, null, responseExtractor);

   /**
     * Returns the current amount of memory which may be allocated until an out-of-memory error occurs.
     * see https://stackoverflow.com/a/12807848/8836232
     */
    private long getAvailableFreeMemoryAmount() {
        long allocatedMemory = (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory());
        return Runtime.getRuntime().maxMemory() - allocatedMemory;
    }
petronius
  • 451
  • 3
  • 11