Preventing the Jersey client from causing an outofmemory error when posting large files

Question

When positing large files as an InputStream using the Jersey client, it appears that the entire contents of the file is being buffered into memory before being sent to the server. This causes a problem with large files as the JVM runs out of heap space. How do I prevent this behaviour in the Jersey client? JAX-RS resource methods on the server side don't seem to have this problem when sending data.

For example:

WebResource dataUploadResource = buildDataUploadResource();
dataUploadResource.type(getMimeType()).put(getLargeInputStream());

Answer 1

In order to prevent this behaviour you need configure the Jersey client to use chunked encoding1 for the request. This removes the need to set a Content-Length header and will stream from the the supplied InputStream without buffering the entire contents in memory.

By default Jersey uses the JDK's HttpURLConection class to handle HTTP requests and responses. Unfortunately this has some bugs related to chunked encoding transfers. Fortunately, Jersey has extension points to allow different HTTP client implementations to be used. One such implementation is based on the Apache Http Client2.

Two implementations of the apache htpp client handler exist, one supports the now end of life 3.x version, the other uses the newer 4.x version. For our project we used implementation based on the older (3.1) version. The library is available in Maven Central under the 'contribs' sub-group.

<dependency>
    <groupId>com.sun.jersey.contribs</groupId>
    <artifactId>jersey-apache-client</artifactId>
    <version>1.14</version>
</dependency>

Next you have to initialise your Jersey client to use the new implementation:

Client jerseyClient = ApacheHttpClient.create(getClientConfig());

In order to enable chunked encoding, you'll have to set the chunked encoding size on the client configuration as it's not enabled by default:

private ClientConfig getClientConfig() {
   ClientConfig config = new DefaultClientConfig();

   config.getProperties().put(
            DefaultApacheHttpClientConfig.PROPERTY_CHUNKED_ENCODING_SIZE, 0);
   return config;
}

As long as this property is not null, chunked encoding will be used. In fact, version 1.14 ignores the encoding size as specifying a size is not supported by the underlying apache commons-httpclient library.