How to forward large files with RestTemplate?

Question

I have a web service call through which zip files can be uploaded. The files are then forwarded to another service for storage, unzipping, etc. For now the file is stored on the file system, then a FileSystemResource is built.

Resource zipFile = new FileSystemResource(tempFile.getAbsolutePath());

I could use a ByteStreamResource in order to save time(the saving of the file on disk is not needed before forwarding) but for that i need to build a byte array. In case of large files I will get an "OutOfMemory : java heap space" error.

ByteArrayResource r = new ByteArrayResource(inputStream.getBytes());

Any solutions to forwarding files without getting an OutOfMemory error using RestTemplate?

Answer 1

Edit: The other answers are better (use Resource) https://stackoverflow.com/a/36226006/116509

My original answer:

You can use execute for this kind of low-level operation. In this snippet I've used Commons IO's copy method to copy the input stream. You would need to customize the HttpMessageConverterExtractor for the kind of response you're expecting.

final InputStream fis = new FileInputStream(new File("c:\\autoexec.bat")); // or whatever
final RequestCallback requestCallback = new RequestCallback() {
     @Override
    public void doWithRequest(final ClientHttpRequest request) throws IOException {
        request.getHeaders().add("Content-type", "application/octet-stream");
        IOUtils.copy(fis, request.getBody());
     }
};
final RestTemplate restTemplate = new RestTemplate();
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);     
restTemplate.setRequestFactory(requestFactory);     
final HttpMessageConverterExtractor<String> responseExtractor =
    new HttpMessageConverterExtractor<String>(String.class, restTemplate.getMessageConverters());
restTemplate.execute("http://localhost:4000", HttpMethod.POST, requestCallback, responseExtractor);

(Thanks to Baz for pointing out you need to call setBufferRequestBody(false) or it will defeat the point)

Answer 2

The only part of @artbristol's answer you really need is this (which you can set up as a RestTemplate Spring bean):

final RestTemplate restTemplate = new RestTemplate();
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);     
restTemplate.setRequestFactory(requestFactory);     

After that, I think just using a FileSystemResource as your request body will do the right thing.

I've also used an InputStreamResource successfully this way, for cases where you already have the data as an InputStream and don't need to consume it multiple times.

In my case, we had gzipped our files and wrapped a GZipInputStream in an InputStreamResource.

Answer 3

I think that the above answer has unnecessary code - you don't need to make an anonymous RequestCallback inner class, and you don't need to use IOUtils from apache.

I spent a bit of time researching a similar solution to yours and this is what I came up with:

You can accomplish your goal much simpler by using the Spring Resource Interface and RestTemplate.

RestTemplate restTemplate = new RestTemplate();

SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);

File file = new File("/whatever");

HttpEntity<FileSystemResource> requestEntity = new HttpEntity<>(new FileSystemResource(file));
ResponseEntity e = restTemplate.exchange("http://localhost:4000", HttpMethod.POST, requestEntity, Map.class);

(This example assumes that the response from where you are POSTing to is JSON. But, this can easily be changed by changing the return type class... set to Map.class above)

Answer 4

One additional thing to note when doing large uploads is presence of Spring Actuator. That automatically injects MetricsRestTemplateCustomizer which will cause your request to be buffered, even if you set setBufferRequestBody(false) on your factory.

Two ways to get around it, probably more options:

1. Disable the customizer globally -

@SpringBootApplication(exclude = HttpClientMetricsAutoConfiguration.class)

2. Disable for only some of the rest templates, if you're building a restTemplate via restTemplateBuilder.

            var restTemplate = restTemplateBuilder.requestFactory(() ->
                    {
                        var reqFact = new HttpComponentsClientHttpRequestFactory(HttpClientBuilder.create()//
                                .setDefaultRequestConfig(RequestConfig.custom().build())//
                                .build());
                        reqFact.setBufferRequestBody(false);
                        return reqFact;
                    })//
                    .setBufferRequestBody(false)//
                    //these are to disable any possible interceptor/customizer that might force buffering on the request
                    //like the MetricsRestTemplateCustomizer
                    .interceptors(List.of())//
                    .customizers(List.of())//
                    .build();