Spring RestTemplate streaming response into another request

Question

I am trying to stream the result of a file download directly into another post using spring's RestTemplate

My current approach is the following:

   ResponseEntity<InputStreamResource> downloadResponse = restTemplate.getForEntity(fileToDownloadUri, InputStreamResource.class);

   InputStreamResource imageInputStreamResource = downloadResponse.getBody();

   ResponseEntity<String> response = restTemplate.exchange(storageUri, POST, new HttpEntity<>(imageInputStreamResource), String.class);

However, I get the following exception running the code above:

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://host:port/path/some.jpg": stream is closed; nested exception is java.io.IOException: stream is closed

    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:6
...
Caused by: java.io.IOException: stream is closed
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.ensureOpen(HttpURLConnection.java:3348)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3373)

It seems that the response is always closed as the final step of processing. With the response, the HttpURLConnection is closed, and the stream is no longer processable.

I would like to be able to implement this scenario without having to hold the file completely in memory or writing it to a file (as described here).

Any hints are highly appreciated.

possibly identical solution here http://stackoverflow.com/questions/15781885/how-to-forward-large-files-with-resttemplate — Redlab, Jan 25 '17 at 16:02
@Redlab there it is a `FileSystemResource` - this would work - my problem is that the `InputStream` is closed after the `GET` request is finished. So I cannot use it further down. — Mathias Dpunkt, Jan 25 '17 at 16:31

score 27 · Answer 1 · answered Aug 01 '17 at 17:53

27

If you want to forward the response directly without ever holding it in memory, you have to directly write to the response:

@RequestMapping(value = "/yourEndPoint")
public void processRequest(HttpServletResponse response) {
    RestTemplate restTemplate = new RestTemplate();

    response.setStatus(HttpStatus.OK.value());

    restTemplate.execute(
        fileToDownloadUri,
        HttpMethod.GET,
        (ClientHttpRequest requestCallback) -> {},
        responseExtractor -> {
            IOUtils.copy(responseExtractor.getBody(), response.getOutputStream());
            return null;
        });
}

answered Aug 01 '17 at 17:53

user1978011

3,419
25
38

Thank you for your snipet, it's works. But can we somehow pass header? – Sergey Ponomarev Apr 18 '18 at 12:28
1

Ok, I found a solution: (ClientHttpRequest requestCallback) -> { requestCallback.getHeaders().add(ACCEPT, TEXT_CSV_MEDIA_TYPE_VALUE); } – Sergey Ponomarev Apr 18 '18 at 12:33

Redlab · Answer 2 · 2017-01-26T13:11:08.273

Since you tell RestTemplate to expect InputStreamResource it will try and use an appropriate converter to convert your message to a InputStreamResource. ( I'm guessing there is none that handles this as you want )

You should be able to let it expect a Resource from where you can get an input stream and read that.

 import org.springframework.core.io.Resource;

 ResponseEntity<Resource> exchange = RestTemplate.exchange(url, HttpMethod.GET, new HttpEntity(httpHeaders), Resource.class);
 InputStream inputStream = exchange.getBody().getInputStream();

using this you can write the response to somewhere else. Files.write(inputStream, new File("./test.json")); wrote the file for me, so I assume the inputstream can also be used somewhere else. ( I used Spring 4.3.5 )

edit:

As the OP states, this will still load the file in memory. Behind the scene the InputStream is a ByteArrayInputStream.

The default RestTemplate and MessageConverters are not made for streaming content at all. You could write your own implementation of a org.springframework.web.client.ResponseExtractor and maybe a MessageConverter. In ResponseExtractor you have access to the org.springframework.http.client.ClientHttpResponse

imho for your use case, you might be better of using Apache Httpcomponents HttpClient where you find HttpEntity#writeTo(OutputStream).

I am afraid I do not get this - that is exactly what I am doing - `ResponseEntity downloadResponse = restTemplate.getForEntity(fileToDownloadUri, InputStreamResource.class);` but the inputStream is closed already after the call. Have you tried consuming the InputStream in your code above? — Mathias Dpunkt, Jan 26 '17 at 10:56
You are right, i mistook String and InputStreamResource (will correct answer). But nevertheless use org.springframework.core.io.Resource instead of InputStreamResource — Redlab, Jan 26 '17 at 10:59
Thanks for the input - if I make it return a `Resource` than `getForEntity` returns a `ByteArrayResource`. So I load the complete file into memory. Which is what I would like to avoid - that is why I tried `InputStreamResource`. I want to stream the response through into the post body. — Mathias Dpunkt, Jan 26 '17 at 12:02
you are right, I think RestTemplate is not ideal for your usecase — Redlab, Jan 26 '17 at 13:12

Spring RestTemplate streaming response into another request

2 Answers2

Linked