Getting InputStream with RestTemplate

Question

I am using URL class to read an InputStream from it. Is there any way I can use RestTemplate for this?

InputStream input = new URL(url).openStream();
JsonReader reader = new JsonReader(new InputStreamReader(input, StandardCharsets.UTF_8.displayName())); 

How can I get InputStream with RestTemplate instead of using URL?

Answer 1

The previous answers are not wrong, but they don't go into the depth that I like to see. There are cases when dealing with low level InputStream is not only desirable, but necessary, the most common example being streaming a large file from source (some web server) to destination (a database). If you try to use a ByteArrayInputStream, you will be, not so surprisingly, greeted with OutOfMemoryError. Yes, you can roll your own HTTP client code, but you'll have to deal with erroneous response codes, response converters etc. If you are already using Spring, looking to RestTemplate is a natural choice.

As of this writing, spring-web:5.0.2.RELEASE has a ResourceHttpMessageConverter that has a boolean supportsReadStreaming, which if set, and the response type is InputStreamResource, returns InputStreamResource; otherwise it returns a ByteArrayResource. So clearly, you're not the only one that asked for streaming support.

However, there is a problem: RestTemplate closes the response soon after the HttpMessageConverter runs. Thus, even if you asked for InputStreamResource, and got it, it's no good, because the response stream has been closed. I think this is a design flaw that they overlooked; it should've been dependent on the response type. So unfortunately, for reading, you must consume the response fully; you can't pass it around if using RestTemplate.

Writing is no problem though. If you want to stream an InputStream, ResourceHttpMessageConverter will do it for you. Under the hood, it uses org.springframework.util.StreamUtils to write 4096 bytes at a time from the InputStream to the OutputStream.

Some of the HttpMessageConverter support all media types, so depending on your requirement, you may have to remove the default ones from RestTemplate, and set the ones you need, being mindful of their relative ordering.

Last but not the least, implementations of ClientHttpRequestFactory has a boolean bufferRequestBody that you can, and should, set to false if you are uploading a large stream. Otherwise, you know, OutOfMemoryError. As of this writing, SimpleClientHttpRequestFactory (JDK client) and HttpComponentsClientHttpRequestFactory (Apache HTTP client) support this feature, but not OkHttp3ClientHttpRequestFactory. Again, design oversight.

Edit: Filed ticket SPR-16885.

Answer 2

Spring has a org.springframework.http.converter.ResourceHttpMessageConverter. It converts Spring's org.springframework.core.io.Resource class. That Resource class encapsulates a InputStream, which you can obtain via someResource.getInputStream().

Putting this all together, you can actually get an InputStream via RestTemplate out-of-the-box by specifying Resource.class as your RestTemplate invocation's response type.

Here is an example using one of RestTemplate's exchange(..) methods:

import org.springframework.web.client.RestTemplate;
import org.springframework.http.HttpMethod;
import org.springframework.core.io.Resource;

ResponseEntity<Resource> responseEntity = restTemplate.exchange( someUrlString, HttpMethod.GET, someHttpEntity, Resource.class );

InputStream responseInputStream;
try {
    responseInputStream = responseEntity.getBody().getInputStream();
}
catch (IOException e) {
    throw new RuntimeException(e);
}

// use responseInputStream

Answer 3

You should not get the InputStream directly. RestTemplate is meant to encapsulate processing the response (and request) content. Its strength is handling all the IO and handing you a ready-to-go Java object.

One of RestTemplate's original authors, Brian Clozel, has stated:

RestTemplate is not meant to stream the response body; its contract doesn't allow it, and it's been around for so long that changing such a basic part of its behavior cannot be done without disrupting many applications.

You'll need to register appropriate HttpMessageConverter objects. Those will have access to the response's InputStream, through an HttpInputMessage object.

As Abdull suggests, Spring does come with an HttpMessageConverter implementation for Resource which itself wraps an InputStream, ResourceHttpMessageConverter. It doesn't support all Resource types, but since you should be programming to interfaces anyway, you should just use the superinterface Resource.

The current implementation (4.3.5), will return a ByteArrayResource with the content of the response stream copied to a new ByteArrayInputStream which you can access.

You don't have to close the stream. The RestTemplate takes care of that for you. (This is unfortunate if you try to use a InputStreamResource, another type supported by the ResourceHttpMessageConverter, because it wraps the underlying response's InputStream but is closed before it can be exposed to your client code.)

Answer 4

I encountered the same issue and solved it by extending RestTemplate and closing the connection only after the stream is read.

you can see the code here: https://github.com/ItamarBenjamin/stream-rest-template

Answer 5

Very simple, yet efficient solution would be using ResponseExtractor. It's especially useful when you want to operate on very large InputStream and your RAM is limited.

Here is how you should be implementing it:

public void consumerInputStreamWithoutBuffering(String url, Consumer<InputStream> streamConsumer) throws IOException {

    final ResponseExtractor responseExtractor =
            (ClientHttpResponse clientHttpResponse) -> {
                streamConsumer.accept(clientHttpResponse.getBody());
                return null;
            };

    restTemplate.execute(url, HttpMethod.GET, null, responseExtractor);
}

And then, invoke the method anywhere you need:

Consumer<InputStream> doWhileDownloading = inputStream -> {
                //Use inputStream for your business logic...
};

consumerInputStreamWithoutBuffering("https://localhost.com/download", doWhileDownloading);

Please, be aware of the following common pitfall:

public InputStream getInputStreamFromResponse(String url) throws IOException {

    final ResponseExtractor<InputStream> responseExtractor =
            clientHttpResponse -> clientHttpResponse.getBody();

    return restTemplate.execute(url, HttpMethod.GET, null, responseExtractor);
}

Here InputStream will be closed before you can access it

Answer 6

I solve it by doing that. I hope it will help you all.

    @GetMapping("largeFile")
    public ResponseEntity<InputStreamResource> downloadLargeFile(
            @RequestParam("fileName") String fileName
    ) throws IOException {

        RestTemplate restTemplate = new RestTemplate();

        // Optional Accept header
        RequestCallback requestCallback = request -> request.getHeaders()
                .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

        // Streams the response instead of loading it all in memory
        ResponseExtractor<InputStreamResource> responseExtractor = response -> {
            // Here I write the response to a file but do what you like
            Path path = Paths.get("tmp/" + fileName);
            Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
            return new InputStreamResource(new FileInputStream(String.format("tmp/%s", fileName)));
        };

        InputStreamResource response = restTemplate.execute(
            String.format("http://%s:%s/file/largeFileRestTemplate?fileName=%s", host, "9091", fileName),
            HttpMethod.GET,
            requestCallback,
            responseExtractor
        );

        return ResponseEntity
            .ok()
            .header(HttpHeaders.CONTENT_DISPOSITION, String.format("attachment; filename=%s", fileName))
            .body(response);
    }

Answer 7

Thanks to Abhijit Sarkar's answer for leading the way.

I needed to download a heavy JSON stream and break it into small streamable manageable pieces of data. The JSON is composed of objects that have big properties: such big properties can be serialized to a file, and thus removed from the unmarshalled JSON object.

Another use case is to download a JSON stream object by object, process it like a map/reduce algorythm and produce a single output without having to load the whole stream in memory.

Yet another use case is to read a big JSON file and only pick a few objects based on a condition, while unmarshalling to Plain Old Java Objects.

Here is an example: we'd like to stream a very huge JSON file that is an array, and we'd like to retrieve only the first object in the array.

Given this big file on a server, available at http://example.org/testings.json :

[
    { "property1": "value1", "property2": "value2", "property3": "value3" },
    { "property1": "value1", "property2": "value2", "property3": "value3" },
    ... 1446481 objects => a file of 104 MB => take quite long to download...
]

Each row of this JSON array can be parsed as this object:

@lombok.Data
public class Testing {
    String property1;
    String property2;
    String property3;
}

You need this class make the parsing code reusable:

import com.fasterxml.jackson.core.JsonParser;
import java.io.IOException;
@FunctionalInterface
public interface JsonStreamer<R> {
    /**
     * Parse the given JSON stream, process it, and optionally return an object.<br>
     * The returned object can represent a downsized parsed version of the stream, or the result of a map/reduce processing, or null...
     *
     * @param jsonParser the parser to use while streaming JSON for processing
     * @return the optional result of the process (can be {@link Void} if processing returns nothing)
     * @throws IOException on streaming problem (you are also strongly encouraged to throw HttpMessageNotReadableException on parsing error)
     */
    R stream(JsonParser jsonParser) throws IOException;
}

And this class to parse:

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import lombok.AllArgsConstructor;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.HttpOutputMessage;
import org.springframework.http.MediaType;
import org.springframework.http.converter.HttpMessageConverter;

import java.io.IOException;
import java.util.Collections;
import java.util.List;

@AllArgsConstructor
public class StreamingHttpMessageConverter<R> implements HttpMessageConverter<R> {

    private final JsonFactory factory;
    private final JsonStreamer<R> jsonStreamer;

    @Override
    public boolean canRead(Class<?> clazz, MediaType mediaType) {
        return MediaType.APPLICATION_JSON.isCompatibleWith(mediaType);
    }

    @Override
    public boolean canWrite(Class<?> clazz, MediaType mediaType) {
        return false; // We only support reading from an InputStream
    }

    @Override
    public List<MediaType> getSupportedMediaTypes() {
        return Collections.singletonList(MediaType.APPLICATION_JSON);
    }

    @Override
    public R read(Class<? extends R> clazz, HttpInputMessage inputMessage) throws IOException {
        try (InputStream inputStream = inputMessage.getBody();
             JsonParser parser = factory.createParser(inputStream)) {
            return jsonStreamer.stream(parser);
        }
    }

    @Override
    public void write(R result, MediaType contentType, HttpOutputMessage outputMessage) {
        throw new UnsupportedOperationException();
    }

}

Then, here is the code to use to stream the HTTP response, parse the JSON array and return only the first unmarshalled object:

// You should @Autowire these:
JsonFactory jsonFactory = new JsonFactory();
ObjectMapper objectMapper = new ObjectMapper();
RestTemplateBuilder restTemplateBuilder = new RestTemplateBuilder();

// If detectRequestFactory true (default): HttpComponentsClientHttpRequestFactory will be used and it will consume the entire HTTP response, even if we close the stream early
// If detectRequestFactory false: SimpleClientHttpRequestFactory will be used and it will close the connection as soon as we ask it to

RestTemplate restTemplate = restTemplateBuilder.detectRequestFactory(false).messageConverters(
    new StreamingHttpMessageConverter<>(jsonFactory, jsonParser -> {

        // While you use a low-level JsonParser to not load everything in memory at once,
        // you can still profit from smaller object mapping with the ObjectMapper
        if (!jsonParser.isClosed() && jsonParser.nextToken() == JsonToken.START_ARRAY) {
            if (!jsonParser.isClosed() && jsonParser.nextToken() == JsonToken.START_OBJECT) {
                return objectMapper.readValue(jsonParser, Testing.class);
            }
        }
        return null;

    })
).build();

final Testing firstTesting = restTemplate.getForObject("http://example.org/testings.json", Testing.class);
log.debug("First testing object: {}", firstTesting);

Answer 8

You can pass in your own response extractor. Here is an example where I write out the json to disk in a streaming fashion -

        RestTemplate restTemplate = new RestTemplateBuilder().basicAuthentication("user", "their_password" ).build();

        int responseSize = restTemplate.execute(uri,
            HttpMethod.POST,
            (ClientHttpRequest requestCallback) -> {
                requestCallback.getHeaders().setContentType(MediaType.APPLICATION_JSON);
                requestCallback.getBody().write(body.getBytes());
            },
            responseExtractor -> {
                FileOutputStream fos  = new FileOutputStream(new File("out.json"));
                return StreamUtils.copy(responseExtractor.getBody(), fos);
            }
    )

Answer 9

As a variant you can consume response as bytes and than convert to stream

byte data[] = restTemplate.execute(link, HttpMethod.GET, null, new BinaryFileExtractor());
return new ByteArrayInputStream(data);

Extractor is

public class BinaryFileExtractor implements ResponseExtractor<byte[]> {

  @Override
  public byte[] extractData(ClientHttpResponse response) throws IOException {
    return ByteStreams.toByteArray(response.getBody());
  }
}