How to resume download with Spring RestTemplate?

Question

I'm using RestTemplate to download a file from a Nexus Server (about 350 mb). The code provided in this post works well for that purpose:

RestTemplate restTemplate // = ...;

// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
        .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
    // Here I write the response to a file but do what you like
    Path path = Paths.get("some/path");
    Files.copy(response.getBody(), path);
    return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);

I'd like to check if file exists, then attempt to resume download:

...
if(Files.exists(path)) {
    log.info("{} exists. Attempting to resume download", path);
    Files.write(path, StreamUtils.copyToByteArray(response.getBody()), StandardOpenOption.APPEND);
} else {
    Files.copy(response.getBody(), path);
}

But this just just results in OOM error:

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source) ~[na:1.8.0_191]
    at java.io.ByteArrayOutputStream.grow(Unknown Source) ~[na:1.8.0_191]
    at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source) ~[na:1.8.0_191]
    at java.io.ByteArrayOutputStream.write(Unknown Source) ~[na:1.8.0_191]
    ...

I have tested the call using the Range header with curl and am certain Nexus supports that. I am setting it this way:

long bytes = path.toFile().length();
...
request.getHeaders().setRange(Arrays.asList(HttpRange.createByteRange(bytes)));

I'm guessing the memory error above occurs because InputStream blocks. So I try to use Channel/Buffer instead:

...
try {
    if(Files.exists(path)) {
        log.info("{} exists. Attempting to resume download", path);
        ReadableByteChannel channel = Channels.newChannel(response.getBody());

        FileChannel fileChannel = FileChannel.open(path, StandardOpenOption.APPEND);
        fileChannel.tryLock();
        ByteBuffer buffer = ByteBuffer.allocate(4096);
        int bytesRead = 0;
        while((bytesRead = channel.read(buffer)) != -1) {
            fileChannel.write(buffer);
            buffer.clear();
        }

        fileChannel.close();    
    } else {
        Files.copy(response.getBody(), path);
    }
...

This at least writes some data to the file but still fails. I don't have much experience with java.nio facilities so any help is appreciated.

*edit: Any answers are appreciated but I am forced to use JDK 8 for this project.

score 0 · Accepted Answer · answered Feb 08 '19 at 19:58

0

You need the following lines of code if you are working with large files to ensure the stream is not read in memory:

SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);     
restTemplate.setRequestFactory(requestFactory);

answered Feb 08 '19 at 19:58

Alexander Petrov

9,204
31
70

Thanks Alexandar, I have been using HttpComponentsClientHttpRequestFactory but the same logic applies. – lucidMonkey Feb 08 '19 at 22:36
@lucidMonkey so is it working ? Setting the buffering to false. – Alexander Petrov Feb 08 '19 at 22:47
It is a necessary addition. Another critical problem with the code above using filechannel and buffer is that the line 'buffer.flip()' was missing and is necessary. With those two changes it seems to be working well. – lucidMonkey Feb 09 '19 at 01:53

How to resume download with Spring RestTemplate?

1 Answers1