Download Large file from server using REST template Java Spring MVC

Question

I have a REST service which sends me a large ISO file ,there are no issues in the REST service . Now I have written a Web application which calls the rest service to get the file ,on the client(web app) side I receive a Out Of memory Exception.Below is my code

HttpHeaders headers = new HttpHeaders();//1 Line

    headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));//2 Line
    headers.set("Content-Type","application/json");//3 Line
    headers.set("Cookie", "session=abc");//4 Line
    HttpEntity statusEntity=new HttpEntity(headers);//5 Line
    String uri_status=new String("http://"+ip+":8080/pcap/file?fileName={name}");//6 Line

    ResponseEntity<byte[]>resp_status=rt.exchange(uri_status, HttpMethod.GET, statusEntity, byte[].class,"File5.iso");//7 Line

I receive out of memory exception at 7 line ,I guess i will have to buffer and get in parts ,but dont know how can i get this file from the server ,the size of the file is around 500 to 700 MB . Can anyone please assist .

Exception Stack:

  org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause

java.lang.OutOfMemoryError: Java heap space
    java.util.Arrays.copyOf(Arrays.java:3236)
    java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
    org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
    org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:58)
    org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:1)
    org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153)
    org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:81)
    org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:627)
    org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1)
    org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:454)
    org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409)
    org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385)
    com.pcap.webapp.HomeController.getPcapFile(HomeController.java:186)

My Server Side REST Service Code which is working fine is

@RequestMapping(value = URIConstansts.GET_FILE, produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE}, method = RequestMethod.GET)
public void getFile(@RequestParam(value="fileName", required=false) String fileName,HttpServletRequest request,HttpServletResponse response) throws IOException{



    byte[] reportBytes = null;
    File result=new File("/home/arpit/Documents/PCAP/dummyPath/"+fileName);

    if(result.exists()){
        InputStream inputStream = new FileInputStream("/home/arpit/Documents/PCAP/dummyPath/"+fileName); 
        String type=result.toURL().openConnection().guessContentTypeFromName(fileName);
        response.setHeader("Content-Disposition", "attachment; filename=" + fileName);
        response.setHeader("Content-Type",type);

        reportBytes=new byte[100];//New change
        OutputStream os=response.getOutputStream();//New change
        int read=0;
        while((read=inputStream.read(reportBytes))!=-1){
            os.write(reportBytes,0,read);
        }
        os.flush();
        os.close();






    }

Answer 1

Here is how I do it. Based on hints from this Spring Jira issue.

RestTemplate restTemplate // = ...;

// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
        .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
    // Here I write the response to a file but do what you like
    Path path = Paths.get("some/path");
    Files.copy(response.getBody(), path);
    return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);

From the aforementioned Jira issue:

Note that you cannot simply return the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.

Update for Spring 5

Spring 5 introduced the WebClient class which allows asynchronous (e.g. non-blocking) http requests. From the doc:

By comparison to the RestTemplate, the WebClient is:

• non-blocking, reactive, and supports higher concurrency with less hardware resources.
• provides a functional API that takes advantage of Java 8 lambdas.
• supports both synchronous and asynchronous scenarios.
• supports streaming up or down from a server.

To get WebClient in Spring Boot, you need this dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

For the moment, I'm sticking with RestTemplate because I don't want to pull in another dependency only to get access to WebClient.

Answer 2

As @bernie mentioned you can use WebClient to achieve this:

public Flux<DataBuffer> downloadFileUrl( ) throws IOException {

    WebClient webClient = WebClient.create();

    // Request service to get file data
    return Flux<DataBuffer> fileDataStream = webClient.get()
            .uri( this.fileUrl )
            .accept( MediaType.APPLICATION_OCTET_STREAM )
            .retrieve()
            .bodyToFlux( DataBuffer.class );
}

@GetMapping( produces = MediaType.APPLICATION_OCTET_STREAM_VALUE )
public void downloadFile( HttpServletResponse response ) throws IOException
{
    Flux<DataBuffer> dataStream = this.downloadFileUrl( );

    // Streams the stream from response instead of loading it all in memory
    DataBufferUtils.write( dataStream, response.getOutputStream() )
            .map( DataBufferUtils::release )
            .blockLast();
}

You can still use WebClient even if you don't have Reactive Server stack - Rossen Stoyanchev (a member of Spring Framework team) explains it quite well in the Guide to "Reactive" for Spring MVC Developers presentation. During this presentation, Rossen Stoyanchev mentioned that they thought about deprecating RestTemplate, but they have decided to postpone it after all, but it may still happen in the future!

The main disadvantage of using WebClient so far it's a quite steep learning curve (reactive programming), but I think there is no way to avoid in the future, so better to take a look on it sooner than latter.

Answer 3

This prevents loading the entire request into memory.

SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate rest = new RestTemplate(requestFactory);

For java.lang.OutOfMemoryError: Java heap space can be solved adding more memory to the JVM:

-Xmxn Specifies the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2 MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is chosen at runtime based on system configuration.

For server deployments, -Xms and -Xmx are often set to the same value. See Garbage Collector Ergonomics at http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html

Examples:

-Xmx83886080
-Xmx81920k
-Xmx80m

Probably the problem you have is not strictly related to the request you are trying to execute (download large file) but the memory allocated for the process is not enough.

Answer 4

A better version of above correct answer could be the below code. This method will send download request to another application or service acting as actual source of truth for downloaded information.

public void download(HttpServletRequest req, HttpServletResponse res, String url)
            throws ResourceAccessException, GenericException {
        try {
            logger.info("url::" + url);
            if (restTemplate == null)
                logger.info("******* rest template is null***********************");
            RequestCallback requestCallback = request -> request.getHeaders()
                    .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

            // Streams the response instead of loading it all in memory
            ResponseExtractor<ResponseEntity<InputStream>> responseExtractor = response -> {

                String contentDisposition = response.getHeaders().getFirst("Content-Disposition");
                if (contentDisposition != null) {
                    // Temporary location for files that will be downloaded from micro service and
                    // act as final source of download to user
                    String filePath = "/home/devuser/fileupload/download_temp/" + contentDisposition.split("=")[1];
                    Path path = Paths.get(filePath);
                    Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);

                    // Create a new input stream from temporary location and use it for downloading
                    InputStream inputStream = new FileInputStream(filePath);
                    String type = req.getServletContext().getMimeType(filePath);
                    res.setHeader("Content-Disposition", "attachment; filename=" + contentDisposition.split("=")[1]);
                    res.setHeader("Content-Type", type);

                    byte[] outputBytes = new byte[100];
                    OutputStream os = res.getOutputStream();
                    int read = 0;
                    while ((read = inputStream.read(outputBytes)) != -1) {
                        os.write(outputBytes, 0, read);
                    }
                    os.flush();
                    os.close();
                    inputStream.close();
                }
                return null;
            };
            restTemplate.execute(url, HttpMethod.GET, requestCallback, responseExtractor);
        } catch (Exception e) {
            logger.info(e.toString());
            throw e;
        }
    }

Answer 5

You should use multipart file attachment, so the file stream isn't load into memory. In this example, I use a rest service implemented with Apache CXF.

...
import org.apache.cxf.jaxrs.ext.multipart.Attachment;
...

@Override
@Path("/put")
@Consumes("multipart/form-data")
@Produces({ "application/json" })
@POST
public SyncResponseDTO put( List<Attachment> attachments) {
    SyncResponseDTO response = new SyncResponseDTO();
    try {
        for (Attachment attr : attachments) {
            log.debug("get input filestream: " + new Date());
            InputStream is = attr.getDataHandler().getInputStream();