how to download large files chunk by chunk using large files using angular.js and java

Question

We have web application with client in agnular.js and server in java spring. I am working on functionality of downloading this log file i.e. logs.tar from client.

Currently we are using blob to download. Our issue is in case this log size becomes huge like greater than 2GB then while streaming it will create load on application memory. so i want way to download large files chunk by chunk and not required to load entire blob into memory. please suggest way out.

Server side java code -

   public ResponseEntity<?> downloadLogs(HttpServletRequest request) {
        File file = preferencesService.downloadLogs();
        if (file != null) {
            FileInputStream inputStream;
            try {
                inputStream = new FileInputStream(file);
                byte[] content = FileCopyUtils.copyToByteArray(inputStream);

                String filename = "com-logs.tar";
                HttpHeaders responseHeaders = new HttpHeaders();
                responseHeaders.add(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=" + filename);
                responseHeaders.add(HttpHeaders.CONTENT_TYPE, "application/octet-stream"

);

        return new ResponseEntity<byte[]>(content, responseHeaders, HttpStatus.OK);

        } catch (Exception e) {
            logger.error("Error while processing log file for download", e);
        }
    } else {
        logger.error("Failed to download logs");
    }
    return ResponseEntity.badRequest().build();
}

Client side Angular.js code -

this._service.downloadLogs().subscribe(
            success => {               
                var blb = new Blob([success], { 'type': "application/octet-stream" });
                if (window.navigator && window.navigator.msSaveOrOpenBlob) {
                    window.navigator.msSaveOrOpenBlob(blb, 'logs.tar');
                }
                else {
                    var link = document.createElement('a');
                    link.href = window.URL.createObjectURL(blb);
                    link.download = "logs.tar";
                    document.body.appendChild(link);
                    link.click();
                    document.body.removeChild(link);
                }
            });

New Server side java code -

public void downloadLogs(HttpServletResponse resonse) {
        File file = preferencesService.downloadLogs(id);       
        if (file != null) {
            try {
                resonse.setContentType("application/octet-stream");
                resonse.setHeader("Content-Disposition", "attachment;filename=" + file.getName());
                BufferedInputStream inStrem = new BufferedInputStream(new FileInputStream(file));
                BufferedOutputStream outStream = new BufferedOutputStream(resonse.getOutputStream());

                byte[] buffer = new byte[1024];
                int bytesRead = 0;
                while ((bytesRead = inStrem.read(buffer)) != -1) {
                    outStream.write(buffer, 0, bytesRead);
                }
                outStream.flush();
                inStrem.close();

            } 
            ...
        }

Consider **gzip** compression too (logs.tar.gz / GZipOutputStream). This can be done automatically by a filter if the browser acknowledges its deflate capabilty in its request header. But simply downloading a . tar,gz (.tgz) would be fine too. — Joop Eggen, Aug 07 '18 at 12:50
I don't know Angular, but why use Angular for something that plain HTML and the browser can just do fine by themselves? I think this might cause heavy load on the browser if the log file is large. — Christian Hujer, Aug 08 '18 at 12:51
For the server-side Spring code, see also this StackOverflow https://stackoverflow.com/questions/5673260/downloading-a-file-from-spring-controllers — Christian Hujer, Aug 08 '18 at 12:52

score 2 · Answer 1 · answered Aug 07 '18 at 13:08

2

The important thing is to not read the file into memory, but to pass the stream on:

public ResponseEntity<?> downloadLogs(HttpServletRequest request) {
    File file = preferencesService.downloadLogs();
    if (file != null) {
        try (InputStream inputStream = Files.newInputStream(file.toPath())) {
            InputStreamResource inputStreamResource =
                    new InputStreamResource(new inputStream);
            HttpHeaders responseHeaders = new HttpHeaders();
            //responseHeaders.setContentLength(Files.size(file.toPath()));
            responseHeaders.add(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename="
                    + filename);
            responseHeaders.add(HttpHeaders.CONTENT_TYPE, "application/octet-stream");
            return new ResponseEntity(inputStreamResource, responseHeaders, HttpStatus.OK);
        }
    }
    ...
}

Consider compression as this will hugely speed things up and cause less server load. Chunking, setting content length, deflate compression web filters, and so on should be looked into.

answered Aug 07 '18 at 13:08

Joop Eggen

107,315
7
83
138

Hi Joop Eggen, Thanks for reply ... will my client side code work with this as your code is sending inputStreamResource in ResponseEntity where i was sending content.. ?? what changes will i need to do at client side. – questp Aug 08 '18 at 05:48
The client side is independent. The difference is that earlier all was loaded in memory and then streamed to the client, and not it is immediately streamed to the client. Without memory overhead and delay of the loading into memory. – Joop Eggen Aug 08 '18 at 07:03
Hi Joop Eggen, since code given by you was giving some errors, i have added new code at server side under "New Server side java code -" ... . this code is working but everytime i download file it is adding file size into browser memory that is my main issue......can you please guide. – questp Aug 08 '18 at 10:21
The server's downloadLogs is fine. The client side AngularJS is indeed suboptimal. I cannot tell how to solve that. Letting the browser download without JavaScript would seem best. On the server side wrapping the output in a GZipOutputStream (.tar.gz) would help a bit. See also http://stackoverflow.com/questions/24080018/download-file-from-a-asp-net-web-api-method-using-angularjs (does not seem helpful) – Joop Eggen Aug 08 '18 at 11:19

how to download large files chunk by chunk using large files using angular.js and java

1 Answers1

Linked