how to download large files without memory issues in java

Question

When I am trying to download a large file which is of 260MB from server, I get this error: java.lang.OutOfMemoryError: Java heap space. I am sure my heap size is less than 252MB. Is there any way I can download large files without increasing heap size?

How I can download large files without getting this issue? My code is given below:

String path= "C:/temp.zip";   
response.addHeader("Content-Disposition", "attachment; filename=\"test.zip\""); 
byte[] buf = new byte[1024];   
try {   

             File file = new File(path);   
             long length = file.length();   
             BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));   
             ServletOutputStream out = response.getOutputStream();   

             while ((in != null) && ((length = in.read(buf)) != -1)) {   
             out.write(buf, 0, (int) length);   
             }   
             in.close();   
             out.close();

Are you able to increase the heap size on the JVM? ie: java -Xm512m -Xmx512m myClass — JJ., Aug 18 '11 at 11:41
yes, but i would like to know if i can do this without increasing jvm heap size, as we have a requirement to download large files of about 1gb to 10 gb — Hima Bindu, Aug 18 '11 at 11:50

JJ. · Accepted Answer · 2014-05-28T14:18:18.617

There are 2 places where I can see you could potentially be building up memory usage:

In the buffer reading your input file.
In the buffer writing to your output stream (HTTPOutputStream?)

For #1 I would suggest reading directly from the file via FileInputStream without the BufferedInputStream. Try this first and see if it resolves your issue. ie:

FileInputStream in = new FileInputStream(file);

instead of:

BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));

If #1 does not resolve the issue, you could try periodically flushing the output stream after so much data is written (decrease chunk size if necessary):

ie:

try
{
    FileInputStream fileInputStream  = new FileInputStream(file);
    byte[] buf=new byte[8192];
    int bytesread = 0, bytesBuffered = 0;
    while( (bytesread = fileInputStream.read( buf )) > -1 ) {
        out.write( buf, 0, bytesread );
        bytesBuffered += bytesread;
        if (bytesBuffered > 1024 * 1024) { //flush after 1MB
            bytesBuffered = 0;
            out.flush();
        }
    }
}
finally {
    if (out != null) {
        out.flush();
    }
}

One small issue is there with the code above. We need to add one more flush in the finally block if a try-catch which surrounds the code or at the end of the while loop. Else it will not flush the complete bytes. Else may cause some issues some times. — Arundev, May 28 '14 at 08:00

score 6 · Answer 2 · answered Aug 18 '11 at 11:41

6

Unfortunately you have not mentioned what type out is. If you have memory issues I guess it is ByteArrayOutpoutStream. So, replace it by FileOutputStream and write the byte you are downloading directly to file.

BTW, do not use read() method that reads byte-by-byte. Use read(byte[] arr) instead. This is much faster.

answered Aug 18 '11 at 11:41

AlexR

114,158
16
130
208

Judging by the "response.setContentType" and "response.setHeader" calls, this is a servlet - ie. he's streaming a file to a web client and "out" is the HttpOutputStream. – JJ. Aug 18 '11 at 11:47
thank you alex, but my main issue is with downloading larger files which can be 1gb, 10 gb size files. Here i get out of memory exception. How can i handle this without increasing the heap size.... – Hima Bindu Aug 18 '11 at 11:53
OK, now you have updated you code and the question is much more clear. I thought that your program is downloading file from remote URL but you are asking opposite question: your servlet provides ability to download file. So, I think that your problem is that you are not calling out.flush() after each out.write(), so all bytes you are reading are stored in memory and are not sent to network until you are closing the output stream. Try to flush and see what's happening. – AlexR Aug 18 '11 at 12:04

score 0 · Answer 3 · answered Aug 18 '11 at 11:52

0

First you can remove the (in != null) from your while statement, it's unnecessary. Second, try removing the BufferedInputStream and just do:

FileInputStream in = new FileInputStream(file);

answered Aug 18 '11 at 11:52

Rocky Pulley

22,531
20
68
106

score 0 · Answer 4 · answered Aug 18 '11 at 12:00

0

There's nothing wrong (in regard to memory usage) with the code you're show. Either the servlet container is configured to buffer the entire response (look at the web.xml configuration), or the memory is being leaked elsewhere.

answered Aug 18 '11 at 12:00

Michael Borgwardt

342,105
78
482
720

how to download large files without memory issues in java

4 Answers4

Linked