How to download large sized Files (size > 50MB) in java

Question

I'm downloading files from a remote location, and the download is complete for smaller sized files and in-complete for large sized files (>10 MB). Here is my code that i have used for downloading files from remote server .

    File dstFile = null;
    // check the directory for existence.
    String dstFolder = LOCAL_FILE.substring(0,LOCAL_FILE.lastIndexOf(File.separator));
    if(!(dstFolder.endsWith(File.separator) || dstFolder.endsWith("/")))
        dstFolder += File.separator;

    // Creates the destination folder if doesn't not exists
    dstFile = new File(dstFolder);
    if (!dstFile.exists()) {
        dstFile.mkdirs();
    }
    try {
        URL url = new URL(URL_LOCATION);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection(); 
        connection.addRequestProperty("User-Agent", "Mozilla/4.76"); 
        //URLConnection connection = url.openConnection();
        BufferedInputStream stream = new BufferedInputStream(connection.getInputStream());
        int available = stream.available();
        byte b[]= new byte[available];
        stream.read(b);
        File file = new File(LOCAL_FILE);
        OutputStream out  = new FileOutputStream(file);
        out.write(b);
    } catch (Exception e) {
        System.err.println(e);
        VeBLogger.getInstance().log( e.getMessage());
    }

What is your actual question or what is not working about this code? — tier1, Jul 30 '14 at 19:14
Call `flush()` on your `FileOutputStream` and then `close()` it. — GriffeyDog, Jul 30 '14 at 19:16

bitkot · Accepted Answer · 2014-07-30T19:28:53.653

4

You can use apache commons IO library. It's easy. I have used it in many projects.

File dstFile = null;
// check the directory for existence.
String dstFolder = LOCAL_FILE.substring(0,LOCAL_FILE.lastIndexOf(File.separator));
if(!(dstFolder.endsWith(File.separator) || dstFolder.endsWith("/")))
    dstFolder += File.separator;

// Creates the destination folder if doesn't not exists
dstFile = new File(dstFolder);
if (!dstFile.exists()) {
    dstFile.mkdirs();
}
try {
    URL url = new URL(URL_LOCATION);
    FileUtils.copyURLToFile(url, dstFile);
} catch (Exception e) {
    System.err.println(e);
    VeBLogger.getInstance().log( e.getMessage());
}

edited Jul 30 '14 at 19:28

answered Jul 30 '14 at 19:23

bitkot

4,466
2
28
39

Amit : Thanks for your reply.. Any restriction in terms of file size is there while using FileUtils.copyURLToFile(url, dstFile), Can we download all files (regardless of its size) ? – Raguram Jul 31 '14 at 09:36
do we need to take care of closing the stream when invoking the FileUtils.copyURLToFile(url, dstFile) method ? – Raguram Jul 31 '14 at 09:38
No need, see the [source code](http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/1.2/org/apache/commons/io/FileUtils.java#FileUtils.copyURLToFile%28java.net.URL%2Cjava.io.File%29), And there is no limit in size – bitkot Jul 31 '14 at 10:23
HiberKnight: Hi, whether this copyURLToFile() copies the list of files present in a directory from server ? i mean that if a directory in server has a list of files and if i provide the the directory location as the URL argument. then whether it holds good ? – Raguram Aug 13 '14 at 05:04
I think it is not possible to download all the files from server. Is there any API which does this ? – Raguram Aug 13 '14 at 05:16
No, It is not. For that you need to download the directory page prase all the anchor tags, get the url for individual files and then download them in loop. – bitkot Aug 13 '14 at 06:00
Thanks, but that seems to be a costlier operation (getting the HTML file and parsing all the tags). Any other way (or open source API) exists for accomplishing this ? – Raguram Aug 13 '14 at 06:17
because everywhere in blogs, they are talking about file download only and not directory download. – Raguram Aug 13 '14 at 06:18
HiberKnight: How to get the URL for individual files in a directory ? – Raguram Aug 13 '14 at 09:28
You don't need to create the folders leading up to the target folder. From the docs: `Copies bytes from the URL source to a file destination. The directories up to destination will be created if they don't already exist. destination will be overwritten if it already exists. Warning: this method does not set a connection or read timeout and thus might block forever. Use copyURLToFile(URL, File, int, int) with reasonable timeouts to prevent this.` – movAX13h Feb 16 '23 at 12:17

score 3 · Answer 2 · answered Jul 30 '14 at 19:16

3

Firstly, I'd suggest you use:

FileInputStream in = new FileInputStream(file);

instead of:

BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));

(To avoid building up memory usage)

try
{
    FileInputStream fileInputStream  = new FileInputStream(file);
    byte[] buf=new byte[8192];
    int bytesread = 0, bytesBuffered = 0;
    while( (bytesread = fileInputStream.read( buf )) > -1 ) {
        out.write( buf, 0, bytesread );
        bytesBuffered += bytesread;
        if (bytesBuffered > 1024 * 1024) { //flush after 1MB
            bytesBuffered = 0;
            out.flush();
        }
    }
}
finally {
    if (out != null) {
        out.flush();
    }
}

answered Jul 30 '14 at 19:16

Rakesh Sharma

301
1
3
11

It seems strange to avoid BufferedInputStream when you are implementing exactly a buffer here. Can you elaborate why BufferedInputStream is not a good fit? – ormurin Jan 12 '21 at 13:28
The question is asking how to download a file from a URL. FileInputStream is not how you read/download a file from a URL. FileInputStream is used to read data from the file system. – Gamebuster19901 Dec 28 '21 at 19:00

score 0 · Answer 3 · answered Jul 30 '14 at 19:21

Please read BufferedInputStream's method .available() in the API.

It returns the number of available bytes already downloaded (ie. the number of bytes you can read out of the stream without accessing/waiting for the network).

You should create a fixed size byte array, fx. 2048 bytes, and use the read() methods until it returns -1.

score 0 · Answer 4 · answered Feb 22 '23 at 01:54

public String DownloadFile(String path, String outputFileName) {    
        try {
            Connection.Response response = Jsoup.connect(path).cookie()
                .maxBodySize(0)
                .ignoreContentType(true)
                .execute();
            FileOutputStream out = new FileOutputStream(new File(outputFileName));
            out.write(response.bodyAsBytes());
            out.close();
            return outputFileName;
        } catch (Exception e) {
            e.printStackTrace();
            return "";
        }
    }

i use Jsoup library. you can consult – Nguyễn Tuấn Hưng Feb 22 '23 at 01:56 — Nguyễn Tuấn Hưng, Feb 22 '23 at 01:56

How to download large sized Files (size > 50MB) in java

4 Answers4

Linked