Faster way to download file in java

Question

I need a faster way to download a textfile from a url in Java. For a file of about 2400 lines, the code takes approximately 2 mins (132 seconds). The problem is that I need an update every minute, which is silly if the processing takes over 2 minutes.

This is what I have tried so far:

public void readFileOnline() throws Exception{
    Long now  = new Date().getTime();

    URL oracle = new URL("myurl");
    BufferedReader in = new BufferedReader(
    new InputStreamReader(oracle.openStream()));

    String currentline;
    while ((currentline = in.readLine()) != null){
//        Location location = gson.fromJson(currentline, Location.class);
//        locations.put(location.getTime(), location);
          locations.add(currentline);
    }

    in.close();

    Long done = new Date().getTime();
    System.out.println("TOTAL TIME " +  (done-now));

}

As you can see, there is some line processing needed. So I tried commenting the processing of the lines out and just save the lines in a collection, but seems to be no real speed optimisation there.

I also tried to just download the file and store it as a temporary file:

public String downloadAsTemp() throws Exception{
    Long now  = new Date().getTime();
    String url = "myurl";
    URLConnection request = null;
    request = new URL(url).openConnection();

    InputStream in = request.getInputStream();
    File downloadedFile = File.createTempFile("temp", "end-of-file");
    FileOutputStream out = new FileOutputStream(downloadedFile);
    byte[] buffer = new byte[1024];
    int len = in.read(buffer);
    while (len != -1) {
        out.write(buffer, 0, len);
        len = in.read(buffer);
        if (Thread.interrupted()) {
            throw new InterruptedException();
        }
    }
    in.close();
    out.close();


    Long done = new Date().getTime();
    System.out.println("TOTAL TIME " +  (done-now));

    return downloadedFile.getAbsolutePath();
}

It gives the same result (approximately 2 mins). I started using retrofit 2 to download the file. Unfortunately, during my search I lost my temper and deleted the code. Overall, you can say that even with Retrofit, it took too long. The file size is about 50MB, the lines of the file tend to get quite long.

I also stumbled upon this but the post dates from 2011, surely there are newer, faster ways? Also, the FileUtils link is dead :-).

So basically, I need to be able to download and process a 50MB file from a server under 1 min and the above is not working. Thanks!

Frankly I doubt it has anything to do with your file-reading code, and everything to do with either your connection to the server, or the server itself. — T.J. Crowder, Jun 17 '16 at 12:02
For me, mostly it depends on your internet speed rather than your code. — Prerak Sola, Jun 17 '16 at 12:03
Are you sure it's a code issue? It seems to me that your server might be taking too long to accept your connection. If that's the case, it won't matter how good your Java is. Try to find out which line of code is taking up all the time. — Dawood ibn Kareem, Jun 17 '16 at 12:04
Aha, ok. Thanks, it has crossed my mind that it might also be the connection speed... which is ofcourse not good news for me... thanks! — ocket-san, Jun 17 '16 at 12:04
Have you considered trying the same code with a local file? Or at least one in closer proximity (network-wise)? — Fildor, Jun 17 '16 at 12:25
How long does it take your browser to download the file? How long does `time wget ` take? — Clark Kent, Jun 17 '16 at 12:50
Google's Postman only needs 20, just ran a test :-/. Hmm, one would think that it should be able to beat 60 seconds in java... — ocket-san, Jun 17 '16 at 14:36
@Fildor My first attempt actually was processing local files. I put about 32 of those files in a directory, read them all and processed them all using streams. It was done under 60 seconds, if i recall that correctly. Unfortunately, I really need to automatically download those files :-/ — ocket-san, Jun 17 '16 at 15:15

score 1 · Answer 1 · answered Jun 17 '16 at 12:09

1

Use Apache Commons IO:

The IOUtils class is fast enough for your work, I suppose.

But of course - it depends on your connections speed...

answered Jun 17 '16 at 12:09

Christian

576
1
4
16

i will look into that, thanks! if it works faster, i ll accept the answer – ocket-san Jun 17 '16 at 14:16
Unfortunately, using FileUtils.copyUrlToFile takes exactly the same amount of time like the other methods :-/ – ocket-san Jun 17 '16 at 15:27

score 0 · Answer 2 · answered Aug 06 '21 at 19:58

I use the following to download files about 500Mb in size. You'll need springFramework though to use it. I found the download speed to be similar to downloading using a browser.

import org.springframework.util.StreamUtils;

final File downloadedFile = restTemplate.execute(FILE_URL, HttpMethod.GET, null, response -> {
            System.out.println("statusTest : "+ response.getStatusText());
            final Path downloadedFilePath = feedsParentDir.resolve("downloaded.type");
            Files.deleteIfExists(downloadedFilePath);
            StreamUtils.copy(response.getBody(), new FileOutputStream(downloadedFilePath.toFile()));
            return downloadedFilePath.toFile();
        });

Faster way to download file in java

2 Answers2