Utilizing full network capacities to download files

Question

I am fetching data, given a URL, and have tried a couple methods described here: How to download and save a file from Internet using Java?

Our download speed is at least 2 MB/s, so it is somewhat discomforting to see the file downloading at speeds that look like 56 KB/s. Using a browser, we can grab a 50 MB file in seconds, but using the methods described above it takes several minutes.

How can I take advantage of the fact that we have a fast connection? The application I am developing fetches data remotely for daily updates and the data is typically in the 10-100 MB range, so it would be nice if we can perform the update routines quickly.

we'd need to see some code, how are you downloading the file? — Collin, May 22 '13 at 15:09
Collin, I copied the code from the referenced link. Sotirios Delimanolis, one of the solutions presented by Ben Noland uses a buffer size of 1024. I tried changing that to 1024000 but it didn't seem to make a difference. — MxLDevs, May 22 '13 at 15:11
Each of those solutions give you the same (~) download speed? — Sotirios Delimanolis, May 22 '13 at 15:14
is this helpful http://stackoverflow.com/questions/3409608/java-multithreaded-file-downloading-performance — Peeyush, May 22 '13 at 15:17

score 5 · Accepted Answer · answered May 22 '13 at 15:18

Using Java NIO (as in the following example):

URL website = new URL("http://cachefly.cachefly.net/100mb.test");
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream("test.test");
long x = System.currentTimeMillis();
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
System.out.println(System.currentTimeMillis()-x);

...results in a download time for me of 25 seconds - Chrome took (near enough) exactly the same time. If you're running on any "normal" OS (Windows, Mac, Linux) then the above will make use of the filesystem cache which means it should be just as fast as a "native" application performing the same job (as it is, for me at least.)

Note that using a simple loop to copy the bytes over (the pre-nio method) will be much slower, because it doesn't make use of the cache - so definitely use the above method to get the fastest speed you can.

If you're seeing drastically different times using the above method then I'd suggest the problem lies elsewhere - something external limiting bandwidth to the Java process for instance (perhaps there's some bizarre external security policy to blame?) I certainly don't see any dramatic speed decrease as you're seeing here with the code above.

That's interesting. Let me test with java NIO again and get back to you. — MxLDevs, May 22 '13 at 15:21
but wait... the code posted is exactly as the one in the link... so @Keikoku are you sure you measured right? — UmNyobe, May 22 '13 at 15:27
I see the problem now. When I run the download code on a test server it is fast, but when I hand it off for actual live downloading for whatever reason the transfer speed is really slow. I didn't consider the possibility that maybe something else was just slow. — MxLDevs, May 22 '13 at 15:37
Is there a way to show how much date has been downloaded? and how much the whole data is? — hamid, May 22 '13 at 17:26
@Sam Not as you go, but once your done you can just check the size of the corresponding file. — Michael Berry, May 22 '13 at 17:54

Utilizing full network capacities to download files

1 Answers1