Downloading binary file from url

Question

I am using this code to download files from a url:

FileUtils.copyURLToFile(url, new File("C:/Songs/newsong.mp3"));

When I create the url using for instance, "https://mjcdn.cc/2/282676442/MjUgU2FhbCAtIFZlZXQgQmFsaml0Lm1wMw==", this works just fine and the mp3 is downloaded. However, if I use another url: "https://dl.jatt.link/hd.jatt.link/a0339e7c772ed44a770a3fe29e3921a8/uttzv/Hummer-(Mr-Jatt.com).mp3", the file is 0kb.

I am able to download files from both these urls from within a web browser. What's wrong here, and how can I fix it.

juzraai · Accepted Answer · 2017-09-11T10:51:12.593

I noticed a difference between your 2 URLs:

The first one just gives back the file without redirection.
But the second one responds with a redirect (HTTP/1.1 302 Moved Temporarily). It's also a special case, because it's a redirect from HTTPS to HTTP protocol.

Browsers can follow redirects, but your program - for some reason (see below) - can't.

I suggest you to use a HTTP client library (e.g. Apache HTTP client or Jsoup), and configure it to follow redirects (if they don't do it by default).

For example, with Jsoup, you would need a code like this:

String url = "https://dl.jatt.link/hd.jatt.link/a0339e7c772ed44a770a3fe29e3921a8/uttzv/Hummer-(Mr-Jatt.com).mp3";
String filename = "C:/Songs/newsong.mp3";
Response r = Jsoup.connect(url)
    //.followRedirects(true) // follow redirects (it's the default)
    .ignoreContentType(true) // accept not just HTML
    .maxBodySize(10*1000*1000) // accept 10M bytes (default is 1M), or set to 0 for unlimited
    .execute(); // send GET request
FileOutputStream out = new FileOutputStream(new File(filename));
out.write(r.bodyAsBytes());
out.close();

Update on @EJP's comment:

I looked up Apache Commons IO's FileUtils class on GitHub. It calls openStream() of the received URL object.
openStream() is a shorthand for openConnection().inputStream().
openConnection() returns an URLConnection object. If there is an appropriate subclass for the protocol used by URL, it will return an instance of that subclass. In this case that's a HttpsURLConnection which is the subclass of HttpURLConnection.
The followRedirects option is defined in HttpURLConnection and it's indeed true by default:

Sets whether HTTP redirects (requests with response code 3xx) should be automatically followed by this class. True by default.

So OP's approach would normally work with redirects too, but it seems that redirection from HTTPS to HTTP is not handled (properly) by HttpsURLConnection. - It's the case that @VGR mentioned in the comments below.
It's possible to handle redirects manually by reading the Location header with HttpsURLConnection, then use it in a new HttpURLConnection. (Example) (I wouldn't be surprised if Jsoup did the same.)
I suggested Jsoup because it already implements a way to handle HTTPS to HTTP redirections correctly and also provides tons of useful features.

Note that a redirect from an `https` URL to an `http` URL is considered insecure, so many libraries will not follow it even when requested to follow redirects. — VGR, Sep 10 '17 at 22:25
You are surely right and thank you. :) Though I've just tried out Jsoup with the URL above, and works perfectly. :) — juzraai, Sep 10 '17 at 22:38
Thanks... I just figured that it was because I wasn't following the redirect. Thanks for letting me know of a simple way to follow the redirect and consume the response as bytes using Jsoup. Coincidentally, I'm already using Jsoup for a part of my application! — superstar, Sep 10 '17 at 23:17
Wouldn't `FileUtils` already use this class? and doesn't it already follow redirects by default? If not, why not? and if not, why not just use `HttpURLConnection', which does follow redirects by default? — user207421, Sep 11 '17 at 00:22
@EJP Thanks for the questions/notes! I made some research upon those, please see my updated answer. I hope I could improve it. :) — juzraai, Sep 11 '17 at 09:54

Downloading binary file from url

1 Answers1