Java: Download from an URL

Question

Could someone try my codes out? It was working a few days ago and now it's not. I did not modify anything, and so I suspect the webmaster of that side has block me. Could someone check it out for me? This is part of my school project.

public class Cost extends TimerTask{

public void run() {
  Calendar rightNow = Calendar.getInstance();
  Integer hour = rightNow.get(Calendar.HOUR_OF_DAY);

  if (hour==1) {
    try {
      URL tariff = new URL("http://www.emcsg.com/MarketData/PriceInformation?downloadRealtime=true");
      ReadableByteChannel tar = Channels.newChannel(tariff.openStream());
      FileOutputStream fos = new FileOutputStream("test.csv");
      fos.getChannel().transferFrom(tar, 0, 1<<24);

    } catch (IOException ex) {
      Logger.getLogger(Cost.class.getName()).log(Level.SEVERE, null, ex);
    } 
  }

  else {

  }
}
}

Answer 1

First of all, clean up your IO exceptions as that might be obscuring the problem - check you can write to D:.

If you are being blocked by the site because of your user-agent header:

This will show you your user-agent header: http://pgl.yoyo.org/http/browser-headers.php. Then the answer to Setting user agent of a java URLConnection tells you how to set your header.

You will either need to add a step between instantiating URL and opening stream:

URL tariff = new URL("http://www.emcsg.com/MarketData/PriceInformation?downloadRealtime=true");
java.net.URLConnection c = tariff.openConnection();
c.setRequestProperty("User-Agent", " USER AGENT STRING HERE ");
ReadableByteChannel tar = Channels.newChannel(c.getInputStream());

or you could try just doing this:

System.setProperty("http.agent", " USER AGENT STRING HERE ");

sometime before you call openStream().

Edit: This works for me. Can you try running it and let us know the output:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;

    public class TestURL {

        public static void main(String[] args) {
            try {
              URL tariff = new URL("http://www.emcsg.com/MarketData/PriceInformation?downloadRealtime=true");
              URLConnection c = tariff.openConnection();
              BufferedReader br = new BufferedReader(new InputStreamReader(c.getInputStream()));
              System.out.println(br.readLine());
            } catch (IOException ex) {
              ex.printStackTrace();
            }
        }
    }

Answer 2

I checked your code and running it I had no problem, everything works fine. Are you working behind a proxy?

In that case you have to configure it:

System.setProperty("http.proxyHost", "my.proxy.name");
System.setProperty("http.proxyPort", "8080");