34

Without the use of any external library, what is the simplest way to fetch a website's HTML content into a String?

Ian Nelson
  • 57,123
  • 20
  • 76
  • 103
pek
  • 17,847
  • 28
  • 86
  • 99
  • 3
    possible duplicate of http://stackoverflow.com/questions/238547/how-do-you-programmatically-download-a-webpage-in-java – jjnguy Apr 06 '10 at 05:29

6 Answers6

46

I'm currently using this:

String content = null;
URLConnection connection = null;
try {
  connection =  new URL("http://www.google.com").openConnection();
  Scanner scanner = new Scanner(connection.getInputStream());
  scanner.useDelimiter("\\Z");
  content = scanner.next();
  scanner.close();
}catch ( Exception ex ) {
    ex.printStackTrace();
}
System.out.println(content);

But not sure if there's a better way.

Community
  • 1
  • 1
pek
  • 17,847
  • 28
  • 86
  • 99
  • 5
    Why "\\Z"? Isn't it an EOF on Windows only? I am just guessing here. – greenoldman Nov 09 '11 at 20:52
  • 1
    Why do you use "\\Z"? What does it do? I tried without it, it didn't work. – Max Husiv Feb 03 '17 at 14:03
  • @MaxHusiv I think it's because if you don't specify a delimiter, scanner.next() will just go through the whole HTML character by character, but if you use a delimiter which won't be found in the HTML, scanner.next() returns the whole thing. – Chris A Nov 15 '20 at 15:27
  • What import statements do you need for that to work? – theerrormagnet Sep 09 '22 at 13:07
22

This has worked well for me:

URL url = new URL(theURL);
InputStream is = url.openStream();
int ptr = 0;
StringBuffer buffer = new StringBuffer();
while ((ptr = is.read()) != -1) {
    buffer.append((char)ptr);
}

Not sure at to whether the other solution(s) provided are any more efficient or not.

Scott Bennett-McLeish
  • 9,187
  • 11
  • 41
  • 47
2

Whilst not vanilla-Java, I'll offer up a simpler solution. Use Groovy ;-)

String siteContent = new URL("http://www.google.com").text
Scott Bennett-McLeish
  • 9,187
  • 11
  • 41
  • 47
2

I just left this post in your other thread, though what you have above might work as well. I don't think either would be any easier than the other. The Apache packages can be accessed by just using import org.apache.commons.HttpClient at the top of your code.

Edit: Forgot the link ;)

Community
  • 1
  • 1
Justin Bennett
  • 8,906
  • 2
  • 27
  • 30
0
 try {
        URL u = new URL("https"+':'+'/'+'/'+"www.Samsung.com"+'/'+"in"+'/');
        URLConnection urlconnect = u.openConnection();
        InputStream stream = urlconnect.getInputStream();
        int i;
        while ((i = stream.read()) != -1) {
            System.out.print((char)i);
        }
    }
    catch (Exception e) {
        System.out.println(e);
    }
Dheeraj Mukharjee
  • 1
  • 1
-4

Its not library but a tool named curl generally installed in most of the servers or you can easily install in ubuntu by 

sudo apt install curl

Then fetch any html page and store it to your local file like an example 

curl https://www.facebook.com/ > fb.html

You will get the home page html.You can run it in your browser as well.

dinesh kandpal
  • 738
  • 7
  • 16