Java HttpURLConnection to get all content include css, img and so on

Question

My requirement is to do web application to get destination content and have to add request header by using HttpURLConnection.

I use this code

    try{
        String urlStr = "http://test/STAM/Login";

        url = new URL(urlStr);
        urlconn = (HttpURLConnection)url.openConnection();
        urlconn.setRequestProperty("Accept-Language","en-us,en;q=0.5");
        urlconn.setRequestProperty("Accept-Charset","ISO-8859-1,utf-8;q=0.7,*;q=0.7");
        urlconn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16");

        urlconn.connect();
        //read the result from the server
        rd  = new BufferedReader(new InputStreamReader(urlconn.getInputStream()));
        sb = new StringBuilder();  
        while ((line = rd.readLine()) != null)
        {
            sb.append(line + '\n');
        }

        out.println(sb.toString());

   } catch(Exception e) {
        e.printStackTrace();
   } finally {
   //close the connection, set all objects to null
        urlconn.disconnect();
        rd = null;
        sb = null;
        wr = null;
        urlconn = null;
   }

Result is: My code will get only source content but cannot get css, image file, javascript and so on because it will load from local host such as "http://localhost:8080/test/img/test.gif" instead of "http://test/STAM/img/test.gif". Please advise me if I'm wrong or you have any way to solve this problem.

Thanks a lot.

I would like to see full content (text,image) from web site of destination by using HttpURLConnection. Thanks — user1770473, Oct 24 '12 at 08:05
What do you mean by full content? Like you see the page in your browser? — Thomas Jungblut, Oct 24 '12 at 08:21
What you want is a browser with several HttpURLConnection. A connection is only used to retrieve a response to a request. It will not parse the HTML content to issue the images/scripts/CSS requests... — mkhelif, Oct 24 '12 at 08:28
@ThomasJungblut Yes, my requirement is user want to see the detination page in my page. — user1770473, Oct 24 '12 at 09:10

score 2 · Answer 1 · edited May 23 '17 at 11:48

From what I understand you expect to fetch all the ressources used in the page (css/js/images/...) when you fetch the html page.

As you might know, HTML is a markup language that reference external resources. It is the web browser job to fetch all those resources and render the page. If you want to have all the resources, then you should parse the source content to extract the links to resources and fetch them individually.

If you want to reuse the resources from their original location you must pay attention that the resources URLs are still reachable from where the page will be opened. (URL could be absolute or relative)

score 1 · Answer 2 · answered Oct 24 '12 at 09:01

1

Don't use HttpUrlConnection, to do this. You will end up with parsing html-files to solve a already solved problem.

Use: HttpClient

answered Oct 24 '12 at 09:01

Christian Kuetbach

15,850
5
43
79

Fridjato Part Fridjat · Answer 3 · 2019-12-11T12:03:22.540

I am already used URLConnection in the web to get the source of Html css and javascript code from any website, I try here for this example to get code of microsoft website using URL and URLConnection and InputStream.

import java.io.BufferedInputStream;
import java.io.IOExecption;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.MalformedURLExeption;
import java.net.URL;
import java.net.URLConnection;

public class Main2{
    public static void main(String[] args) throws IOException{
        URL u = new URL("http://www.microsoft.com/");
        URLConnection uc = u.openConnection();
        InputStream is = uc.getInputStresm();
        InputStream buf = new BuffredInputStream(is);
        Reader r = new InputStreamReader(buf);
        int i;
        while((i = r.read()) != -1){
            System.out.println((char)i);
        }
    }
}

Or try this folowing code.

    URLConnection uc = new URL("http://www.google.com").openConnection();
    BufferedInputStream bis = new BufferedInputStream(uc.getInputStream());
    int i;
    while(( i == bis.read()) != -1){
        System.out.println((char)i);
    }
    bis.close();
    }
}

Java HttpURLConnection to get all content include css, img and so on

3 Answers3