How to extract source html from webpage?

Question

I am trying to extract the html source of this page, http://www.fxstreet.com/rates-charts/currency-rates/

I want what I see when I save the page from chrome as a .html file.

I tried to do this in java, using bufferedreader, and then using jsoup. I also tried to do it in python, however I keep getting the following message:

"This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your browser."

The end goal is to extract the values in the main table.

Xeon was right. It works fine with HtmlUnit, with JS enabled. — Spart, Jun 08 '12 at 18:15

score 4 · Accepted Answer · edited May 23 '17 at 12:09

4

Try using HtmlUnit and setting setJavascriptEnabled(true)

Look also at: this and this

JSoup isn't headless browser to execute Javascript so you must choose other library to get the page and then you can use JSoup to parse it.

edited May 23 '17 at 12:09

Community

1
1

answered Jun 01 '12 at 21:51

Xeon

5,949
5
31
52

score 1 · Answer 2 · answered Dec 09 '12 at 15:56

1

Just to extract the main table can be easily done using Jsoup

here's a method that will take all the content from the main table on the page

public void parse(){
        try{

        Document doc = Jsoup.connect("http://www.fxstreet.com/rates-charts/currency-rates/").get();
        Element content = doc.getElementById("ddlPairsChoose");
        Elements table = doc.getElementsByClass("applet-content");      

        System.out.print(table);

        }

        catch(Exception e){

            System.out.print("error --> " + e);
        }       
    }

It prints out the table on the page

answered Dec 09 '12 at 15:56

JohnnyQ

484
10
23

Like this too, as Jsoup can simulate a browser quite well if you use the connect mechanism. If this wont work either, run a local http proxy and browse the site manually recording the communication. Perhaps the Browser sends additional AJAX background requests to filter out robots or the site requires different user agents (then check this: http://stackoverflow.com/questions/6581655/jsoup-useragent-how-to-set-it-right) – Stefan Dec 09 '12 at 16:01

How to extract source html from webpage?

2 Answers2

Linked