How can I get a web page into a string using JavaScript?

Question

I need to get the html content of a page using JavaScript, the page could be also on another domain, kind of what does wget but in JavaScript. I want to use it for a kind of web-crawler.

Using JavaScript, how can I get content of a page, provided I have an URL, and get it into a string?

XMLHttpRequest? Unless you're expecting someone to give you lots of code to actually do it, you need more information in your question. What have you tried? What did you rule out? Why are you using javascript? there may be better ways. — Paystey, Oct 23 '12 at 11:53
Is this for client side JS (in the browser) or for server side JS (like node.js) ? — Sirko, Oct 23 '12 at 11:55
@Paystey I thought that too but then on other domains, see answer below about the concerns I had in mind — Eduard Florinescu, Oct 23 '12 at 12:12

score 1 · Answer 1 · edited May 23 '17 at 12:21

The general way to load content over HTTP via JavaScript is to use the XMLHttpRequest object. This is subject to the same origin policy so to access content on other domains you have to circumvent it.

This assumes you are running JS in a web browser (implied by "the page could be also on another domain"). If you were not that other options would be open to you. For example, with nodejs you could use the http client it has.

qwertymk · Accepted Answer · 2012-10-23T12:04:28.707

1

Try this:

function cbfunc(html) { alert(html.results[0]); }
$.getScript('http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22' + 
encodeURIComponent(url) + '%22&format=xml&diagnostics=true&callback=cbfunc');

DEMO

More about YQL

edited Oct 23 '12 at 12:04

answered Oct 23 '12 at 11:59

qwertymk

34,200
28
121
184

score 0 · Answer 3 · edited May 23 '17 at 12:27

0

If you want to also capture the hmtl tags you could concatenate them to the html like this:

 function getPageHTML() {
       return "<html>" + $("html").html() + "</html>";
    }

How do I get the entire page's HTML with jQuery?

edited May 23 '17 at 12:27

Community

1
1

answered Oct 23 '12 at 12:06

brightboy2004

258
1
3

How can I get a web page into a string using JavaScript?

3 Answers3

DEMO

Linked