Get URL Contents in Node.js with Express

Question

How would I go about downloading the contents of a URL in Node when using the Express framework? Basically, I need to complete the Facebook authentication flow, but I can't do this without GETing their OAuth Token URL.

Normally, in PHP, I'd use Curl, but what is the Node equivalent?

score 30 · Accepted Answer · answered Oct 14 '11 at 21:36

30

var options = {
  host: 'www.google.com',
  port: 80,
  path: '/index.html'
};

http.get(options, function(res) {
  console.log("Got response: " + res.statusCode);
}).on('error', function(e) {
  console.log("Got error: " + e.message);
});

http://nodejs.org/docs/v0.4.11/api/http.html#http.get

answered Oct 14 '11 at 21:36

chovy

72,281
52
227
295

2

Thanks for the pure-Node solution. While the whole event driven thing is cool, the simplicity of the request module makes the code a lot simpler for the project I'm working with. Because I didn't specify module or module-less, I'm going to mark this as the answer. – Andrew M Oct 15 '11 at 19:28
2

I found the accepted answer at http://stackoverflow.com/questions/6695143/how-to-make-web-service-calls-in-expressjs to be a better example of the same solution. – ashack Feb 13 '13 at 00:25
4

Um?? The question asked for the HTML, not the status code. – Jack M Sep 20 '18 at 20:16
@JackM https://stackoverflow.com/a/68230345/16359609 – RixTheTyrunt Jun 10 '22 at 07:16

score 16 · Answer 2 · edited Mar 11 '17 at 15:42

16

The problem that you will front is: some webpage loads its contents using JavaScript. Thus, you needs a package, like After-Load which simulates browser's behavior, then gives you the HTML content of that URL .

var afterLoad = require('after-load');
afterLoad('https://google.com', function(html){
   console.log(html);
});

edited Mar 11 '17 at 15:42

Sergio

28,539
11
85
132

answered Jul 27 '16 at 05:12

Abdennour TOUMI

87,526
38
249
254

2

Unfortunately, the link is broken. Here the npm page: https://www.npmjs.com/package/after-load – somejhereandthere Jun 17 '19 at 13:56

Natesh bhat · Answer 3 · 2018-06-23T11:06:34.653

8

Using http way requires way more lines of code for just a simple html page .

Here's an efficient way : Use request

var request = require("request");

request({uri: "http://www.sitepoint.com"}, 
    function(error, response, body) {
    console.log(body);
  });
});

Here is the doc for request : https://github.com/request/request

2nd Method using fetch with promises :

    fetch('https://sitepoint.com')
    .then(resp=> resp.text()).then(body => console.log(body)) ;

edited Jun 23 '18 at 11:06

answered Jun 23 '18 at 10:50

Natesh bhat

12,274
10
84
125

As of Feb 11th 2020, request is fully deprecated – PaulH Nov 28 '22 at 09:06

score 0 · Answer 4 · answered Jul 02 '21 at 19:20

Using http module:

const http = require('http');

http.get('http://localhost/', (res) => {
    let rawHtml = '';
    res.on('data', (chunk) => { rawHtml += chunk; });
    res.on('end', () => {
        try {
            console.log(rawHtml);
        } catch (e) {
            console.error(e.message);
        }
    });
});

rawHtml - complete html of the page.

I just simplified example from official docs.

score -2 · Answer 5 · answered Mar 28 '22 at 08:51

-2

using Axios is much simpler

const axios = require("axios").default

const response = axios.get("https://google.com")

console.log(response.data)

or

const axios = require("axios").default

const response = axios.get("https://google.com").then((response)=>{
console.log(response.data)
})

for full docs, you can head over Axios Github

answered Mar 28 '22 at 08:51

Yazan Zoghbi

127
8

Retuns undefined – RixTheTyrunt Jun 10 '22 at 07:22

Get URL Contents in Node.js with Express

5 Answers5

2nd Method using fetch with promises :

Linked