In Node.js / Express, how do I "download" a page and gets its HTML?

Question

Inside the code, I want to download "http://www.google.com" and store it in a string. I know how to do that in urllib in python. But how do you do it in Node.JS + Express?

Is there an easy way to do this? I'm hoping for a good set of "tools" that I can download as a module... — TIMEX, Apr 27 '11 at 08:50

score 33 · Answer 1 · answered Apr 27 '11 at 09:04

var util = require("util"),
    http = require("http");

var options = {
    host: "www.google.com",
    port: 80,
    path: "/"
};

var content = "";   

var req = http.request(options, function(res) {
    res.setEncoding("utf8");
    res.on("data", function (chunk) {
        content += chunk;
    });

    res.on("end", function () {
        util.log(content);
    });
});

req.end();

David · Accepted Answer · 2011-04-27T08:56:32.220

22

Using node.js you can just use the http.request method

http://nodejs.org/docs/v0.4.7/api/all.html#http.request

This method is built into node you just need to require http.

If you just want to do a GET, then you can use http.get

http://nodejs.org/docs/v0.4.7/api/all.html#http.get

var options = {
  host: 'www.google.com',
  port: 80,
  path: '/index.html'
};

http.get(options, function(res) {
  console.log("Got response: " + res.statusCode);
}).on('error', function(e) {
  console.log("Got error: " + e.message);
});

(Example from node.js docs)

You could also use mikeal's request module

https://github.com/mikeal/request

edited Apr 27 '11 at 08:56

answered Apr 27 '11 at 08:51

David

8,340
7
49
71

1

The request module works best. I tried both and the native http module hangs on multiple/concurrent requests. – reubano Sep 24 '13 at 09:23
1

That's true until you have to work with redirects, optional basic authorization etc. `npm install download` works best. – polkovnikov.ph Sep 08 '14 at 17:28
3

you need to declare `http = require("http")` – João Pimentel Ferreira Mar 05 '19 at 21:29
Doesn't actually download the page contents into a variable. – John Feb 23 '20 at 22:17
1

request is deprecated now https://github.com/request/request/issues/3142 – Code Maniac May 05 '20 at 12:46

score 20 · Answer 3 · edited Jun 20 '20 at 09:12

20

Simple short and efficient code :)

var request = require("request");

request(
    { uri: "http://www.sitepoint.com" },
    function(error, response, body) {
        console.log(body);
    }
);

doc link : https://github.com/request/request

edited Jun 20 '20 at 09:12

Community

1
1

answered Jun 23 '18 at 10:52

Natesh bhat

12,274
10
84
125

Definitely the easiest way here. – Rishav Mar 06 '19 at 12:05
Just don't forget to run `npm install request` first ;) – Jon Sep 11 '19 at 17:05
1

But deprecated now https://github.com/request/request/issues/3142 – Code Maniac May 05 '20 at 12:46
2

Here's a list of [alternative libraries to request](https://github.com/request/request/issues/3143). – Marco Lackovic Jul 21 '20 at 21:55

score 1 · Answer 4 · answered Jul 29 '20 at 01:41

Yo can try with axios

var axios = require('axios');

axios.get("http://www.sitepoint.com", {
  headers: {
    Referer: 'http://www.sitepoint.com',
    'X-Requested-With': 'XMLHttpRequest'
  }
}).then(function (response) {
    console.log(response.data);
  });

In Node.js / Express, how do I "download" a page and gets its HTML?

4 Answers4

Simple short and efficient code :)

Linked