How do I parse a HTML page with Node.js

Question

I need to parse (server side) big amounts of HTML pages.
We all agree that regexp is not the way to go here.
It seems to me that javascript is the native way of parsing a HTML page, but that assumption relies on the server side code having all the DOM ability javascript has inside a browser.

Does Node.js have that ability built in?
Is there a better approach to this problem, parsing HTML on the server side?

score 104 · Accepted Answer · edited Nov 23 '17 at 03:30

104

You can use the npm modules jsdom and htmlparser to create and parse a DOM in Node.JS.

Other options include:

BeautifulSoup for python
you can convert you html to xhtml and use XSLT
HTMLAgilityPack for .NET
CsQuery for .NET (my new favorite)
The spidermonkey and rhino JS engines have native E4X support. This may be useful, only if you convert your html to xhtml.

Out of all these options, I prefer using the Node.js option, because it uses the standard W3C DOM accessor methods and I can reuse code on both the client and server. I wish BeautifulSoup's methods were more similar to the W3C dom, and I think converting your HTML to XHTML to write XSLT is just plain sadistic.

edited Nov 23 '17 at 03:30

carla

1,970
1
31
44

answered Sep 10 '11 at 16:24

kzh

19,810
13
73
97

3

What do you mean by good? Reliable, fast, easy? Well with these two, it is robust enough so that you can [use jQuery serverside if you wanted to](http://stackoverflow.com/questions/1801160/can-i-use-jquery-with-node-js). – kzh Sep 10 '11 at 16:29
1

@kzh Reliable and easy are more important to me then if the process ends in one hour or one day. – Itay Moav -Malimovka Sep 10 '11 at 21:25
I would say that the node option is reliable and is definitely easy if you are already used to the DOM. – kzh Sep 11 '11 at 20:22
If you shoot for htmlparser, try going with https://github.com/fb55/node-htmlparser first. It seems to be a reworked version and is more actively maintained. – cburgmer Sep 16 '12 at 19:31
I searched all over the internet but cannot find a good tutorial for htmlparser.. – songyy May 17 '13 at 01:29
jsdom.env is not a function – Penguin9 Oct 20 '17 at 19:04
I personally feel jsdom is easier – Rishav Mar 06 '19 at 12:12
Other options, old link on answer, use this instead: `pip install beautifulsoup4` – Robert Rendell Jan 16 '23 at 14:40

score 67 · Answer 2 · answered Nov 12 '13 at 16:36

67

Use Cheerio. It isn't as strict as jsdom and is optimized for scraping. As a bonus, uses the jQuery selectors you already know.

❤ Familiar syntax: Cheerio implements a subset of core jQuery. Cheerio removes all the DOM inconsistencies and browser cruft from the jQuery library, revealing its truly gorgeous API.

ϟ Blazingly fast: Cheerio works with a very simple, consistent DOM model. As a result parsing, manipulating, and rendering are incredibly efficient. Preliminary end-to-end benchmarks suggest that cheerio is about 8x faster than JSDOM.

❁ Insanely flexible: Cheerio wraps around @FB55's forgiving htmlparser. Cheerio can parse nearly any HTML or XML document.

answered Nov 12 '13 at 16:36

Meekohi

10,390
6
49
58

10

But doesn't build DOM and doesn't allow XPath. jQuery syntax is surely a downside of that library. – polkovnikov.ph Sep 22 '14 at 06:16
2

@polkovnikov.ph in my experience very few applications require full DOM parsing, and building the DOM is very expensive compared to the fast "lazy" evaluation in jQuery/Cheerio. In this sense jQuery-style parsing is a benefit, but if your application requires manipulating the DOM server-side you might prefer to try jsdom. – Meekohi Sep 22 '14 at 12:01
`jsdom` is too slow for that :/ – polkovnikov.ph Sep 22 '14 at 13:28
2

@MohamedMansour for what it's worth we're using Cheerio in production and scraping thousands of pages in a few seconds. "fast" and "slow" are all relative to your application and bandwidth of course. – Meekohi Feb 04 '16 at 14:10
Non-strict: +1. jQuery syntax: +1. – Jonas Sourlier Dec 15 '19 at 21:52

score 18 · Answer 3 · answered Nov 17 '20 at 18:54

November 2020 Update

I searched for the top NodeJS html parser libraries.

Because my use cases didn't require a library with many features, I could focus on stability and performance.

By stability I mean that I want the library to be used long enough by the community in order to find bugs and that it will be still maintained and that open issues will be closed.

Its hard to understand the future of an open source library, but I did a small summary based on the top 10 libraries in openbase.

I divided into 2 groups according to the last commit (and on each group the order is according to Github starts):

Last commit is in the last 6 months:

jsdom - Last commit: 3 Months, Open issues: 331, Github stars: 14.9K.

htmlparser2 - Last commit: 8 days, Open issues: 2, Github stars: 2.7K.

parse5 - Last commit: 2 Months, Open issues: 21, Github stars: 2.5K.

swagger-parser - Last commit: 2 Months, Open issues: 48, Github stars: 663.

html-parse-stringify - Last commit: 4 Months, Open issues: 3, Github stars: 215.

node-html-parser - Last commit: 7 days, Open issues: 15, Github stars: 205.

Last commit is 6 months and above:

cheerio - Last commit: 1 year, Open issues: 174, Github stars: 22.9K.

koa-bodyparser - Last commit: 6 months, Open issues: 9, Github stars: 1.1K.

sax-js - Last commit: 3 Years, Open issues: 65, Github stars: 941.

draftjs-to-html - Last commit: 1 Year, Open issues: 27, Github stars: 233.

I picked Node-html-parser because it seems quiet fast and very active at this moment.

(*) Openbase adds much more information regarding each library like the number of contributors (with +3 commits), weekly downloads, Monthly commits, Version etc'.

(**) The table above is a snapshot according to the specific time and date - I would check the reference again and as a first step check the level of recent activity and then dive into the smaller details.

I love this answer because it highlights the unavoidable process node devs have to go through to vet the large number of available nearly-duplicate modules out there. [Here](https://gitlab.com/moodboom/reusable/-/blob/master/JavaScript/scrap/node/htmlParse/htmlParse.js)'s a 2023 survey I did. The only constant is change. — moodboom, Mar 02 '23 at 19:56

score 13 · Answer 4 · answered Nov 28 '14 at 12:04

13

Use htmlparser2, its way faster and pretty straightforward. Consult this usage example:

https://www.npmjs.org/package/htmlparser2#usage

And the live demo here:

http://demos.forbeslindesay.co.uk/htmlparser2/

answered Nov 28 '14 at 12:04

Anderson Madeira

420
9
29

How to get the exact kind of output, that one gets in this demo? – Penguin9 Oct 20 '17 at 19:04

score 6 · Answer 5 · answered Apr 20 '13 at 18:09

6

Htmlparser2 by FB55 seems to be a good alternative.

answered Apr 20 '13 at 18:09

esp

7,314
6
49
79

4

And what should one do with [this return format](http://demos.forbeslindesay.co.uk/htmlparser2/)? Write a bunch of for loops and tree traversals? – polkovnikov.ph Sep 22 '14 at 06:18
You can register to open/close tag events, so depending on what you want, this is a really good alternative imho. – Phil May 04 '15 at 19:20
@polkovnikov.ph There is also [domutils](https://github.com/FB55/domutils) package by the same author that works with the format returned by htmlparser2 - it has lots of methods, some of which have the same syntax as DOM methods, some are different; you won't really need to traverse the object manually. No docs there, but the source code is super clear - it all works as you would expect. – esp May 04 '15 at 19:50
not yet, but what stops you extending it? it's not that difficult using functions it already has. – esp May 09 '15 at 09:37

score 2 · Answer 6 · answered Aug 24 '13 at 11:40

2

jsdom is too strict to do any real screen scraping sort of things, but beautifulsoup doesn't choke on bad markup.

node-soupselect is a port of python's beautifulsoup into nodejs, and it works beautifully

answered Aug 24 '13 at 11:40

Yarek T

9,715
2
28
38

How do I parse a HTML page with Node.js

6 Answers6

November 2020 Update

Linked

Related