212

Is there something like Ruby's nokogiri on nodejs? I mean a user-friendly HTML-parser.

I'd seen on Node.js modules page some parsers, but I can't find something pretty and fresh.

Yves M.
  • 29,855
  • 23
  • 108
  • 144
asci
  • 2,540
  • 4
  • 16
  • 15
  • 3
    What do you mean by "friendly"? Convenient to work and select nodes with, like Nokogiri's XPath and CSS selector support? Amenable to parsing invalid "tag soup" HTML? – Phrogz Nov 02 '11 at 15:37
  • If you are comfortable with jQuery, consider [this answer](http://stackoverflow.com/a/31361044/778272). – Lucio Paiva Jul 11 '15 at 22:14

3 Answers3

463

If you want to build DOM you can use jsdom.

There's also cheerio, it has the jQuery interface and it's a lot faster than older versions of jsdom, although these days they are similar in performance.

You might wanna have a look at htmlparser2, which is a streaming parser, and according to its benchmark, it seems to be faster than others, and no DOM by default. It can also produce a DOM, as it is also bundled with a handler that creates a DOM. This is the parser that is used by cheerio.

parse5 also looks like a good solution. It's fairly active (11 days since the last commit as of this update), WHATWG-compliant, and is used in jsdom, Angular, and Polymer.

If the website you're trying to scrape is dynamic then you should be using a headless browser like phantomjs. Also have a look at casperjs, if you're considering phantomjs. And you can control casperjs from node with SpookyJS.

Beside phantomjs there's zombiejs. Unlike phantomjs that cannot be embedded in nodejs, zombiejs is just a node module.

There's a nettuts+ toturial for the latter solutions.

Farid Nouri Neshat
  • 29,438
  • 6
  • 74
  • 115
  • 1
    You can get DOM from htmlparser2 using DomHandler module (bundled with htmlparser2). They are separated on purpose to allow for other kinds of processing HTML without overhead of creating DOM. – esp Apr 20 '13 at 18:19
  • @esp Thanks, Before I thought it was non-standard DOM, I changed that section accordingly. – Farid Nouri Neshat Apr 21 '13 at 02:36
  • I'm not sure how you YQL for crawling - it's more for joining web service results not processing markup. – dardenfall Feb 25 '14 at 01:38
  • @dardenfall You are right, crawling is not the right term. I changed it with scraping :) – Farid Nouri Neshat Feb 25 '14 at 04:21
  • @Farid - (would've just messaged you if I could) at the risk of debating in comments (sorry!) I still don't see how you use it for scraping. It works with web _services_ not sites, and with wservices, you're rarely parsing html. Maybe xml, but not html. – dardenfall Feb 25 '14 at 18:38
  • Well it's like YQL(the webservice) does the hard work(fetching and parsing) for you. For example you want to find some stackoverflow questions: You can either use cheerio and request: `request('stackoverflow.com', function (err, body) { console.log(cheerio.load(body)('.question-hyperlink')) };` or use YQL: `yql('select * from data.html.cssselect where url="stackoverflow.com" and css=".question-hyperlink"', function (err, results) { console.log(results); })`. The latter will use way less of your system resources. – Farid Nouri Neshat Feb 26 '14 at 05:39
15

Try https://github.com/tmpvar/jsdom - you give it some HTML and it gives you a DOM.

thejh
  • 44,854
  • 16
  • 96
  • 107
6

You can also take a look at x-ray: https://github.com/lapwinglabs/x-ray

png
  • 5,990
  • 2
  • 25
  • 16