node.js \ why do I get RangeError: Maximum call stack size exceeded

Question

The purpose of the below program is to crawl CNN, and write all its text to a single file (using couple of third parties)

I get

RangeError: Maximum call stack size exceeded

How to troubleshoot this, and how can I bypass that? is there a way I can "free" memory ? and how?

//----------Configuration--------------

var startingUrl = "http://cnn.com"; //keep the http\https or www prefix
var crawlingDepth = "50";
var outputFileName = "cnn.txt";

//-------------------------------------

var Crawler = require("js-crawler");
var sanitizeHtml = require('sanitize-html');
var htmlToText = require('html-to-text');
var fs = require('fs');

var index = 0;

new Crawler().configure({depth: crawlingDepth})
  .crawl(startingUrl, function onSuccess(page) {

  var text = htmlToText.fromString(page.body, {
        wordwrap: false,
        hideLinkHrefIfSameAsText: true,
        ignoreHref: true,
        ignoreImage: true
    });

    index++;
    console.log(index + " pages were crawled"); 
    fs.appendFile(outputFileName, text, function (err) {
        if (err) {
            console.log(err);
        };
        console.log('It\'s saved! in same location.');
    }); 
  });

Did you try to lower `crawlingDepth`? I guess you simply crawl to much pages that are all kept in memory. — Reto Aebersold, Jul 26 '15 at 19:01
naturally it works with lower depth. So you imply that the js-crawler works recursively? Maybe I'll try other crawlers.. — user1025852, Jul 26 '15 at 19:08

stdob-- · Accepted Answer · 2015-07-26T21:03:45.807

1) This is a problem with the depth of recursion.

2) It is necessary to avoid it:

At each depth level is traversed by the links in the loop current levels (on the first level is one primary reference);
Access using 'Crawler.prototype._getAllUrls' links of the current page and if these links have not yet been processed - loops through them;

3) Only concept:

var Urls = [ ["http://cnn.com/"] ]; // What we crawling
var crawledUrls = {}; // Check if already crawled
var crawlingDepth = 3;
var depth = 0; // Current depth
var index = 0; // Current index
var Crawler = require("js-crawler");

function crawling() {
  console.log(depth, index, Urls[depth][index]);

  // Prepare next level
  if (typeof Urls[depth+1] === "undefined") Urls.push([]);

  // Already crawled flag
  crawledUrls[ Urls[depth][index] ] = true;

    new Crawler().configure({depth: 1}).crawl({
        url: Urls[depth][index],
        success: function(page) {
            // Do some with crawled page

            // Collect urls at crawled page
            var urls = Crawler.prototype._getAllUrls( page.url, page.body );
            for(var j=0; j<urls.length; j++) {
                // Check same domain and now crawled yet
                if ( typeof crawledUrls[urls[j]] === "undefined"
                     && urls[j].indexOf(Urls[0][0])===0 ) {
                    Urls[depth+1].push(urls[j]);
                }
            }
        },
        failure: function(page) {
        },
        finished: function(crawled) {
          index++;
          if (index<Urls[depth].length) {
            setTimeout(crawling,0);
          } else {
            depth++;
            index = 0;
            if (depth<crawlingDepth) {
              setTimeout(crawling,0);
            } else {
              // Finished
            }
        }
        }
    });
}

crawling();

thanks - not sure I'm following - are you suggesting some code I can write to overcome this? like to handle my own allUrls variable with some logic? — user1025852, Jul 26 '15 at 19:48
oh this is great :) - for some reason around 2K pages the code above is just hanging...trying to see what's holding it... — user1025852, Jul 27 '15 at 05:54
It is only the concept of getting rid of recursion :) In fact, this library objectively unsuitable for such a large number of pages. If you want a less serious decision, it is necessary to look at the decision based on the database. Perhaps something like elasticsearch. — stdob--, Jul 27 '15 at 06:00

node.js \ why do I get RangeError: Maximum call stack size exceeded

1 Answers1