0

I have no formal code training and this is my first attempt at a Javapscript project and it's incredibly frustrating to be hung up on such a simple problem now that I'm 99.9% done.

I've stitched together the following scraper code from https://www.codementor.io/johnnyb/how-to-write-a-web-scraper-in-nodejs-du108266t and adapted it to a different target website:

// Define the scrape function
function scrape(url, data, cb) {
    // 1. Create the request
    req(url, (err, body) => {
        if (err) { return cb(err); }

        // 2. Parse the HTML
        let $ = cheerio.load(body)
          , pageData = {}
          ;

        // 3. Extract the data
        Object.keys(data).forEach(k => {
            pageData[k] = $(data[k]).text();
        });

        // Send the data in the callback
        cb(null, pageData);
    });
}

// Extract some data from a website
scrape("http://www.esbnyc.com/explore/tower-lights", {
    // Get the website title (from the top header)
    title: ".view-empty #page-title"
}, (err, data) => {
    console.log(err || data);
});

And I was delighted to see it worked! The console log displayed (almost, I'd like to get rid of the 'title:' that precedes the scraped web text) exactly what I wanted and I figured it would be a cinch to add that information to a string.

Yet 3 hours later of fumbling around and trying all the wrong answers I realize I'm no closer to a solution. All I want is to save the information I'm seeing in the log (e.g. "Tonight, the Empire State Building will be lit in its signature white.") as a string in the form of a Const/Var/whatever. Yet the best I've been able to get so far is literally nothing or an 'undefined'.

C S
  • 1
  • 1
  • `const someVarName = data;` ?? – CertainPerformance Jul 03 '18 at 04:33
  • Do you intend to pass it to another function after you have it stored in a variable? And a function that doesn't return anything returns `undefined` by default – LexJacobs Jul 03 '18 at 04:41
  • @CertainPerformance I've tried variations of that but I'm not even sure where to put it. someVarName either ends up null, undefined, or I get an error saying 'data is not defined'. – C S Jul 03 '18 at 04:42
  • `(err, data) => { if (err) return console.log(err); /* do stuff with data */ }` – CertainPerformance Jul 03 '18 at 04:44
  • @LexJacobs I've tried making the function 'return pageData' or pageData.toString(), but that just seemed to return some kind of null value. And yes, I need this variable to be usable outside of the function. I just pasted the code's "default" state since I figured tossing in any of my dozens of failed attempts to properly define a variable wouldn't be productive either. – C S Jul 03 '18 at 04:46
  • @CertainPerformance This is what I tried, the string output for Lights still came out as "Test": var Lights = "Test"; ... (err, data) => { if (err) return console.log(err); Lights = data }); – C S Jul 03 '18 at 04:52
  • Since data is an object, I think what you want is `const { title } = data;`, now you have a string variable `title` with the website title. – nijm Jul 03 '18 at 14:58
  • @nijm I've updated my console log to output var lights = String(data); const speechOutput = GET_FACT_MESSAGE + lights; console.log(speechOutput); console.log(data); It is telling that the first console message outputs [object Object] where the 'lights' should be, but no matter how I try to define lights, including with your approach, I always seem to get [object Object] for the log output. – C S Jul 03 '18 at 17:08
  • So I tried JSON.stringify(data) and that apparently solved ONE of the causes for my undefined issue - but it still appears that there's an async problem since it remains undefined in the string output and the console log outside of the function is displaying before the one inside the function - but when I try to move code inside the function then it just doesn't get run at all... arghhh – C S Jul 03 '18 at 17:30

0 Answers0