An array from a NodeList while scraping with Puppeteer doesn't appear

Question

I'm trying to scape a website but I cannot find a way to show the results in the console.log. The script I've created is the following:

const puppeteer = require("puppeteer");
(async () => {
    try { 
        const browser = await puppeteer.launch({ headless: true });
        const page = await browser.newPage();
        await page.goto(`https://www.coches.net/nuevo/km-0/`);
        await page.waitFor(4000);

        const news = await page.evaluate(() => {
            const urlsArray = Array.from(document.querySelectorAll('.mt-CardAd-link')).map(a => a.href);
            return urlsArray;
        });
        console.log(news);
        await browser.close();
        console.log("Browser Closed");
    } catch (err) {
        console.log(err);
        await browser.close();
        console.log("Browser Closed");
    }
})();

While the variable urlsArray works in the devconsole of Chrome, it doesn't when launching the script in the terminal with the previous script. I tried everything but I don't find anything to solve this issue. What I can do to finally be able to show this array with the console.log?

Thank you!

For me this URL shows 403 now, so it doesn't have an anchor – Fox Amadeus Mar 30 '20 at 16:08 — Fox Amadeus, Mar 30 '20 at 16:08

Eduardo Conte · Accepted Answer · 2020-03-30T18:45:26.883

0

Your script seems to be fine, but the page you´re trying to enter is examinating the cookies and blocking your script's access.

I believe you should take a look at this: How to avoid being detected as bot on Puppeteer and Phantomjs?

edited Mar 30 '20 at 18:45

answered Mar 30 '20 at 18:39

Eduardo Conte

1,145
11
18

An array from a NodeList while scraping with Puppeteer doesn't appear

1 Answers1