How to solve and locate an element using Puppeteer

Question

Screen Scraping Problems

I am trying to learn how to locate an element using querySelector.

In this small NodeJS screen scrape app using Puppeteer, I want to add the "location" of an item listed on Marketplace.

Before my attempts to add this new node, the script works great, it will output the following:

[
  {
    itemTitle: 'Iphone 7 Unlocked 32GB Gold',
    itemPrice: '$195',
    itemURL: 'https://facebook.com//marketplace/item/1094335687596621/',
    itemImg: 'https://scontent-mia3-1.xx.fbcdn.net/v/t1.0-0/c0.29.261.261a/p261x260/94707992_155656332613238_7356296336775315456_o.jpg?_nc_cat=108&_nc_sid=843cd7&_nc_oc=AQk4lXrzsGBkPFIWqx-sRgcDWuNd2kZlc1XJavKl0ZLJBcZZfaLFFUpZbSTTStFuT9U&_nc_ht=scontent-mia3-1.xx&oh=bc6f948b7c5930f1f81f85cbdf889ad5&oe=5ECEB781'
  },
  {
    itemTitle: 'iPhone X 64gb Unlocked Everything Works',
    itemPrice: '$180',
    itemURL: 'https://facebook.com//marketplace/item/2576925735968131/',
    itemImg: 'https://scontent-mia3-2.xx.fbcdn.net/v/t1.0-0/c0.83.750.750a/s261x260/84641729_222957132082738_3587337422137982976_o.jpg?_nc_cat=105&_nc_sid=843cd7&_nc_oc=AQmueUdOvX-NNTmxZOZp5gvyt-Szz9emk0it5HseboNA38BvQvKV0FT7mDwtqJo6z1g&_nc_ht=scontent-mia3-2.xx&oh=fbf3ffe04007566227eb76c608dd7fae&oe=5ECF5178'
  }
]

The Selectors For Above Results

Here is what is used to grab those results:

const itemTitle = item.querySelector('div > div > span > div > a > div > div > div > span > div > span > div[class="l9j0dhe7 stjgntxs ni8dbmo4"').innerText;
const itemPrice = item.querySelector('div > div > span > div > a > div > div > div > div > span[class="oi732d6d ik7dh3pa d2edcug0 qv66sw1b c1et5uql a8c37x1j s89635nw ew0dbk1b a5q79mjw g1cxx5fr lrazzd5p oo9gr5id"').innerText;
const itemURL = `https://facebook.com/${item.getAttribute('href')}`;
const itemImg = item.querySelector('div > div > span > div > a > div > div > div > div > div > div > img').getAttribute('src');

So now I want to add the location of the item:

and here is the SPAN I am trying to grab:

so I tried Inspecting the element and COPY->XPATH in Chrome Dev Tools, and was given this:

//*[@id="u_0_c"]/div/div[1]/div/div/div/div/div[2]/div/div[2]/div/div/div[1]/div/span/div/a/div/div[2]/div[3]/span/div/span

When I plugged that into the existing code...

var location = item.querySelectorAll('div[@id="u_0_c"]/div/div[1]/div/div/div/div/div[2]/div/div[2]/div/div/div[1]/div/span/div/a/div/div[2]/div[3]/span/div/span').innerText;

it breaks. All sorts of errors.

How do I DO THIS?

I didn't write this code. The selectors appear very complicated. Am I on the right track using CHrome's Dev Tools? What am I doing wrong??

Entire Source

const puppeteer = require('puppeteer');

const getItems = async searchTerm => {
    //{headless: false, defaultViewport: null} --> put this in launch() method below as parameter for developtment purposes --> opens up browser window
    const browser = await puppeteer.launch({
        headless: false,
        defaultViewport: null
    });
    const page = await browser.newPage();
 /* Go to page */
 await page.goto(`https://facebook.com/marketplace/search/?query=${encodeURI(searchTerm)}`);


    const itemList = await page.waitForSelector('div > div > span > div > a[tabindex="0"]')
        .then(() => page.evaluate(() => {
            const itemArray = [];
            const itemNodeList = document.querySelectorAll('div > div > span > div > a[tabindex="0"]');

            itemNodeList.forEach(item => {
                const itemTitle = item.querySelector('div > div > span > div > a > div > div > div > span > div > span > div[class="l9j0dhe7 stjgntxs ni8dbmo4"').innerText;
                const itemPrice = item.querySelector('div > div > span > div > a > div > div > div > div > span[class="oi732d6d ik7dh3pa d2edcug0 qv66sw1b c1et5uql a8c37x1j s89635nw ew0dbk1b a5q79mjw g1cxx5fr lrazzd5p oo9gr5id"').innerText;
                const itemURL = `https://facebook.com/${item.getAttribute('href')}`;
                const itemImg = item.querySelector('div > div > span > div > a > div > div > div > div > div > div > img').getAttribute('src');
                //var location = item.querySelectorAll('div[@id="u_0_c"]/div/div[1]/div/div/div/div/div[2]/div/div[2]/div/div/div[1]/div/span/div/a/div/div[2]/div[3]/span/div/span').innerText;
                itemArray.push({
                    itemTitle,
                    itemPrice,
                    itemURL,
                    itemImg
                });
            });
            return itemArray;
        }))
        .catch(() => console.log("Selector error."));

    return itemList;

}

const initScraper = async () => {
    const items = await getItems('iphone');
    items.sort(function (a, b) {
        return a.itemPrice - b.itemPrice
    });
    console.log(items);
}

initScraper();

Thanks for looking. Any help would be appreciated.

score 1 · Accepted Answer · answered Apr 29 '20 at 08:45

1

I think this is what you are looking for:

Basically, if you get the XPATH from the chrome DevTools, you can't just use querySelector, you need to use the document.evaluate function.

Best!

answered Apr 29 '20 at 08:45

charly rl

845
2
7
15

I read through that link you sent. Lots of good info there. What ended up working for me was a tiny unknown dev-tool function I have never even heard of! $x(' my x-path here ') - and that gives me the info I needed. Well almost. It gave me this: span.oi732d6d ik7dh3pa d2edcug0 qv66sw1b c1et5uql a8c37x1j s89635nw ew0dbk1b a5q79mjw g1cxx5fr lrazzd5p oo9gr5id - and I had to modify it a bit, like this: querySelector('span[class="oi732d6d ik7dh3pa d2edcug0 qv66sw1b c1et5uql a8c37x1j s89635nw ew0dbk1b a5q79mjw g1cxx5fr lrazzd5p oo9gr5id"').innerText(); That did the trick! Thank you. – John S. Apr 29 '20 at 23:30

How to solve and locate an element using Puppeteer

Screen Scraping Problems

The Selectors For Above Results

How do I DO THIS?

Entire Source

1 Answers1