Loop increment not working over element [Puppeteer]

Question

I am working with Puppeteer and trying to get each item informations from Amazon.

I want get the 10 first result items from this article, but nothing can do

I already test all class on Chrome console, this returns what I want

(async() => {
   const browser = await puppeteer.launch({headless: false});
   const page = await browser.newPage();
   await page.setViewport({ width: 1366, height: 768});
   await page.goto("https://www.amazon.com/s?k=rtx+3070&ref=nb_sb_ss_pltr-ranker-24hours_2_2");

const itemList = [];

//class elements
const item = await page.$(".sg-col-20-of-24.s-result-item.s-asin.sg-col-0-of-12.sg-col-16-of-20.sg-col.s-widget-spacing-small.sg-col-12-of-16");
const item_title = await page.$(".a-size-medium.a-color-base.a-text-normal");
const item_img = await page.$(".s-image");
const item_price = await page.$(".a-price-whole");
const item_review = await page.$(".a-size-base.s-underline-text");

for (let i = 0; i < Math.min(9, item.length); i++) { 
    const title = await (await item_title[i].getProperty('innerText')).jsonValue();
    const img = await (await item_img[i].getAttribute('src'));
    const price = await (await item_price[i].getProperty('innerText')).jsonValue();
    const review = await (await item_review[i].getProperty('innerText')).jsonValue();

    //scroll action
    if (i % 4 === 5) {
      await page.evaluate( () => {
        window.scrollBy(0, window.innerHeight);
        new Promise(function(resolve) { 
          setTimeout(resolve, 3000)
        });
      });

      itemList.push(title, img, price, review)
      console.log(itemList);
    }
    console.log(textOnTheDiv);
    //console.log(imgAuteur);
  }; 
})()

Do you think my problem comes from the increment, guys? Thanks

@Errorbot1122THESECOND the problem is the people on this site want the code to be complete — TheRevenant95, Nov 14 '22 at 00:48
well, i currently cant uses dev tools, BUT you should try using adding `document.querySelector(".sg-col-20-of-24.s-result-item.s-asin.sg-col-0-of-12.sg-col-16-of-20.sg-col.s-widget-spacing-small.sg-col-12-of-16")` to the **devconsole** as that's all that `page.$` dose. [more info](https://pptr.dev/api/puppeteer.page._/) — Errorbot1122 THE SECOND, Nov 14 '22 at 00:53
This is the correct amount of code for a [mcve]. I don't see the problem here, though--isn't the data available simply in the static HTML, without scrolling? — ggorlen, Nov 14 '22 at 22:52
@Errorbot1122THESECOND yeah, i already try like this on chrome console — TheRevenant95, Nov 14 '22 at 22:52
True... page.$$ should work since page.$ only selects the first item — Errorbot1122 THE SECOND, Nov 15 '22 at 18:03
@ggorlan i try this but i dont know why i doesent work with my img variable too — TheRevenant95, Nov 15 '22 at 19:32

Errorbot1122 THE SECOND · Answer 1 · 2022-11-15T20:40:12.527

0

Since it seems like you are trying to loop through all elements in your class, you should probably replace:

const item = await page.$(".sg-col-20-of-24.s-result-item.s-asin.sg-col-0-of-12.sg-col-16-of-20.sg-col.s-widget-spacing-small.sg-col-12-of-16");

with:

const item = await page.$$(".sg-col-20-of-24.s-result-item.s-asin.sg-col-0-of-12.sg-col-16-of-20.sg-col.s-widget-spacing-small.sg-col-12-of-16");

as page.$ only selecta the first option while page.$$ selects all elements.

See More HERE

edited Nov 15 '22 at 20:40

answered Nov 15 '22 at 18:19

Errorbot1122 THE SECOND

68
10

It seem he isn't happy when i add `$$` on `const item_img = await page.$(".s-image");` he made this : `TypeError: item_img.getAttribute is not a function` – TheRevenant95 Nov 15 '22 at 19:30
Dont add it to `item_img` only to **item** – Errorbot1122 THE SECOND Nov 15 '22 at 19:53

ggorlen · Answer 2 · 2022-11-16T17:28:21.000

page.$ retrieves one element. Use page.$$ to select multiple. That said, I prefer not to assume all of these "parallel arrays" have the same length. Instead, I'd select the root elements, then query for all of the child elements within each parent container.

The first 10 elements are already on the page in the static HTML, so there's no need to scroll.

Perhaps try something like:

const puppeteer = require("puppeteer"); // ^19.0.0

const url = "<your URL>";

let browser;
(async () => {
  browser = await puppeteer.launch({headless: false});
  const [page] = await browser.pages();
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const data = await page.$$eval(
    ".s-result-item.s-asin.sg-col-0-of-12.sg-col-16-of-20",
    els => els.map(el => {
      const text = s => el.querySelector(s)?.textContent;
      return {
        title: text(".a-size-medium.a-color-base.a-text-normal"),
        img: el.querySelector(".s-image")?.src,
        price: text(".a-price-whole"),
        review: text(".a-size-base.s-underline-text"),
      };
    })
  );
  console.log(data);
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

You might need to tweak your review selector--I'm not sure what the expectation was for this.

Loop increment not working over element [Puppeteer]

2 Answers2