get post title after Infinite scroll finished

Question

I manage to show all the post on a site where it has load_more button to go to the next page, but something is missing,

I got error of

e Error: Node is either not visible or not an HTMLElement
    at ElementHandle._clickablePoint (/Users/minghann/Documents/productnation_scraper/node_modules/puppeteer/lib/ExecutionContext.js:331:13)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

Which doesn't happen if I don't load all the post. It's hard to debug because I don't know which post is missing what. Full code as below:

const browser = await puppeteer.launch({
  devtools: true
});
const page = await browser.newPage();

await page.goto("https://example.net");

await page.waitForSelector(".load_more_btn");

const load_more_exist = !!(await page.$(".load_more_btn"));

while (load_more_exist > 0) {
  await page.click(".load_more_btn");
}

const posts = await page.$$(".post");

let result = [];
for (const post of posts) {
  result = [
    ...result,
    {
      title: await post.$eval(".post_title a", e => e.innerText)
    }
  ];
}

console.log(result);

browser.close();

Please be more specific: `getAllPostTitle function isn't working as expected` — how *is* it expected to work? — Vaviloff, Oct 28 '18 at 05:30
Thanks. But do you mean you load all the post or posts (plural)? How many times do you need to press load more button? Can you show the site? — Vaviloff, Oct 28 '18 at 09:19
@Vaviloff a lot, the site is http://productnation.co, the real selector is `jeg_post ` — Thian Kian Phin, Oct 28 '18 at 15:07

score 1 · Answer 1 · answered Oct 28 '18 at 17:33

There are multiple ways and best way is to combine the following two different ways.

Look for Ajax

Wait for request instead. Whenever you click on Load More, it will do a simple ajax request to ?ajax-request=jnews. We can use .waitForRequest or .waitForResponse for this use case. Here is a working example,

await Promise.all([
 page.waitForRequest(response => response.url().includes('?ajax-request=jnews') && response.status() === 200), 
 page.click(".load_more_btn")
])

Clean DOM and wait for new Element

Refer to these answers here and here.

Basically you can remove the dom elements that you collected, so next time you collect more data, there won't be any duplicates.

So, once you remove all current elements like document.querySelectorAll('.jeg_post'), you can simply do another page.waitFor('.jeg_post') later if you need.

get post title after Infinite scroll finished

1 Answers1

Look for Ajax

Clean DOM and wait for new Element