1

I have a puppeteer script that inputs some text into a field, submits the query, and processes the results.

Currently, the script only processes 1 search term at a time, but I need it to be able to process an array of items consecutively.

I figured I would just put the code in a loop (see code below), however, it just types in all the items from the array at once into the field and doesn't execute the code block for each search term:

  for (const search of searchTerms) {
    await Promise.all([
      page.type('input[name="q"]', 'in:spam ' + search + String.fromCharCode(13)),
      page.waitForNavigation({
          waitUntil: 'networkidle2'
        })
    ]);

    const count = await page.evaluate((sel) => {
      return document.querySelectorAll(sel)[1].querySelectorAll('tr').length;
    }, 'table[id^=":"]');

    if (count > 0) {
      const more = await page.$x('//span[contains(@class, "asa") and contains(@class, "bjy")]');
      await more[1].click();

      await page.waitFor(1250);
      const markRead = await page.$x('//div[text()="Mark all as read"]');
      await markRead[0].click();

      const selectAll = await page.$x('//span[@role="checkbox"]');
      await selectAll[1].click();

      const move = await page.$x('//div[@act="8"]');
      await move[0].click();

      await page.waitFor(5000);
    }
  }

I tried using a recursion function from Nodejs Synchronous For each loop

I also tried using a function generator with yields, as well as promises and even tried the eachSeries function from the async package from this post Nodejs Puppeteer Wait to finish all code from loop

Nothing I tried was successful. Any help would be appreciated, thanks!

Hyyan Abo Fakher
  • 3,497
  • 3
  • 21
  • 35
Native Coder
  • 57
  • 7
  • >Currently, the script only process 1 search term at a time, but I need it to be able to process an array of items consecutively. Do you mean... in parallel? Consecutively means one after the other, in order. – Cody Ｇ Sep 12 '18 at 20:51

1 Answers1

3

There is no way to visit two websites at same time with same tab. You can try it on your browser to make sure.

Jokes aside, if you want to search multiple items, you have to create a page or tab for that. 

for (const search of searchTerms) {
  const newTab = await browser.newPage()
  // other modified code here
}

... wait that will still search one by one. But if you use a map with concurrency limit, it will work well.

We can use p-all for this.

const pAll = require('p-all');
const actions = []
for (const search of searchTerms) {
  actions.push(async()=>{
    const newTab = await browser.newPage()
    // other modified code here
  })
}
pAll(actions, {concurrency: 2}) // <-- set how many to search at once

So we are looping thru each term, and adding a new promise on the action list. Adding functions won't take much time. And then we can run the promise chain.

You will still need to modify the code above to have what you desire. Peace!

Md. Abu Taher
  • 17,395
  • 5
  • 49
  • 73
  • That is not what I'm trying to do at all. I have only 1 tab open, and want to execute the block of code x number of times within the same page. – Native Coder Sep 12 '18 at 21:10
  • @NativeCoder I believe the code you posted should already do that. `for (const search of searchTerms)` means that terms will be processed in series. If you need to do this in parallel, you need a thing like suggested in this answer. – Estus Flask Sep 12 '18 at 21:16
  • actually, you're right, it does work just fine lol, no results were being returned so it would execute the next search. I just needed to clear the field and now it works perfectly. Thank you guys for your help lol and sorry for my mistakes! – Native Coder Sep 12 '18 at 21:38
  • @NativeCoder If you return the data, you can get a result after all actions are done. ;) pAll itself returns a promise. – Md. Abu Taher Sep 12 '18 at 22:50