0

So I am trying to scrap some useful data from a website .

Current Steps are

Wait for the particular Selector which holds the data.

and then use Page.evaluate to do some processing

Source Code

const link = "https://solscan.io/nfts#trades";

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

  await page.goto(link);
  await page.waitForSelector("div.ant-table-content");

  const tradeTable = await page.evaluate(() =>
    document.querySelector("div.ant-table-content")
  );
  console.log(tradeTable);
  await browser.close();
})();

All I try to do is console.log the result but for some reason it returns a undefined value.

rakshith yadav
  • 1
  • 3
  • `document.querySelector("div.ant-table-content")` should be `document.querySelector("div.ant-table-content").innerHTML`? You can't return DOM nodes from `evaluate`, only `evaluateHandle`. What data are you trying to get? – ggorlen Feb 03 '22 at 15:36
  • @ggorlen Hey, Thank You. I am trying to access a table of information So the above div would contain a table inside. – rakshith yadav Feb 03 '22 at 15:46
  • Sure, but what data exactly from that table, and in what format? Showing the exact output you expect saves a lot of guesswork and makes it possible for me to write a concrete answer taking you right to the solution. Otherwise, I have to make guesses that are often wrong, then there's a lot of back-and-forth and everyone wastes time. – ggorlen Feb 03 '22 at 15:46
  • Oh right, the format of the data is mostly text . Please feel free to check the below for better understanding . Thank You !! [https://solscan.io/nfts#trades](https://solscan.io/nfts#trades) – rakshith yadav Feb 03 '22 at 15:54
  • Does this answer your question? [Get elements from page.evaluate in Puppeteer?](https://stackoverflow.com/questions/53032903/get-elements-from-page-evaluate-in-puppeteer) – ggorlen Oct 01 '22 at 21:25

2 Answers2

0

To get a value using page.evaluate, you need to return it:

const someText = await page.evaluate(() =>
  var text = "Hi";
  return text;
);

In your case it looks like you're trying to return some div elements which I don't think is possible this way.

In puppeteer, waitForSelector will give you the element(s) you need, and you can then use them to do actions (clicks, get attributes, etc...):

const tradeTable = await page.waitForSelector("div.ant-table-content");

// get the div's text:
const textHandle = await tradeTable.getProperty("innerText");
const text = textHandle._remoteObject.value;

console.log(text);
lezhumain
  • 387
  • 2
  • 8
  • Thank You, I am trying to use page.waitForSelector in the above source code. But this is returning a timeout 30000 ms error. I think this is because the website which i am trying to scrap keeps updating the table every few seconds . So page.waitForSelector keeps waiting after 30000ms it throws a timeout error. I might be wrong in my reasoning but is there a work around for this ? – rakshith yadav Feb 03 '22 at 17:17
0

The reason you're getting undefined is because you're trying to return a non-serializable value from evaluate. Node objects from the DOM cannot be returned, so you would need to do something like this instead to instead extract a serializable value:

const result = await page.evaluate(() => return document.querySelector('.your-selector').innerText);

Alternatively you can return an ElementHandle for the specific node(s) you're looking for using this:

const node = await page.$('.your-selector');

// OR if you have multiple nodes

const nodeList = await page.$$('.your-selector');

Source: puppeteer docs

syedmh
  • 450
  • 2
  • 11