Why cant Puppeteer find this link element on page?

Question

^^UPDATE^^
Willing to pay someone to walk me through this, issue posted on codeMentor.io: https://www.codementor.io/u/dashboard/my-requests/9j42b83f0p

I've been looking to click on the element:

<a id="isc_LinkItem_1$20j" href="javascript:void" target="javascript" tabindex="2"
onclick="if(window.isc_LinkItem_1) return isc_LinkItem_1.$30i(event);"
$9a="$9d">Reporting</a>

In: https://stackblitz.com/edit/js-nzhhbk (I haven't included the acutal page because its behind a username & pass)

seems easy enough
----------------------------------------------------------------------

solution1:

page.click('[id=isc_LinkItem_1$20j]') //not a valid selector

solution2:

const linkHandlers = await frame.$x("//a[contains(text(), 'Reporting')]");

if (linkHandlers.length > 0) {
    await linkHandlers[0].click();
} else {
    throw new Error('Link not found');
} //link not found

----------------------------------------------------------------------

I have looked at every which way to select and click it and it says it isn't in the document even though it clearly is (verified by inspecting the html in chrome dev tools and calling:page.evaluate(() => document.body.innerHTML))

**tried to see if it was in an iframe
**tried to select by id
**tried to select by inner text
**tried to console log the body in the browser (console logging not working verified on the inspect _element) //nothing happens
**tried to create an alert with body text by using:
_evaluate(() => alert(document)) // nothing happens
**tried to create an alert to test to see if javascript can be injected by:
_evaluate(() => alert('works')) // nothing happens
**also tried this: How to select elements within an iframe element in Puppeteer // doesn't work

Here is the code I have built so far

const page = await browser.newPage();

const login1url =
    'https://np3.nextiva.com/NextOSPortal/ncp/landing/landing-platform';
await page.goto(login1url);
await page.waitFor(1000);
await page.type('[name=loginUserName', 'itsaSecretLol');
await page.type('[name=loginPassword]', 'nopeHaha');
await page.click('[type=submit]');
await page.waitForNavigation();
const login3url = 'https://np3.nextiva.com/NextOSPortal/ncp/admin/dashboard';
await page.goto(login3url);
await page.click('[id=hdr_users]');
await page.goto('https://np3.nextiva.com/NextOSPortal/ncp/user/manageUsers');
await page.goto('https://np3.nextiva.com/NextOSPortal/ncp/user/garrettmrg');
await page.waitFor(2000);
await page.click('[id=loginAsUser]');
await page.waitFor(2000);
await page.click('[id=react-select-5--value');
await page.waitFor(1000);
await page.click('[id=react-select-5--option-0]');
await page.waitFor(20000);
const elementHandle = await page.$('iframe[id=callcenter]');
const frame = await elementHandle.contentFrame();
const linkHandlers = await frame.$x("//a[contains(text(), 'Reporting')]");

if (linkHandlers.length > 0) {
    await linkHandlers[0].click();
} else {
    throw new Error('Link not found');
}

Just ran it with that selector, this is the error: `UnhandledPromiseRejectionWarning: Error: No node found for selector: [id="isc_LinkItem_1$20j"]` — Raphael Castro, May 27 '20 at 20:45
Are you sure `await page.waitFor(time)` suffices? Did you try `page.waitForSelector()`? — vsemozhebuty, May 27 '20 at 20:51
Error: `UnhandledPromiseRejectionWarning: TimeoutError: waiting for selector "[id="isc_LinkItem_1$20j"]" failed: timeout 30000ms exceeded` and I checked in the developer tools, it does indeed exist. Do you think it might be in an iframe? — Raphael Castro, May 27 '20 at 20:55
Sorry, if it is in an iframe, this should be `frame.waitForSelector()`. But it is really hard to tell the issue without the live page( — vsemozhebuty, May 27 '20 at 20:57
How do I get the frame object, How would I find out what Iframe its in? — Raphael Castro, May 27 '20 at 21:00
Your way in the code seems formally valid, but without the live page, it is hard to understand what is going on. — vsemozhebuty, May 27 '20 at 21:01
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/214775/discussion-between-vsemozhetbyt-and-raphael-castro). — vsemozhebuty, May 27 '20 at 21:11
BTW, do you use headless? Could it be that the site prevents using a headless browser? Maybe you need to try headful? — vsemozhebuty, May 28 '20 at 19:01
scrape function uses - const browser = await puppet.launch({ headless: false, defaultViewport: null, }); — Raphael Castro, May 28 '20 at 19:02

score 1 · Answer 1 · answered May 28 '20 at 16:01

1

due isc_LinkItem_1$20j is not a valid selector, maybe you can try finding elements STARTING WITH isc_LinkItem_1 , like this

await page.waitForSelector("[id^=isc_LinkItem_1]", {visible: true, timeout: 30000});
await page.click("[id?=isc_LinkItem_1]);

?

answered May 28 '20 at 16:01

Andrea Bisello

1,077
10
23

innis · Answer 2 · 2020-05-28T17:26:31.043

1

On your solution1:

await page.click('a[id=isc_LinkItem_1\\$20j]');

Or try to:

await page.click('#isc_LinkItem_1\\$20j]');

I have the slight impression that you must provide what kind of element your trying to select before the brackets, in this case, an < a > element.

On the second solution, the # character means we're selecting an element by it's id

edited May 28 '20 at 17:26

answered May 28 '20 at 17:09

innis

370
1
3
14

I think that for whatever reason it isn't on the page but somewhere else, how is that possible? – Raphael Castro May 28 '20 at 17:11
1

I've edited my answer providing two backslashes before the $ in the selectors, try with that. Also, as I can see, the iframes on the link you've passed are closed right after they're open, which means they're empty – innis May 28 '20 at 17:28
not working, If it would make it easier we could do a screen share and I can show you the live element tree, NDA doesn't allow me to share credentials – Raphael Castro May 28 '20 at 17:33
No node found for selector. – Raphael Castro May 28 '20 at 17:34
1

What happens when you try to select the element by the browser console after the page is loaded? – innis May 28 '20 at 17:55
its there in the browser console, but I cant find it in puppeteer – Raphael Castro May 28 '20 at 18:10
1

If you're trying to click the element right after accessing the page, the DOM content might not be loaded yet, which may be the reason why you cannot find the element. Try to set an `await page.waifFor(5000)` before performing the click – innis May 28 '20 at 18:16

score 1 · Answer 3 · answered May 29 '20 at 16:20

1

It turns out that the previous click triggered a new tab. Puppeteer doesn't move to the new tab, all previous code was being executed on the old tab. To fix all we had to do was find the new tab, select it and execute code, here is the function we wrote to select for the tab:

async function getTab(regex, browser, targets) {
let pages = await browser.pages();
if (targets) pages = await browser.targets();
let newPage;
for (let i = 0; i < pages.length; i++) {
    const url = await pages[i].url();
    console.log(url);
    if (url.search(regex) !== -1) {
        newPage = pages[i];
        console.log('***');
        console.log(url);
        console.log('***');
        break;
    }
}
console.log('finished');
return newPage;
}

answered May 29 '20 at 16:20

Raphael Castro

907
1
8
26

1

BTW, you need not `await` for page or target `.url()`. – vsemozhebuty May 29 '20 at 17:06
1

Also, instead of the loop, you can try just `const newPage = pages.find(page => regex.test(page.url()));`, – vsemozhebuty May 29 '20 at 17:11
Great input both of these thing make the code more concise! – Raphael Castro May 30 '20 at 01:16

Why cant Puppeteer find this link element on page?

3 Answers3