Element not found when executing findElement in Selenium

Question

I'm trying to fill in multiple forms that come after each other, all the forms get filled swiftly with no errors because I make sure to add

WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("")));

before doing anything on a new page, and I know I'm on the correct page.

On the last form, I encounter this error :

Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to locate element: //*[@id="formtovalidate"]/fieldset[1]/div/label/input For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html

So I went to check on the browser by taking a screenshot and the browser is on the correct page with the correct form, I also checked the xpath values and even tried other attributes.. nothing seemed to work.

So I went ahead and printed out the PageSource which showed a totally different page (not the previous page), I also noticed the this page flashed for a second before the final form appeared.

I also tried driver.navigate().refresh() but that didn't work. I kept searching and looking but nothing appeared. I also changed browsers, that did nothing..

This is the method I'm trying to execute:

private void method() {

WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//*[@id=\"formtovalidate\"]/fieldset[1]/div/label/input")));
driver.findElement(By.xpath("//*[@id=\"formtovalidate\"]/fieldset[1]/div/label/input")).sendKeys(email); }

Update

Here's the form screenshot:

Here's the execution results:

Code:

String body_text = driver.findElement(By.tagName("body")).getText();
System.out.println(body_text);

Result: The form but in text

Code:

String body_innerHTML = driver.findElement(By.tagName("body")).getAttribute("innerHTML");
System.out.println(body_innerHTML);

Result: A different page :(

<zendesk-ticketing-form base-url="https://www.runescape.com/a=870/c=K0aO9WO69EI" css-cachebust="129" sitekey="6Lcsv3oUAAAAAGFhlKrkRb029OHio098bbeyi_Hv" grecaptcha="" has-valid-session="true" weblogin-url="https://secure.runescape.com/m=weblogin/a=870/c=K0aO9WO69EI/loginform?mod=www&amp;ssl=1&amp;dest=zendesk/support-form?form=360000065898">
<div class="x-display-none ie-error-display" data-js-ie-error="">
    <section class="c-article">
        <div class="c-article__content">
            <h1>Error: Unsupported Browser</h1>
            <p>
                We do not support your web browser. Please use a supported web browser by choosing one below.
                <br>
                <a href="https://www.mozilla.org/firefox/" target="_blank" rel="noopener">FireFox</a>
                <br>
                <a href="https://www.google.com/chrome/" target="_blank" rel="noopener">Chrome</a>
            </p>
        </div>
    </section>
</div>

Code:

 String pagesource = driver.getPageSource();
        System.out.println(pagesource);

Result: Same as the previous one.. different page..

Firefox Page Source: https://pastebin.com/Kv15V2SK

Firefox Inspect Element of the page screenshot: http://prntscr.com/qvi6hc

This is weird, as the page source is different to the form!

Firstly, you might want to provide URL, or even full page source so we can have a look. Second, please check if your element is under an iframe. If it's so, you have to switch your frame to that iframe in order to find your element. Third, if it isn't under an iframe, your xpath might be the problem. You should use relative xpath rather than full xpath. — Minh Dao, Jan 31 '20 at 04:56
@MinhDao Thank you for your reply, I've added in the page source.. I right clicked the form and clicked View Page Source.. its weird as its different than the form itself.. — DM Malkawi, Jan 31 '20 at 09:42
Well, your website has blocked your browser, and returned an error with message: `Error: Unsupported Browser`. In your page source, check the line 93, it looks the same with your screenshot, but the error shows up at line 99. — Minh Dao, Jan 31 '20 at 10:33
Honestly, I have never met this error. In my opinion, you need to update both your browser and driver, and try other browsers to find any differences. Hope this will help. — Minh Dao, Jan 31 '20 at 10:38
The browsers are at the latest update, i tried it on Firefox using the gecko driver and on Chrome using the chromedriver79 — DM Malkawi, Jan 31 '20 at 11:44

undetected Selenium · Answer 1 · 2020-01-31T14:20:47.977

1

The PageSource from the <body> tag, containing...

<zendesk-ticketing-form base-url="https://www.runescape.com/a=870/c=K0aO9WO69EI" css-cachebust="129" sitekey="6Lcsv3oUAAAAAGFhlKrkRb029OHio098bbeyi_Hv" grecaptcha="" has-valid-session="true" weblogin-url="https://secure.runescape.com/m=weblogin/a=870/c=K0aO9WO69EI/loginform?mod=www&amp;ssl=1&amp;dest=zendesk/support-form?form=360000065898">
<div class="x-display-none ie-error-display" data-js-ie-error="">
    <section class="c-article">
    <div class="c-article__content">
        <h1>Error: Unsupported Browser</h1>
        <p>
        We do not support your web browser. Please use a supported web browser by choosing one below.
        <br>
        <a href="https://www.mozilla.org/firefox/" target="_blank" rel="noopener">FireFox</a>
        <br>
        <a href="https://www.google.com/chrome/" target="_blank" rel="noopener">Chrome</a>
        </p>
    </div>
    </section>
</div>

...implies that the WebDriver driven Browsing Context was detected as a BOT and the navigation was blocked due to presence of reCAPTCHA.

There are different approaches to solve captcha / recaptcha. You can find a couple of relevant discussion in:

Update

From your comments now it is clear that you want to fill up the fields within the form:

form

At this point it is worth to mention that you had been redirected to this page for either of the following reasons:

You EmailID / UserID is banned / blocked from accessing the site.
You EmailID / UserID is black-listed from accessing the site.

As you have used a BOT to access/scrape the site which may have violated the T&C.

Solution

It would be tough to propose a solution to automatically fillup the fields as presumably the elements in the BAN APPEAL REQUEST page may be protected by Invisible reCAPTCHA and you may have to Programmatically invoke the challenge

edited Jan 31 '20 at 14:20

answered Jan 31 '20 at 10:40

undetected Selenium

183,867
41
278
352

Before I enter that page i solved a recaptcha using 2captcha's api.. how can I do the same for this because i guess its an invisible recaptcha? – DM Malkawi Jan 31 '20 at 11:43
@DMMalkawi Yes, it is _invisible captcha_, use the **sitekey** – undetected Selenium Jan 31 '20 at 11:49
aha, what should i submit? to solve the captcha? i can generate a response token. – DM Malkawi Jan 31 '20 at 11:55
Even when i try on my normal browser.. without selenium and i click view source.. it shows unsupported browser – DM Malkawi Jan 31 '20 at 13:06
@DMMalkawi I am still not sure if you are trying to fillup the fields within the fields within this [form](https://i.gyazo.com/257e54f2b32427d0882cfca14ae08292.png) or it is the resultant screen of the actions which you were trying to automate. – undetected Selenium Jan 31 '20 at 13:23
I'm trying to fill in the textfields and selectors found in the form http://prntscr.com/qvm42l but it's returning Element not found.. and when I try to load the page on my browser without selenium and then view page source.. it shows browser not supported.. tried it with several browsers and on the latest browsers.. – DM Malkawi Jan 31 '20 at 13:31
also, i noticed that the "Browser not supported" page flashes for a second before the form appears – DM Malkawi Jan 31 '20 at 13:32
@DMMalkawi Checkout the updated answer and let me know the status. – undetected Selenium Jan 31 '20 at 14:21
I tried solving the captcha by running this: https://pastebin.com/VR3jiyYU it didn't work unfortunately.. That form in the picture in your answer is the form I want to fill.. I want to be able to appeal using Selenium.. I tried to find if there are iframes or frames on the page but there aren't.. the form is made using zendesk.. any idea? – DM Malkawi Jan 31 '20 at 19:41
The button: "Submit" on the form displays a picture captcha when clicked. I've looked through the inspect element code and found somewhere where it said invisible `function onloadCallback(){grecaptcha.render("recaptchaHolder", {"size" : "invisible","sitekey" : "6Lcsv3oUAAAAAGFhlKrkRb029OHio098bbeyi_Hv","callback":function(resp){$("zendesk-ticketing-form").attr("grecaptcha",resp)}});}` But why is it when i go on the website by myself without selenium the view page source is still "Unsupported web browser" but the page itself is visible – DM Malkawi Jan 31 '20 at 22:25
also when i execute the script: `grecaptcha.execute();` it prompts up a picture captcha.. when i solved it, it went to the next page without filling anything.. so basically an empty form – DM Malkawi Jan 31 '20 at 22:27

Ahmet Aziz Beşli · Accepted Answer · 2020-02-01T03:31:24.407

I couldn't find time to solve your problem. If you want to do it on your own, please Search this on Google, "Shadow Root, Selenium", I had this kind of error before. What I know is, you cannot directly reach an element that stays inside of a shadow root, This is why you are not getting the source code inside of it.

What you need to do is go through the element step by step:

You have to expand the shadow root,

Here is shadow root expand function:

public static WebElement expand_shadow_element(WebElement element)
    {
        WebElement shadow_root = (WebElement)((JavascriptExecutor)driver).executeScript("return arguments[0].shadowRoot", element);
        return shadow_root;
    }

You can imagine this function like

.switchTo.frame()

for now..

After some researches you will understand the shadow root.

I hope I got the problem right..

Try this function, If you cannot, I will help you later on. Good Luck.

score 0 · Answer 3 · answered Jan 31 '20 at 12:23

0

As others have suggested, it appears RuneScape's website has detected that you're using a bot to interact with their site. It doesn't matter that you solved the captcha manually, as they can still detect automated behavior quite easily without one (and no, the navigator.webdriver flag is not their only way to detect this).

The captcha is meant to prevent automated interaction with their site, which means they don't want you using Selenium/WebDriver to interact with it. You should respect this, especially as it seems you want your account unbanned (going by the pasted snippets and screenshots), so trying to do exactly what they don't want won't win you any favors.

answered Jan 31 '20 at 12:23

Chris NeJame

11
1

I understand that, but I really do want to solve this. It'll give me knowledge in this field . – DM Malkawi Jan 31 '20 at 12:49
Solve what exactly? If it's captchas, you won't be able to, because there's some massive and very capable companies developing that technology. Google, for example, and they're so confident in it that they have a very hefty bounty out there for anyone who can beat it. If it's automating RuneScape's website, their website isn't your playground. If you want to practice, you can make a simple website of your own to practice against and host it locally (no need to pay for actual servers), which is actually pretty easy nowadays. That'll give you even more experience in the field. – Chris NeJame Jan 31 '20 at 22:37
I have already solved the first 2 captchas using the 2captcha api.. but this page.. i need its page source to be visible – DM Malkawi Jan 31 '20 at 23:04
I misspoke. I shouldn't have just said captcha. What you've gotten passed is a very old version of captcha. What you won't get passed is automated behavior detection. Even if you could, you'd still be abusing their website. Like I said, it's not your playground. You'll get much better experience in the field by making your own website, hosting it locally, and writing automation scripts against that. If you want the best experience, define your own acceptance criteria for your date website, and write automated tests for it. – Chris NeJame Feb 01 '20 at 13:28

Element not found when executing findElement in Selenium

Update

3 Answers3

Update

Solution