Selenium Python - "No such element: Unable to locate element" locating an input element within an iframe with headless chrome

Question

Hi I've been stuck for awhile on automating login for apple appstore as I am trying to make it headless without seeing the browser on execution.The problem is on execution with headless option and other options overall it doesn't find the field for appleid saying:

 selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="account_name_text_field"]"}

This is the code I tried so far:

options = selenium_chrome()
options.add_argument("--enable-javascript")
options.add_argument('headless')
options.add_argument('--window-size=1920x1080')
options.add_argument('--start-maximized')
options.add_argument("--disable-notifications")
options.add_argument('--no-sandbox')
options.add_argument('--verbose')
options.add_experimental_option("prefs", {
        "download.default_directory": DOWNLOAD_DIR,
        "download.prompt_for_download": False,
        "download.directory_upgrade": True,
        "safebrowsing_for_trusted_sources_enabled": False,
        "safebrowsing.enabled": False
})
options.add_argument('--disable-gpu')
options.add_argument('--disable-software-rasterizer')


driver = webdriver.Chrome('/path/to/chromedriver', chrome_options=options)

driver.get('https://appstoreconnect.apple.com/login')
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
    driver.add_cookie(cookie)
time.sleep(10)
   
f = driver.find_element_by_id("aid-auth-widget-iFrame")
driver.switch_to.frame(f)
# driver.implicitly_wait(15)
time.sleep(20)
driver.find_element_by_id('account_name_text_field').send_keys('appleid')
driver.find_element_by_id("sign-in").click()
time.sleep(5)

driver.find_element_by_id('remember-me').click()
time.sleep(3)
driver.find_element_by_id('password_text_field').send_keys('password')
driver.find_element_by_id("sign-in").click()

Notes:

I have tried getting the XPATH and CSS selectors instead of the ID and the result was the same.
Instead of time.sleep() I tried:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='account_name_text_field']"))).send_keys('appleid')

and the result was TimeoutException

Did a retry function,whenever the field is not found to keep trying until it is found, but that was kind of a bad approach and it didn't worked.

So I am out of ideas what causes this behavior,the field is being found without the options for selenium and it logins fine visually, but whenever options are being added it doesn't find the input text field for appleid anymore... Might be an Apple thing that I don't know.

So, it finds the element of you remove the "headless " option ? — PSR, Jan 24 '22 at 13:06
When selenium runs visually(without any option) it finds the element and it logins fine. Adding headless and other options seems to fail to find the element — mastaofthepasta, Jan 24 '22 at 13:08
does this help you => https://stackoverflow.com/questions/47316810/unable-to-locate-elements-on-webpage-with-headless-chrome — PSR, Jan 24 '22 at 13:08
No, I tried that as well even though I'm using a similar option for safe browsing in the question. — mastaofthepasta, Jan 24 '22 at 13:13
Are you able to identify which exact options aprt from "headless ", is giving you this error ? — PSR, Jan 24 '22 at 13:17
@PSR I commented all of the options and left only `options = selenium_chrome()`. Seems having `options` as `chrome_options` in driver is enough to give this error. — mastaofthepasta, Jan 24 '22 at 13:36

undetected Selenium · Accepted Answer · 2022-01-24T13:56:17.367

The <input> element with the placeholder as Apple ID is within an iframe so you have to:

Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.

You can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.get("https://appstoreconnect.apple.com/login")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#aid-auth-widget-iFrame")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#account_name_text_field"))).send_keys("mastaofthepasta")

Using XPATH:

driver.get("https://appstoreconnect.apple.com/login")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@id='aid-auth-widget-iFrame']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='account_name_text_field']"))).send_keys("mastaofthepasta")

Note : You have to add the following imports :

 from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.common.by import By
 from selenium.webdriver.support import expected_conditions as EC

Browser Snapshot:

appleStoreConnect

Reference

You can find a couple of relevant discussions in:

Update

When you use google-chrome-headless and print the page_source the output on the console is:

<html><head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>Apple</center>


<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
</body></html>

403 Forbidden

The HTTP 403 Forbidden response status code indicates that the server understands the request but refuses to authorize it.

This status is similar to 401, but for the 403 Forbidden status code re-authenticating makes no difference. The access is permanently forbidden and tied to the application logic, such as insufficient rights to a resource.

Conclusion

ChromeDriver initiated Chrome Browser gets detected as a bot and further navigation is blocked.

References

You can find a couple of relevant detailed discussions in:

When sending the keys. When it is being run without the options it works fine and you can see visually how the login is successful, but whenever options are being added to work in headless mode and other options it doesn't find the element or timeouts with the code that you provided. — mastaofthepasta, Jan 24 '22 at 13:21
@mastaofthepasta Checkout the updated answer and let me know the status. — undetected Selenium, Jan 24 '22 at 13:37
@undetectedSelenium Connection was refused - Failed to establish a new connection: [Errno 111] Connection refused')) — mastaofthepasta, Jan 24 '22 at 13:49
[Errno 111 - Connection refused](https://stackoverflow.com/questions/52686968/maxretryerror-httpconnectionpool-max-retries-exceeded-caused-by-protocolerror/52701504#52701504) is a much broader error. — undetected Selenium, Jan 24 '22 at 13:53
That worked! Removing `driver.quit()` fixes the connection error and regarding the 403 it seems proxy has to be used. — mastaofthepasta, Jan 24 '22 at 13:59

Prophet · Answer 2 · 2022-01-24T13:57:37.333

0

Looks like your problem is here:
Instead of

options.add_argument('headless')

It should be

options.add_argument('--headless')

i.e. you are missing -- before the option argument
UPD
Instead of

driver = webdriver.Chrome('/path/to/chromedriver', chrome_options=options)

It could be

driver = webdriver.Chrome('/path/to/chromedriver', options=options)

edited Jan 24 '22 at 13:57

answered Jan 24 '22 at 13:22

Prophet

32,350
22
54
79

That was a typo, but didn't change anything – mastaofthepasta Jan 24 '22 at 13:30
This typo could cause problems.. – Prophet Jan 24 '22 at 13:31
As mentioned by PSR, try removing all the unnecessary options. Leave only `('--headless')` and `('--window-size=1920x1080')` and remove all the others for now and see if this makes your code work better – Prophet Jan 24 '22 at 13:34
Tried that as well... as with little options, with more options and no options. The only working way is with no options... – mastaofthepasta Jan 24 '22 at 13:51