I need help Identifying a web form field for use with Selenium

Question

In an effort to learn more about Python, Selenium, Beautifulsoup, and webscraping in general I started working a small personal project that should ultimately be able to open a webpage's home screen, navigate to a search field, and enter and search for a specific account number.

Currently I have been successful in using Selenium to open the sites home page and then navigate to the page that contains the form I need to fill. I have done this by using Chromes developer tools to copy the xpath of the area that I need to click on and then feeding that into Selenium's "find element" and "click" functions.

The trouble I am having is that the xpath for the specific field that I am trying to click in, when fed into my code, spits out teh following error

Traceback (most recent call last):
  File "C:/Users/WFrazierIII/PycharmProjects/Selenium Prcatice/Selenium Practice File.py", line 24, in <module>
    driver.find_element_by_xpath('/html/body/form/table/tbody/tr[2]/td[2]/input').click()
  File "C:\Users\WFrazierIII\PycharmProjects\Selenium Prcatice\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "C:\Users\WFrazierIII\PycharmProjects\Selenium Prcatice\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
    'value': value})['value']
  File "C:\Users\WFrazierIII\PycharmProjects\Selenium Prcatice\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Users\WFrazierIII\PycharmProjects\Selenium Prcatice\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/form/table/tbody/tr[2]/td[2]/input"}
  (Session info: chrome=83.0.4103.97)


Process finished with exit code 1

My question is, should I work on finding the proper xpath for this field or should I attempt to use a different "find_element" for a better chance of making my code work.

If it helps, the field I am trying to click on is the "account" field on this website: https://www.alabamagis.com/Tallapoosa/frameset.cfm?cfid=3986761&cftoken=54089502

Here is my current code so far:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import presence_of_element_located

driver = webdriver.Chrome('C:\WebDriver\chromedriver.exe')

driver.get('https://www.alabamagis.com/Tallapoosa/')
driver.find_element_by_xpath('/html/body/center/font[2]/a').click()

time.sleep(6)

driver.find_element_by_xpath('/html/body/font/a[1]').click()

time.sleep(6)

driver.find_element_by_xpath('/html/body/form/table/tbody/tr[2]/td[2]/input').click()

Finally, I'll just add that I know very little of Python or Selenium and almost nothing about HTML (hence the lame starter project) so if you think there are better ways to approach this then I would love the guidance.

Thank!

Answer 1

The problem you are facing is that the form you want to access, resides inside an iFrame. You have to specifically tell selenium to switch to the iFrame. You can do it like this.

driver = webdriver.Chrome('./chromedriver')

driver.get('https://www.alabamagis.com/Tallapoosa/')
driver.find_element_by_xpath('/html/body/center/font[2]/a').click()

time.sleep(6)

driver.find_element_by_xpath('/html/body/font/a[1]').click()

time.sleep(6)

# new code starts here
driver.switch_to.frame('searchFrame')
driver.find_element_by_name('Master__Account').send_keys('your desired input')

Also, an easier way to find the input you are looking for is by using find_element_by_name. If you check the HTML code for the website you are trying to scrape, the input field you want to access has a name attribute with the value Master__Account. You can use that to access the input.

Answer 2

You are not sending the account number to the Textfield, try this driver.find_element_by_xpath('/html/body/form/table/tbody/tr[2]/td[2]/input').send_keys('account_no')

send_keys is used to send input to textfield and

driver.find_element_by_xpath('/html/body/form/table/tbody/tr[11]/td/input[1]').click()

to hit the search button

Answer 3

The Account field is within a <frame> so you have to:

• Induce WebDriverWait for the desired frame to be available and switch to it.
• Induce WebDriverWait for the desired element to be clickable.

• You can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get('https://www.alabamagis.com/Tallapoosa/frameset.cfm?cfid=3986761&cftoken=54089502')
    WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='searchFrame']")))
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='Master__Account']"))).send_keys("Sean Ken")
    

  • Using XPATH:

    driver.get('https://www.alabamagis.com/Tallapoosa/frameset.cfm?cfid=3986761&cftoken=54089502')
    WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//frame[@name='searchFrame']")))
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='Master__Account']"))).send_keys("Sean Ken")
    

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    

• Browser Snapshot:

---

Reference

You can find a couple of relevant discussions in:

• Ways to deal with #document under iframe
• Switch to an iframe through Selenium and python