Python Selenium. How to make the "Keys" attribute work? How to find plain text on a site page? How to change page id to next automatically?

Question

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time



def main():
    driver = webdriver.Chrome()
    driver.maximize_window()
    driver.get("https://site.ru/")
    btn_elem1 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/ul/li[2]")
    btn_elem1.click()
    btn_elem2 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/p[1]/label/input")
    btn_elem2.click()
    btn_elem2.send_keys("login")
    btn_elem3 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/p[2]/label/input")
    btn_elem3.click()
    btn_elem3.send_keys("pass")
    btn_elem4 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/a")
    btn_elem4.click()
    time.sleep(1)
    driver.get("https://site/1")
    btn_elem5 = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[1]/fieldset/div[2]/img[1]")
    btn_elem5.click()

    action = ActionChains(driver)
    action.key_down(Keys.CONTROL).send_keys("f").key_up(Keys.CONTROL).perform()
    time.sleep(5)

if __name__ == "__main__":
    main()

Everything works up to a point.

The browser opens, expands. I pass to the necessary site. I go to the Login tab, enter the login and password. I’m going to another page that I don’t have access to if I’m not logged in. I open there a spoiler with hidden text.

After which I want to call "Find" via Ctrl + F, but for some reason, it doesn’t work, nothing happens. And in general, as I understand it, the keys themselves do not work. As if they were not added, although it seems to be written at the beginning of the code.

And if the necessary text is found, then take a screenshot and go to https: // site / 2, if not, then immediately go to site / 2 and search there. In total there are about ten thousand pages, probably the address of the village should be somehow made through a variable? And something like through ++ how to do it in PHP, like plus one, so that it is added every time?

Or maybe somehow it can be done differently? Search for plain text throughout the page. And if there is, then a screenshot and transition further, if not, then immediately move on. (The asterisk calls a screenshot)

sila = driver.find_element_by_xpath("/html")
if 'Селява' in sila.text:
   action.send_keys("*")
   action.key_down(Keys.CONTROL).send_keys("s").key_up(Keys.CONTROL).perform()
   driver.get("https://site/+1id") (вот как это сделать?)
else:
   driver.get("https://site/+1id") (вот как это сделать?)

Two problems remain in general. This is how to find the text and how to make it go to the next id page. I would be very grateful for any help.

Answer 1

Firstly, selenium does not allow you to simulate ctrl + f conveniently. If you want to find some text on the page, you can try one of two things-

• Find said text in the .page_source. However, be aware that .page_source contains the entire page's source code. So you might end up finding your desired text in here even though it wouldn't be visible on the frontend.

  if "my text" in driver.page_source:
      # "my text" is present in the source
      # do stuff
  

• Find an element that contains said text. You can also add extra steps to verify this element is actually visible. This method gives you a lot of flexibility and I'll leave the specifics up to you.

  # Make sure you've waited long enough for the page to be loaded
  # Find by XPATH
  try:
      elements = driver.find_elements_by_xpath('//*[contains(text(), "my text")]')
      # "my text" was found in page
      # do stuff
  except NoSuchElementException:
      # Text was not found in page
  

  One thing to keep in mind however, is that contains(text(), 'text') is case sensitive. If you'd like it to be case insensitive, refer to this answer (specifically the javascript preparation part, you can achieve the same with python)

Now, if you want the same operations to be performed on multiple URLs, by far the best approach would be to encapsulate the operations inside a function and call that function in a loop, passing all the URLs.

def automation_func(url):
    driver = webdriver.Chrome()
    driver.maximize_window()
    driver.get(url)
    btn_elem1 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/ul/li[2]")
    btn_elem1.click()
    btn_elem2 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/p[1]/label/input")
    btn_elem2.click()
    btn_elem2.send_keys("login")
    btn_elem3 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/p[2]/label/input")
    btn_elem3.click()
    btn_elem3.send_keys("pass")
    btn_elem4 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/a")
    btn_elem4.click()
    time.sleep(1)
    driver.get("https://site/1")
    btn_elem5 = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[1]/fieldset/div[2]/img[1]")
    btn_elem5.click()

    # more code here

Now you can call this function and pass in the urls-

for i in range(1, 5):
    automation_func(f"https://site.ru/{i}")

You may need to change up this loop according to how the URL looks like, but this is the main idea.

f"https://site.ru/{i}" evaluates to https://site.ru/1 in the first iteration, then https://site.ru/2 and so on.

Edit: This is how the code should be fitted together-

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time

def automation_func(url, i):
    driver = webdriver.Chrome()
    driver.maximize_window()
    driver.get(url)
    btn_elem1 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/ul/li[2]")
    btn_elem1.click()
    btn_elem2 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/p[1]/label/input")
    btn_elem2.click()
    btn_elem2.send_keys("login")
    btn_elem3 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/p[2]/label/input")
    btn_elem3.click()
    btn_elem3.send_keys("pass")
    btn_elem4 = driver.find_element_by_xpath("/html/body/div[2]/div[3]/div[2]/div[1]/div/div[2]/div/a")
    btn_elem4.click()
    time.sleep(1)
    driver.get("https://site/1")
    btn_elem5 = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[1]/fieldset/div[2]/img[1]")
    btn_elem5.click()
    time.sleep(5)
    # Find the text using one of the methods here
    if "small" in driver.page_source:
        screenshot = driver.save_screenshot("{i}.png")


def main():
    for i in range(1, 5):
        automation_func(f"https://site.ru/{i}", i)

if __name__ == "__main__":
    main()