Python Selenium: How do I print the values from a website in a text file?

Question

I'm trying to write a script that will grab the following 6 values from the website tulsaspca.org and print them in a .txt file.

6 Values

The final output should be:

HTML for "Animals Placed"

<span class="number" data-to="905">905</span>
</div>
<p class="title">Animals Placed</p>

I wrote the following code, but it doesn't seem to be working.

for element in driver.find_elements_by_class_name('Animals Placed'):
  print(element.text)

@cruisepandey I have the 3 lines and the screenshot. Should I add more lines? — zackchess1, Nov 22 '21 at 05:59
Please replace the screenshot with the text of the html code. — karel, Nov 22 '21 at 06:02
Please check out the solution below. Also, Agree with Karel. It's easy for us to look into text rather than seeing an image. — cruisepandey, Nov 22 '21 at 06:02
You are missing the part where each of them are in some sort of list. So you can grab it in a xpath where it goes //span[@class='number'] — Arundeep Chohan, Nov 22 '21 at 06:12

cruisepandey · Answer 1 · 2021-11-22T06:35:16.717

I do not see HTML for all 6 numbers.

But for this HTML

<span class="number" data-to="905">905</span>
</div>
<p class="title">Animals Placed</p>

Your script should look something like this :

XPath

//p[text()='Animals Placed']/preceding-sibling::div/span[@class='number']

Please check in the dev tools (Google chrome) if we have unique entry in HTML DOM or not.

Steps to check:

Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.

Code trial 1:

time.sleep(5)
animal_num = driver.find_element_by_xpath("//p[text()='Animals Placed']/preceding-sibling::div/span[@class='number']").text
print(animal_num)

Code trial 2:

animal_num = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//p[text()='Animals Placed']/preceding-sibling::div/span[@class='number']"))).text
print(animal_num)

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Update:

Please use the below xpath

//span[@class='number' and @data-to]

it should represent all the number of nodes in HTML DOM.

driver.maximize_window()
driver.get("https://tulsaspca.org/")
driver.execute_script("window.scrollTo(0, 250)")
all_numbers = driver.find_elements(By.XPATH, "//span[@class='number' and @data-to]")
for number in all_numbers:
    print(number.text)

Output :

Thank you for the answer. So I ran it and I'm getting "25". Any ideas? https://i.imgur.com/JMB2pjL.png — zackchess1, Nov 22 '21 at 06:11
Please check the updated 1 code above for all the list items. — cruisepandey, Nov 22 '21 at 06:16
It also needed some scrolling as well to let selenium knows where exactly are the web elements. Please refer updated 1 code. — cruisepandey, Nov 22 '21 at 06:36

undetected Selenium · Accepted Answer · 2021-11-24T23:06:25.757

To grab the six values from the website TULSASPCA and print them in a text file you need to induce WebDriverWait for the visibility_of_all_elements_located() and then using List Comprehension you can create a list and subsequently create a DataFrame and finally export the values to a TEXT file excluding the Index using the following Locator Strategies:

Code Block:

driver.get("https://tulsaspca.org/")
driver.execute_script("window.scrollTo(0, 250)")
# read into a DataFrame
df = pd.DataFrame([my_elem.get_attribute("data-to") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='number']")))])
# Exporting as TEXT file excluding the Index
df.to_csv("C:\\Data_Files\\output_files\\new_text_marks.txt", index=False)
driver.quit()

Snapshot of Output Text file:

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

PS: You may like to drop the first row from the DataFrame

Python Selenium: How do I print the values from a website in a text file?

2 Answers2

Linked