How to extract the text from the webelements using Selenium and Python

Question

Code trials:

driver.get(url)
cards = driver.find_elements_by_class_name("job-cardstyle__JobCardComponent-sc-1mbmxes-0")
for card in cards:
    data = card.get_attribute('text')
    print(data)

    
driver.close()
driver.quit()

The "cards" is returning selenium webelement and I am not able to extract the text from it by for loop.

Answer 1

Instead of get_attribute('text') you need to use the text attribute as follows:

data = card.text

---

Solution

To locate the visible elements you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use the following solution:

driver.get(url)
cards = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "job-cardstyle__JobCardComponent-sc-1mbmxes-0")))
for card in cards:
    data = card.text
    print(data)

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

---

Outro

In a single line you can use List Comprehension as follows:

print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "job-cardstyle__JobCardComponent-sc-1mbmxes-0")))])

Answer 2

1. Check your webelement path whether correctly mentioned or not
2. get the text from element
3. Print it

Answer 3

Issue is at this line

 data = card.get_attribute('text')

You can do the following:

1. Use .text

   for card in cards:
    data = card.text
    print(data)
   

2. Use innerText

   for card in cards:
    data = card.get_attribute('innerText')
    print(data)
   

Also, as per the comment above, you should print cards list length to debug it better.

print(len(cards))

so if it has something in it or not.

Answer 4

This works to an extent:

driver.get("https://www.monster.com/jobs/search?q=Python-Developer&where=Las+Vegas%2C+NV&page=1")
WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//*[@data-test-id = 'svx-job-title']")))
jobs = driver.find_elements(By.XPATH, "//div[contains(@class, 'job-cardstyle__JobCardHeader')]")
all_jobs = [job.text for job in jobs]
print(all_jobs)

WebdriverWait imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Output:

['Software engineer III\nRandstad USA\nLas Vegas, NV', 'C\nPython Developer\nconfidential\n$55 - $65 / Per Hour', 'C\nSenior Software Engineer\nCox Communications Inc\nLas Vegas, NV', 'Mission Systems Engineer\nDCS Corporation\nLas Vegas, NV', 'G\nSoftware Engineer - 914\nGCR Technical Staffing\nHenderson, NV', 'Z\nNetSuite Developer\nZone & Company Software Consulting\nLas Vegas, NV', 'IT Project Engineer\nRauland Florida by Ametek, Inc.\nSunrise, NV', 'A\nWeb Developer\nArdor Global', 'Senior Software Engineer – Node\nMeridian Technology Group Inc.']

Process finished with exit code 0

You may split the list with \n delimiter for further usage. Also, seems like this site loads the cards dynamically, i.e., as you scroll down, new cards load up, so you may not get all the cards in one instance.