Selenium find attribute value (Python)

Question

I would like to get all attribute values names 'href' from a website, there are like 10 of them. I have successfully got one using the following method:

url = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[2]/div[2]/div/div/div[2]/section[1]/div/div[2]/div[3]/ul/li[1]/div/div[1]/span/a"))).get_attribute("href")

The problem with this is it's only giving back one not all of them. I have tried to go by ID but it doesn't return anything:

url = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "a"))).get_attribute("href")

Also, the other href values are located in different xpaths:

/html/body/div[2]/div[2]/div/div/div[2]/section[1]/div/div[2]/div[3]/ul/li[2]/div/div[1]/span/a
/html/body/div[2]/div[2]/div/div/div[2]/section[1]/div/div[2]/div[3]/ul/li[3]/div/div[1]/span/a
/html/body/div[2]/div[2]/div/div/div[2]/section[1]/div/div[2]/div[3]/ul/li[4]/div/div[1]/span/a

Here's my element:

<a ph-tevent="job_click" ref="linkEle" href.bind="getUrl(linkEle, 'job', eachJob, '', eachJob.jobUrl)" data-ph-at-id="job-link" data-ph-id="ph-page-element-page20-CRUCUZ" class="au-target" au-target-id="181" ph-click-ctx="job" ph-tref="12313123213" ph-tag="ph-search-results-v2" href="https://hyperlink.com" data-ph-at-job-title-text="title" data-ph-at-job-location-text="Unknown" data-ph-at-job-location-area-text="asd" data-ph-at-job-category-text="Manufacturing" data-access-list-item="2" data-ph-at-job-id-text="A123124" data-ph-at-job-type-text="Regular" data-ph-at-job-industry-text="Manufacturing" data-ph-at-job-post-date-text="2021-12-09T00:00:00.000Z" data-ph-at-job-seqno-text="ASD212ASFS" aria-label="Senior Manager"> 
<div class="job-title" data-ph-id="ph-page-element-page20-0Mi3Ce"> 
<!--anchor--> 
<!--anchor--> 
<span data-ph-id="ph-page-element-page20-PLxqta">Senior Manager </span> 
</div><!--anchor--> </a>

Any help is appreciated!

try using CSS_SELECTOR instead of ID find_elements_by_css_selector("a") returns a list of elements, then iterate over that list and get_attribute("href") — Bendik Knapstad, Jan 13 '22 at 13:19
@vboxer00 Can you update the question with the HTML of another position similar to _Senior Manager_ so we can find the common attributes for the desired elements? — undetected Selenium, Jan 13 '22 at 13:50

cruisepandey · Accepted Answer · 2022-01-13T14:06:34.300

For finding more than one web element, you should either use find_elements or if you are using Explicit waits then you can use presence_of_all_elements_located or visibility_of_all_elements_located.

Based on the HTML that you've shared, if

this css

a[ph-tevent='job_click'][ref='linkEle']

or this xpath

//a[@ph-tevent='job_click' and @ref='linkEle']

represent all the nodes, to check below are the steps:

Please check in the dev tools (Google chrome) if we have all desired nodes entry in HTML DOM or not.

Steps to check:

Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired elements are getting highlighted or not.

If they are then the below code should work:

for ele in driver.find_elements(By.XPATH, "//a[@ph-tevent='job_click' and @ref='linkEle']"):
    print(ele.get_attribute('href'))

Possibly your xpath won't compile due to invalid syntax. You need an _`and`_ between the attributes. — undetected Selenium, Jan 13 '22 at 14:05

undetected Selenium · Answer 2 · 2022-01-13T14:06:15.387

To extract the value of the href attributes using Selenium and python you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[ph-tevent='job_click'][ref='linkEle'][data-ph-at-id='job-link'][ph-click-ctx='job'][href]")))])

Using XPATH:

print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[@ph-tevent='job_click' and @ref='linkEle'][@data-ph-at-id='job-link' and @ph-click-ctx='job'][@href]")))])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Selenium find attribute value (Python)

2 Answers2