Selenium not finding custom attribute 'data-target-section-id'

Question

I'm trying to scrape some profiles of people in linkedin from a specific job. To do this I was trying to find the people button and click it to specifically look at the relevant people.

The path is as follows:

From signed out Linkedin home -> I sign in and go to LinkedIn home -> I write in the search bar "hr" and hit enter.

In the result page of hr, on the left side of the page, there is a navigation list that says "On this page". One of the options includes "People" and that is what I want to target.

The link to the page is: https://www.linkedin.com/search/results/all/?keywords=hr&origin=GLOBAL_SEARCH_HEADER&sid=Xj2

The HTML of the button for 'People' in the navigation list is:

<li>
   <button aria-current="false" class="search-navigation-panel_button" data-target-section-id="PTFmMNSPSz2LQRzwynhRBQ==" role="link" type="button"> People

I have tried to find this button through By.Link_text and found the keyword People but did not work. I have also tried to do By.XPATH "//button[@data-target-section-id='RIK0XK7NRnS21bVSiNaicw==']")"" but it also does not find it.

How can I make selenium find this custom attribute so I can find this button through data-target-section-id="PTFmMNSPSz2LQRzwynhRBQ=="?

Another issue that I am having is that I can target all the relevant people on the page and loop through them but I cannot extract the link of each of the profiles. It only takes the first link of the first person and never updates the variable again through the loop.

For example, if the first person is Ian, and the second is Brian, it gives me the link for Ian's profile even if 'users' is Brian.

Debugging the loop I can see the correct list of people in all_users but it only gets the href of the first person in the list and never updates.

Here is the code of that:

all_users = driver.find_elements(By.XPATH, "//*[contains(@class, 'entity-result__title-line entity-result__title-line--2-lines')]")

for users in all_users:
    print(users)
    get_links = users.find_element(By.XPATH, "//*[contains(@href, 'miniProfileUrn')]")
    print(get_links.get_attribute('href'))

Answer 1

If you want to locate several elements with the same attribute replace find_element with find_elements. See if that works to find not just the first element matching your search, but all elements with that attribute.

Review the Selenium: Locating Elements documentation and see if you can try each and every option they have for locating elements.

Something else to try:

button_element = driver.find_element(By.XPATH, "//button[@data-target-section-id='RIK0XK7NRnS21bVSiNaicw==']")
list_element.find_element(By.TAG_NAME, "button").click()

Answer 2

I have also tried to do By.XPATH "//button[@data-target-section-id='RIK0XK7NRnS21bVSiNaicw==']")"" but it also does not find it.

The data-target-section-id that you mention is not the same as the one that the button has (PTFmMNSPSz2LQRzwynhRBQ==). Check that this is not dynamic before targeting it.

Your xPath is not bad but as I told you, fix the target-id:

driver.findElement(By.xpath("//button[@data-target-section-id='PTFmMNSPSz2LQRzwynhRBQ==']")).click()

Where "driver" is your WebDriver instance.

Answer 3

Given the HTML:

<li>
   <button aria-current="false" class="search-navigation-panel_button" data-target-section-id="PTFmMNSPSz2LQRzwynhRBQ==" role="link" type="button"> People </button>
</li>

The data-target-section-id attribute values like PTFmMNSPSz2LQRzwynhRBQ== are dynamically generated and is bound to chage sooner/later. They may change next time you access the application afresh or even while next application startup. So can't be used in locators.

---

Solution

The desired element being a dynamic element to click on the clickable element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.search-navigation-panel_button[data-target-section-id]"))).click()
  

• Using XPATH:

  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@class='search-navigation-panel_button' and @data-target-section-id][contains(., 'People')]"))).click()
  

• Note: You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

Answer 4

It looks like the reason your People button locator isn't working is because the data-target-section-id is dynamic. Mine is showing as hopW8RkwTN2R9dPgL6Fm/w==. We can get around that by using an XPath to locate the element based on the text contained, "People", e.g.

//button[text()='People']

Turns out that matches two elements on the page because many of the left nav links are repeated as rounded buttons on the top of the page so we can further refine our locator to

//button[text()='People'][@data-target-section-id]

Having said that, that link only scrolls the page so you don't really need to click that.

From there, you want to get the links to each person listed under the People heading. We first need the DIV that contains the People section. It's kinda messy because the IDs on those elements are also dynamic so we need to find the H2 that contains "People" and then work our way back up the DOM to the DIV that contains only that section. We can get that using the XPath below

//div[@class='search-results-container']/div[.//h2[text()='People']]

From there, we want all of the A tags that uniquely link to a person... and there's a lot of A tags in that section but most are not ones we want so we need to do more filtering. I found that the below XPath locates each unique URL in that section.

//a[contains(@href,'miniProfileUrn')][contains(@class,'scale-down')]

Combining the two XPaths, we get

//div[@class='search-results-container']/div[.//h2[text()='People']]//a[contains(@href,'miniProfileUrn')][contains(@class,'scale-down')]

which locates all unique URLs belonging to a person in the People section of the page.

Using this, your code would look like

all_users = driver.find_elements(By.XPATH, "//div[@class='search-results-container']/div[.//h2[text()='People']]//a[contains(@href,'miniProfileUrn')][contains(@class,'scale-down')]")

for user in all_users:
    print(user.get_attribute('href'))

---

NOTE: The reason your code was only returning the first href repeatedly is because you are searching from an existing element with an XPath so you need to add a "." at the start of the XPath to indicate to start searching from the referenced element.

get_links = users.find_element(By.XPATH, ".//*[contains(@href, 'miniProfileUrn')]")
                                          ^ add period here

I've eliminated that step in my code so you won't need it there.