How can I extract a list of id's from a webpage?

Question

I have this code in a web page I am trying to scrape with selenium:

 <h4 class="category" id="sc_14295">...</h4>
 <h4 class="category" id="sc_14292">...</h4>
 <h4 class="category" id="sc_14291">...</h4>
 <h4 class="category" id="sc_14299">...</h4>

I have tried to implement a method like this (which I found here on SO) to get at least one id:

String id = driver.findElement(By.xpath("//div[@class='category']")).getAttribute("id");

in this way:

String id = driver.findElement(By.xpath("//h4[@class='category']")).getAttribute("id");

It doesn't work. I assume I should be using findElements() but that can't have getAtttribute. How can I extract a list of all id's? Thanks.

Answer 1

edit##

Does not saw the java tag, but procedure is the same.

---

This get one element and you can grab the id directly:

driver.find_element_by_xpath("//h4[@class = 'category']").get_attribute("id")   

This get multiple elements, but you have to loop them to get the id of each:

driver.find_elements_by_xpath("//h4[@class = 'category']"):

Example

from selenium import webdriver
driver = webdriver.Chrome('C:\Program Files\ChromeDriver\chromedriver.exe')

html_content = """
<h4 class="category" id="sc_14295">...</h4>
 <h4 class="category" id="sc_14292">...</h4>
 <h4 class="category" id="sc_14291">...</h4>
 <h4 class="category" id="sc_14299">...</h4>
"""

driver.get("data:text/html;charset=utf-8,{html_content}".format(html_content=html_content))


for element in driver.find_elements_by_xpath("//h4[@class = 'category']"):
        print(element.get_attribute("id"))

driver.close()

Output

sc_14295
sc_14292
sc_14291
sc_14299

Answer 2

To print the List of id attribute of the elements you need to induce WebDriverWait for the visibilityOfAllElementsLocatedBy() and you can use Java8 stream() and map() and you can use either of the following Locator Strategies:

• cssSelector:

  System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.cssSelector("h4.category"))).stream().map(element->element.getAttribute("id")).collect(Collectors.toList()));
  

• xpath:

  System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//h4[@class='category']"))).stream().map(element->element.getAttribute("id")).collect(Collectors.toList()));