Can't pull out a specific field available in every row in a table from a webpage

Question

I'm trying to get a particular field Further Information out of a table from a webpage using vba selenium bindings. I got success when I tried with python in combination with selenium using textContent instead of text as the latter is scraping nothing. However, the problem is I can't make use of this textContent within vba selenium. This is the link to my earlier post where I've asked this same question tagging different language.

Website link

I've tried with:

Sub ScrapeContent()
    Const URL$ = "https://www.sharedividends.com.au/mlt-dividend-history/"
    Dim driver As New ChromeDriver, elem As Object, R&

    driver.get URL

    For Each elem In driver.FindElementsByXPath("//*[@id='divTable']//tbody//tr[@role='row']", timeout:=10000)
        R = R + 1: Cells(R, 1) = elem.FindElementByXPath("(.//td)[8]").Text
    Next elem
End Sub

The field that I'm interested in:

When I run my above script, It fetches nothing. It doesn't throw any error either. FYI, the xpaths I've defined within the script is accurate.

How can I get that particular field (available in every row) out of the table from that website?

Answer 1

I suppose there isn't any elegant way of using textContent in vba selenium bindings. However, you can go with the following approach for now.

Sub ScrapeContent()
    Const URL$ = "https://www.sharedividends.com.au/mlt-dividend-history/"
    Dim driver As New ChromeDriver, elem As Object, oItem As Object, R&

    driver.get URL

    For Each elem In driver.FindElementsByXPath("//*[@id='divTable']//tbody//tr", timeout:=10000)
        Set oItem = elem.FindElementByXPath("(.//td)[8]", Raise:=False)

        If Not oItem Is Nothing Then
            R = R + 1: Cells(R, 1) = driver.ExecuteScript("return arguments[0].textContent;", oItem)
        End If
    Next elem
End Sub

Answer 2

Apologies didn't see your stipulation (my bad) on textContent and xpath so here is an alternative for future readers. However, looks like @sim has the covered.

Gather a collection of all elements matched by

.sorting_1

Loop that collection and click each one.

Then gather all elements matched by

[data-dt-column='7'] .dtr-data

and extract the .text property.

Something like:

Dim elem As Object, elems As Object

Set elems = driver.FindElementsByCss(".sorting_1")

For Each elem in elems
    elem.click
Next

Set elems = driver.FindElementsByCss("[data-dt-column='7'] .dtr-data")
For Each elem in elems
    Debug.Print elem.text
Next

Answer 3

There are a couple of things to observe in this answer to your previous question

Actually, the text Further Information 10.4C FRANKED @ 30%; DRP NIL DISCOUNT is available is 2 seperate place:

• When the circular button having plus sign is green with the attribute style="display: none;":

This the place from where your accepted solution reads the text using get_attribute('textContent')

---

Of coarse there can be a better approach.

• In this answer you will find the solution clicks on all the circular buttons having plus sign is green so they turns red as follows:

  for elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@aria-describedby='divTable_info']//tbody//tr/td[@class='sorting_1']"))):
      elem.click()
  

• Snapshot:

• The text Further Information 10.4C FRANKED @ 30%; DRP NIL DISCOUNT is read from a <span> tag using get_attribute("innerHTML") as follows:

---

Conclusion

Using the same logic you can use either .Text or .Attribute("innerHTML") through vba to cater to your needs.

You can find a detailed discussion regarding the usage of .Text or .Attribute("innerHTML") in Trying with Selenium + Excel VBA to scrape code from a site in Chrome Browser

Answer 4

You can try using .getAttribute method.

elem.FindElementByXPath("(.//td)[8]").getAttribute(...)