Get "href=link" from html page and navigate to that link using vba

Question

I am writing a code in Excel VBA to get href value of a class and navigate to that href link (i.e) here is the href value I want to get into my particular Excel sheet and I want to navigate to that link automatically through my VBA code.

<a href="/questions/51509457/how-to-make-the-word-invisible-when-its-checked-without-js" class="question-hyperlink">How to make the word invisible when it's checked without js</a>

The result I'm getting is that I'm able to get that containing tag's class value How to make the word invisible when it's checked without js <---- this is title is what I am getting in my sheet. What I want to get is this title's holding a href link /questions/51509457/how-to-make-the-word-invisible-when-its-checked-without-js this is what I want to get and navigate through my code.

Please help me out. Thanks in advance

Below are the entire coding:

Sub useClassnames()
    Dim element As IHTMLElement
    Dim elements As IHTMLElementCollection
    Dim ie As InternetExplorer
    Dim html As HTMLDocument

    'open Internet Explorer in memory, and go to website
    Set ie = New InternetExplorer
    ie.Visible = True
    ie.navigate "https://stackoverflow.com/questions"
    'Wait until IE has loaded the web page

    Do While ie.readyState <> READYSTATE_COMPLETE
        DoEvents
    Loop

    Set html = ie.document
    Set elements = html.getElementsByClassName("question-hyperlink")

    Dim count As Long
    Dim erow As Long
    count = 0

    For Each element In elements
        If element.className = "question-hyperlink" Then
            erow = Sheets("Exec").Cells(Rows.count, 1).End(xlUp).Offset(1, 0).Row
            Sheets("Exec").Cells(erow, 1) = html.getElementsByClassName("question-hyperlink")(count).innerText
            count = count + 1
        End If
    Next element

    Range("H10").Select
End Sub

I cant find any answer in this website asked by anyone. Please don't suggest this question as duplicate.

<div class="row hoverSensitive">
        <div class="column summary-column summary-column-icon-compact  ">
                                <img src="images/app/run32.png" alt="" width="32" height="32">
                        </div>
        <div class="column summary-column  ">
            <div class="summary-title summary-title-compact text-ppp">
                                        <a href="**index.php?/runs/view/7552**">MMDA</a>

            </div>
            <div class="summary-description-compact text-secondary text-ppp">
                                                                            By on 7/9/2018                                                  </div>
        </div>      
        <div class="column summary-column summary-column-bar  ">
                            <div class="table">
<div class="column">
    <div class="chart-bar ">
                                                                                                                        <div class="chart-bar-custom link-tooltip" tooltip-position="left" style="background: #4dba0f; width: 125px" tooltip-text="100% Passed (11/11 tests)"></div>
                                                                                                                                                                                                                                                                                                                                                                                                            </div>
</div>
    <div class="column chart-bar-percent chart-bar-percent-compact">
    100%'

it looks like where you have innerText you need getAttribute("href") — Jeremy Kahan, Jul 25 '18 at 01:47

QHarr · Answer 1 · 2018-07-25T11:50:12.923

Method ①

Use XHR to make initial request using question homepage URL; apply CSS selector to retrieve links and then pass those links to IE to navigate to

CSS selectors to select elements:

You want the href attribute of the element.You have been given an example already. You can use getAttribute, or, as pointed out by @Santosh, combine an href attribute CSS selector with other CSS selectors to target the elements.

CSS selector:

a.question-hyperlink[href]

Looks for elements with parent a tag having class of question-hyperlink and an href attribute.

You then apply the CSS selector combination with the querySelectorAll method of document to gather a nodeList of the links.

XHR to get initial list of links:

I would issue this first as an XHR, as much faster, and gather your links into a collection/nodeList you can later loop with your IE browser.

Option Explicit
Public Sub GetLinks()
    Dim sResponse As String, HTML As New HTMLDocument, linkList As Object, i As Long
    Const BASE_URL As String = "https://stackoverflow.com"
    With CreateObject("MSXML2.XMLHTTP")
        .Open "GET", "https://stackoverflow.com/questions", False
        .send
        sResponse = StrConv(.responseBody, vbUnicode)
    End With
    sResponse = Mid$(sResponse, InStr(1, sResponse, "<!DOCTYPE "))

    With HTML
        .body.innerHTML = sResponse
        Set linkList = .querySelectorAll("a.question-hyperlink[href]")
        For i = 0 To linkList.Length - 1
            Debug.Print Replace$(linkList.item(i), "about:", BASE_URL)
        Next i
    End With
    'Code using IE and linkList
End Sub

In the above linkList is a nodeList holding all the matched elements from the homepage i.e. all the hrefs on the question landing page. You can loop the .Length of the nodeList and index into it to retrieve a particular href e.g. linkList.item(i). As the link returned is relative, you need to replace the relative about: part of the path with the protocol + domain i.e. "https://stackoverflow.com".

Now that you have quickly obtained that list, and can access items, you can pass any given updated href onto IE.Navigate.

Navigating to questions using IE and nodeList

For i = 0 To linkList.Length - 1
    IE.Navigate Replace$(linkList.item(i).getAttribute("href"), "about:", BASE_URL)
Next i

Method ②

Use XHR to make initial request using GET request and searched for question title; apply CSS selector to retrieve links and then pass those links to IE to navigate to.

Option Explicit
Public Sub GetLinks()
    Dim sResponse As String, HTML As New HTMLDocument, linkList As Object, i As Long
    Const BASE_URL As String = "https://stackoverflow.com"
    Const TARGET_QUESTION As String = "How to make the word invisible when it's checked without js"
    With CreateObject("MSXML2.XMLHTTP")
        .Open "GET", "https://stackoverflow.com/search?q=" & URLEncode(TARGET_QUESTION), False
        .send
        sResponse = StrConv(.responseBody, vbUnicode)
    End With
    sResponse = Mid$(sResponse, InStr(1, sResponse, "<!DOCTYPE "))

    With HTML
        .body.innerHTML = sResponse
        Set linkList = .querySelectorAll("a.question-hyperlink[href]")
        For i = 0 To linkList.Length - 1
            Debug.Print Replace$(linkList.item(i).getAttribute("href"), "about:", BASE_URL)
        Next i
    End With
    If linkList Is Nothing Then Exit Sub
    'Code using IE and linkList
End Sub

'https://stackoverflow.com/questions/218181/how-can-i-url-encode-a-string-in-excel-vba   @Tomalak
Public Function URLEncode( _
   StringVal As String, _
   Optional SpaceAsPlus As Boolean = False _
) As String

  Dim StringLen As Long: StringLen = Len(StringVal)

  If StringLen > 0 Then
    ReDim result(StringLen) As String
    Dim i As Long, CharCode As Integer
    Dim Char As String, Space As String

    If SpaceAsPlus Then Space = "+" Else Space = "%20"

    For i = 1 To StringLen
      Char = Mid$(StringVal, i, 1)
      CharCode = Asc(Char)
      Select Case CharCode
        Case 97 To 122, 65 To 90, 48 To 57, 45, 46, 95, 126
          result(i) = Char
        Case 32
          result(i) = Space
        Case 0 To 15
          result(i) = "%0" & Hex(CharCode)
        Case Else
          result(i) = "%" & Hex(CharCode)
      End Select
    Next i
    URLEncode = Join(result, "")
  End If
End Function

To avoid `.getAttribute("href")` in for loop use `.querySelectorAll("a.question-hyperlink[href]")` — Santosh, Jul 25 '18 at 07:26
Not really once you change, the value can be accessed using `Debug.Print linkList(i)` — Santosh, Jul 25 '18 at 07:41

score 0 · Answer 2 · answered Jul 25 '18 at 06:21

This If element.className = "question-hyperlink" Then is useless because it is always true because you getElementsByClassName("question-hyperlink") so all elements are definitely of class question-hyperlink. The If statement can be removed.
You have each link in the variable element so you don't need the count. Instead of html.getElementsByClassName("question-hyperlink")(count).innerText use element.innerText.

So it should look like this:

Set elements = html.getElementsByClassName("question-hyperlink")
Dim erow As Long

For Each element In elements
    erow = Worksheets("Exec").Cells(Rows.count, 1).End(xlUp).Offset(1, 0).Row
    Worksheets("Exec").Cells(erow, 1) = element.innerText
    Worksheets("Exec").Cells(erow, 2) = element.GetAttribute("href") 'this should give you the URL
Next element

Get "href=link" from html page and navigate to that link using vba

2 Answers2

Method ①

Method ②

Linked