Can't find any way to click on the dropdown menu to choose any preferable item

Question

I've written some code in vba using IE to choose England Premier League from a dropdown which is visible as Next To Play at this moment. My target is to click on that dropdown link captioned as Next To Play and then choose the Australia A-League from option. However, I can't find any way to click on that very link. If I run my below script the way it is, it neither works nor throws any error and finally quits the browser.

Link to that webpage: URL

Here is what I've tried so far:

Sub soccer()
    Dim IE As New InternetExplorer, html As HTMLDocument
    Dim post As Object

    With IE
        .Visible = True
        .navigate "replace_with_above_link"
        Do Until .readyState = READYSTATE_COMPLETE: Loop
        Set html = .document
    End With

    Application.Wait Now + TimeValue("00:00:05")
    For Each post In html.getElementsByTagName("option")
''        If InStr(post.innerText, "Next To Play") > 0 Then MsgBox post.innerText: Exit For  ''checked to see if I'm on the right place
        If InStr(post.innerText, "Next To Play") > 0 Then post.Click: Exit For
    Next post

   IE.Quit
End Sub

This is the partial picture of the webpage where I wanna trigger a click.

I would like to be able to click on that link and choose any option from there.

It seems I'm close to the solution. My below script can select the portion I wanna perform a click:

Sub Soccer_Dropdown()

    Dim IE As New InternetExplorer, html As HTMLDocument
    Dim post As Object

    With IE
        .Visible = True
        .navigate "https://ubet.com/sports/soccer/nexttoplay"
        Do Until .readyState = 4: DoEvents: Loop
        Set html = .document
    End With

    Application.Wait Now + TimeValue("00:00:05")
    Set post = html.querySelector(".ubet-fixed-price-navigator-selector-league select")
    post.Focus

'   IE.Quit
End Sub

Picture of the result:

However, when it comes to perform a click I can't do anymore.

Answer 1

Does GUI web-browser is absolutely required for webscraping ? Let me doubt...

If you try to debug the target page you will see that the data you are looking for is JSON type and is resulting from an XHR request.

So assuming that you got all required cookies to sent an XHR request to the target website, you may directly get the JSON data (from curl) instead of browsing the website ...

---

Apparently each league have an unique ID (50 for england, 76 for Australia, etc...)

So in order to get access to the Australia A-League data , you have to send a GET request :

https://ubet.com/api/sportsViewData/nexttoplay/false/3/76 Australia A-League

---

In this resulting JSON file, you will get an Offers key, which contain a dict which contain all offers and contain also all required data I expect that you are looking to scrap....

Please see an offer example :

LongDisplayName "Central Coast Mariners"
DisplayOrder    1
OfferOrder  1
HAD 1
PlaceDeduction  0
ShowOfferBasedDynamicOffer  false
OfferId 308151
OfferName   "Central Coast"
DisplayHAD  1
WinReturn   1.85
PlaceReturn 0
DynamicOffer    null
Status  "o"

Regards

Answer 2

Navigating to https://ubet.com/sports/soccer/england-premier-league gets you that web page.

I know that is a short answer. I have been looking at this for some time (even though the answer is brief) and it is an Angular.js application but I cannot yet figure out how it works. Somehow one needs to call vm.selectedKeyChange manually. I'm working on it.

Answer 3

This is do-able as a small extension of your existing code.

I avoid using querySelector and querySelectorAll because they are flakey (per your other bounty question).

I didn't use Focus and used Click instead as I'm guessing this executes whatever required script runs in the background.

You can then either manually select the correct index (using selectedIndex) or search the child elements for the <label> value you need:

Working code:

Option Explicit

Sub Soccer_Dropdown()

    Dim IE As New InternetExplorer, html As HTMLDocument
    Dim divElement As Object, selectElement As Object, optionElement As Object
    Dim optionIndex As Long
    Dim targetOption  As String

    ' load page
    With IE
        .Visible = True
        .navigate "https://ubet.com/sports/soccer/nexttoplay"
        Do Until .readyState = 4: DoEvents: Loop
        Set html = .document
    End With

    ' let page do its thing
    Application.Wait Now + TimeValue("00:00:05")

    ' get the div
    Set divElement = html.getElementsByClassName("ubet-fixed-price-navigator-selector-league")

    ' get the select in the div
    Set selectElement = divElement(0).getElementsByTagName("select")(0)

    ' click it to activate any script to populate select
    selectElement.Click

    ' select by index - this works but showing the other technique below
    'selectElement.selectedIndex = 9

    ' select by iterating the options
    optionIndex = 0
    targetOption = "Australia A-League"
    For Each optionElement In selectElement
        If LCase(optionElement.localName) = "option" And optionElement.Label = targetOption Then
            selectElement.selectedIndex = optionIndex
            Debug.Print targetOption & " at index " & optionIndex
            Exit For
        End If
        optionIndex = optionIndex + 1
    Next

    ' quit IE
    'IE.Quit

End Sub

Edit

@QHarr - per your question How does Set selectElement = divElement(0).getElementsByTagName("select")(0) access the "Australia A-League"?

The div containing the dropdown has class ubet-fixed-price-navigator-selector-league and that <div> has a single <select> element - which is the dropdown. Because getElementsByTagName returns a set of HTML elements, we are just saying to return the first <select> element within the <div>. The 0 index doesn't refer to the list of things in the dropdown (i.e. the <option>s). See the HTML:

A bit further down the HTML you can see the <option> elements (grouped in <optgroup>s) where you can access the dropdown values:

@Topto - good question - the localName property, along with a few others for the optionElement variable was one that I could see in the Locals window as having a value of option:

I guess you could use nodeName just as well as localName. There is tagName as well, which I suppose is the most obvious sounding:

HTH