0

Help gratefully received on this one. I have some VBA running in Excel that inspects a series of webpages displaying betting odds for football matches and puts the odds into my spreadsheet. It has been working perfectly for months and has stopped working in the last few weeks. Here is a simplified version of the code I'm using:

Sub TestImport()

Dim http As New MSXML2.XMLHTTP60
Dim html As New MSHTML.HTMLDocument
Dim htmlEle1 As MSHTML.IHTMLElement
Dim columncounter As Integer
Dim rowccounter As Integer
Dim targetwebpage As String
Dim ColumnHeader As Variant

On Error GoTo ErrorStop

trowNum = 1

    targetwebpage = "https://www.oddschecker.com/football/english/premier-league"
        With http
            .Open "get", targetwebpage, False
            .send
        End With

Set table_data = html.getElementsByTagName("tr")

If table_data.Length = 0 Then GoTo SkipLeague

For Each trow In table_data

    For Each tcell In trow.Children

      If tcell.innerText <> "TIP" Then 'Ignore this
        tcellNum = tcellNum + 1
        Cells(trowNum, tcellNum) = tcell.innerText
      End If
    
    Next tcell
    
    Cells(trowNum, 1) = Worksheets("Leagues").Cells(j, 1)
    trowNum = trowNum + 1
    tcellNum = 1
    
Next trow

SkipLeague:

ErrorStop:

End Sub

No data gets returned because [table_data] is always null. It's always null because there are no tags in my variable, [html]. Instead, [html] seems to be simply this:

"<HEAD></HEAD>
<BODY>
<P>&nbsp;</P></BODY>"

Why would [html] return this value when the actual webpage (https://www.oddschecker.com/football/english/premier-league) is much more complex when displayed in my browser? And why has this problem only started in the last few weeks?

I'd be grateful for any help on this.

braX
  • 11,506
  • 5
  • 20
  • 33

1 Answers1

1

I did a quick test and had no issue. Some page, like Google require the User-Agent to be sent, but not the oddschecker page.

Sub TestURL()
    Debug.Print GetResult("https://www.oddschecker.com/football/english/premier-league")
End Sub 

Function GetResult(url As String) As String
    Dim XMLHTTP As Object, ret As String
    Set XMLHTTP = CreateObject("MSXML2.ServerXMLHTTP")
    XMLHTTP.Open "GET", url, False
    XMLHTTP.setRequestHeader "Content-Type", "text/xml"
    XMLHTTP.setRequestHeader "Cache-Control", "no-cache"
    XMLHTTP.setRequestHeader "Pragma", "no-cache"
    XMLHTTP.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0"
    XMLHTTP.send
    ret = XMLHTTP.responseText
    GetResult = ret
End Function
Keith Swerling
  • 136
  • 1
  • 6
  • Sorry, I'm a noob at this. How did you 'acquire' those `setRequestHeader`s and why did you choose `MSXML2.ServerXMLHTTP` against `MSXML2.XMLHTTP` or against `WinHttp.WinHttpRequest.5.1`? Why does it work on my computer (`Windows 10 64bit, Office 2019 64bit`) only with `MSXML2.XMLHTTP` (otherwise it returns `error code 1020`)? Links to web pages to study this would also be helpful. – VBasic2008 Feb 10 '22 at 21:06
  • The TestURL() doesn't work for me. Like Keith, I'm getting a 1020 error. Changing from MSXML2.ServerXMLHTTP to MSXML2.XMLHTTP allows the code to run but seems to suggest I've been blocked. The title on the result is "Access denied | www.oddschecker.com used Cloudflare to restrict access". I've tried another PC, clearing cookies and using a VPN so I can't see how this can be blocked. Any thoughts? – carolinetoynbee Feb 11 '22 at 16:25
  • When I change to MSXML2.XMLHTTP, I get the same Access Denied error as you describe. This worked for me too: Msxml2.ServerXMLHTTP.6.0 – Keith Swerling Feb 11 '22 at 19:43
  • I do see this post about MSXML2.ServerXMLHTTP : https://stackoverflow.com/a/44500548/11666888 – Keith Swerling Feb 11 '22 at 19:54
  • Still no joy on this. I've tried: Set XMLHTTP = CreateObject("MSXML2.XMLHTTP") and get 'Access Denied' on the webpage that is returned. Set XMLHTTP = CreateObject("MSXML2.ServerXMLHTTP") and get a 1020 Error Set XMLHTTP = CreateObject("MSXML2.ServerXMLHTTP6.0") and get a 'Run-time error 429. ActiveX component can't create object' Set XMLHTTP = CreateObject("MSXML2.XMLHTTP6.0") and get a 'Run-time error 429. ActiveX component can't create object'. – carolinetoynbee Feb 14 '22 at 15:35