Can MSXML2.XMLHTTP retrieve ALL of the HTML data for a given webpage?

Question

With dynamic web pages that display a table of retrieved data, I’ve found that both MSXML2.XMLHTTP and the Internet Explorer object usually can’t access this data. A good example is https://www.tiff.net/tiff/films.html. Both techniques won’t retrieve any of the movie data – just the surrounding web page. The code I’ve tried is as follows:

Function getHTTP(ByVal sReq As String) As Variant
    On Error GoTo onErr
    With CreateObject("MSXML2.XMLHTTP")
        .Open "GET", sReq, False
        .send
        getHTTP = StrConv(.responseBody, 64)
    End With
    Exit Function
    onErr:   MsgBox "Error " & Err & ": " & Err.Description, 49, "Error opening site"
End Function

Function GetHTML(ByVal strURL As String) As Variant
  Dim oIE As InternetExplorer
  Dim hElm As IHTMLElement
  Set oIE = New InternetExplorer
  oIE.Navigate strURL
  Do While (oIE.Busy Or oIE.ReadyState <> READYSTATE_COMPLETE)
     DoEvents
  Loop
  Set hElm = oIE.Document.all.tags("html").Item(0)
  GetHTML = hElm.outerHTML
  Set oIE = Nothing
  Set hElm = Nothing
End Function

But there is a way to manually retrieve the movie data – just follow these steps with Microsoft Edge or Internet Explorer:

Right-click on one of the movies 

Choose “inspect element." The DevTools console opens. 

At the bottom-left of the screen, click on the “html” tab. 

Right-click the tab.  Choose “copy.” 

Open notepad and paste what you’ve copied.

You now have the movie data and can save it to a file for parsing. My question: Is there any way to get this data programmatically?

Answer 1

Here are the film titles using IE (you can use same process to get directors)

Option Explicit
Public Sub GetFilms()
    Dim IE As New InternetExplorer, html As HTMLDocument, films As Object, i As Long
    With IE
        .Visible = True
        .navigate "https://www.tiff.net/tiff/films.html"

        While .Busy Or .readyState < 4: DoEvents: Wend
        Set films = .document.querySelectorAll("[target=_self]")

        For i = 0 To films.Length - 1
            Debug.Print films.item(i).innerText
        Next
        .Quit '<== Remember to quit application
    End With
End Sub

XHR is too fast for this, with the URL provided, but IE is just fine.

If you inspect the HTML you can see each film has the following commonality:

There is an attribute within the a tag called target whose value is _self.

You can use an attribute CSS selector to gather all of these matching elements using the querySelectorAll method of document.

---

CSS selector (sample):

---

I would be interested in if this can be solved for getting the film descriptions by parsing the HTML. I had thought the presence of the comments was obscuring the film descriptions. A regex which selects the text within these in theory "<!-- react-text: \d+ -->([^...].+?(?=<))" seems to fail when applied to the .innerHTML as did attempts to swop out the comment start and finish with regex.

Answer 2

Why Json? Because the page is loaded using json data

To View: Use Google Chrome --> Press F12 --> Load URL --> Goto Network tab

---

Code:

Sub getHTTP()

    Dim Url As String, data As String
    Dim xml As Object, JSON As Object, colObj, item


    Url = "https://www.tiff.net/data/films-events-2018.json?q=1513263947586"

    Set xml = CreateObject("MSXML2.ServerXMLHTTP")
    With xml
        .Open "GET", Url, False
        .send
        data = .responseText
    End With


    Set JSON = JsonConverter.ParseJson(data)
    Set colObj = JSON("items")

    For Each item In colObj
        Debug.Print item("title")
        Debug.Print item("description")

        For Each c1 In item("cast")
            Debug.Print c1
        Next

        For Each c2 In item("countries")
            Debug.Print c2
        Next
    Next
End Sub

---

Output

---

Installation of JsonConverter

1. Download the latest release
2. Import JsonConverter.bas into your project (Open VBA Editor, Alt + F11; File > Import File) Add Dictionary reference/class
3. For Windows-only, include a reference to "Microsoft Scripting Runtime"
4. For Windows and Mac, include VBA-Dictionary

---

Tree View of Data