Can't parse name from a webpage using POST request

Question

I've written a macro in vba to get a "name" from a website using POST request. To reach the target page it is necessary to send POST request twice. Firstly, a page opens up like the first image underneath. After clicking on the "search by address" button on the starting page it leads to another page where two boxes to be filled in which is shown in the image 2 below. One for street number and the other for street name. After clicking on the search button when the form is done filling then it leads to the target page with the information i'm after. I tested it using msgbox in the script to be sure i'm on the right page. I'm surely on that page and i can see the title of that page which is "HARRIS COUNTY APPRAISAL DISTRICT". However, I can't parse anything from that page. I'm after this name "LARA PEDRO A & MARIA G" from that page.

This is the macro I'm trying with:

Sub httpPost()

    Dim http As New XMLHTTP60, html As New HTMLDocument
    Dim rec As HTMLHtmlElement
    Dim ArgStr As String, ArgStr_ano As String

    ArgStr = "search=addr"
    ArgStr_ano = "TaxYear=2017&stnum=15535&stname=CAMPDEN+HILL+RD"

    With http
        .Open "POST", "https://public.hcad.org/records/QuickSearch.asp", False
        .setRequestHeader "Content-type", "application/x-www-form-urlencoded"
        .setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
        .setRequestHeader "Referer", "https://public.hcad.org/records/quicksearch.asp"
        .send ArgStr
    End With

    With http
        .Open "POST", "https://public.hcad.org/records/QuickRecord.asp", False
        .setRequestHeader "Content-type", "application/x-www-form-urlencoded"
        .setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
        .setRequestHeader "Referer", "https://public.hcad.org/records/quicksearch.asp"
        .send ArgStr_ano
        html.body.innerHTML = .responseText
    End With

    MsgBox http.responseText
End Sub

Search to be made with:

Street No: 15535
Street Name: CAMPDEN HILL RD

These are the image of two pages following which target page can be reached:

"https://www.dropbox.com/s/e9on9zwqzmcboze/1Untitled.jpg?dl=0" "https://www.dropbox.com/s/0lchpde8uq63jps/pics.jpg?dl=0"

I somehow caught the url using chrome developer tool and using that url in my below code I get the result I'm after. However, what's wrong with my "POST" request? Why can't I get the same using the above method? For your consideration, here is another bit of code to get the result using the collected url from chrome dev tool what i got by sending post request twice as i did in my above code:

Sub Web_Data()
    Dim http As New XMLHTTP60, html As New HTMLDocument
    Dim post As Object

    With http
        .Open "GET", "https://public.hcad.org/records/details.asp?crypt=%94%9A%B0%94%BFg%85%8D%83%82og%8El%87tXvXQJXJzDTpHjEyr%D4%BE%C2%AF%AE%AA%9Fpk%88%5Do%5B%B8%96%A3%C0q%5E&bld=1&tab=", False
        .send
        html.body.innerHTML = .responseText
    End With

    For Each post In html.getElementsByClassName("data")(2).getElementsByTagName("th")
        i = i + 1: Cells(i, 1) = post.innerText
    Next post
End Sub

The result is:

LARA PEDRO A & MARIA G
15531 CAMPDEN HILL RD
HOUSTON TX 77053-3302

here is a similar question that retrieves data from the same website (it uses selenium though) https://stackoverflow.com/questions/45559835/how-can-i-switch-iframe-using-selenium-vba — jsotola, Aug 13 '17 at 00:14
Thanks jsotola for the link but I'm not thinking to do it using selenium. — SIM, Aug 13 '17 at 14:54

score 1 · Accepted Answer · answered Aug 13 '17 at 20:40

Finally solved it myself. Here is the working code:

Sub httpPost()

    Dim http As New WinHttp.WinHttpRequest, html As New HTMLDocument
    Dim post As Object
    Dim ArgStr As String, ArgStr_ano As String

    ArgStr = "search=addr"
    ArgStr_ano = "TaxYear=2017&stnum=15535&stname=CAMPDEN+HILL+RD"

    With http
        .Option(6) = True
        .Open "POST", "https://public.hcad.org/records/QuickSearch.asp", False
        .setRequestHeader "Content-type", "application/x-www-form-urlencoded"
        .setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
        .setRequestHeader "Referer", "https://public.hcad.org/records/quicksearch.asp"
        .send ArgStr
    End With

    With http
        .Option(6) = True
        .Open "POST", "https://public.hcad.org/records/QuickRecord.asp", False
        .setRequestHeader "Content-type", "application/x-www-form-urlencoded"
        .setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
        .setRequestHeader "Referer", "https://public.hcad.org/records/quicksearch.asp"
        .send ArgStr_ano
        html.body.innerHTML = .responseText
    End With

    For Each post In html.getElementsByClassName("data")(2).getElementsByTagName("th")
        i = i + 1: Cells(i, 1) = post.innerText
    Next post

End Sub

Can't parse name from a webpage using POST request

1 Answers1