I am very new to VBA and trying to scrape through a website. So far I have been able to get cookies from initial get request and use them in POST for a successful login. The next step is to capture the session and user cookies and use them in the subsequent requests.
Unfortunately, this is where my problem begins.
Post successful login I am using .getAllResponseHeaders() to capture all headers but it seems the two cookies (Set-Cookie: xf_user AND Set-Cookie: xf_session) are missing and hence I am not able to capture them for later use. For comparison and easier understanding, I am posting the fiddler (correct) response and response captured by vba (incorrect).
I am not sure what am I doing wrong. Please suggest any options, I am happy to take an alternate approach. I am sure I am very close to success, just need your expert advice.
Fiddler Response
HTTP/1.1 303 See Other
Date: Thu, 30 Apr 2020 04:55:14 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Last-Modified: Thu, 30 Apr 2020 04:55:24 GMT
Location: https://f95zone.to/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: private, no-cache, max-age=0
Set-Cookie: xf_user=19872%2CUsOoxkBS4bzvLttbYhWkicE-JFQ-vBWo2L68LEVS; expires=Fri, 30-Apr-2021 04:55:24 GMT; Max-Age=31536000; path=/; secure; HttpOnly
Set-Cookie: xf_session=nlJRIrZOrbAiQGVAo_wRJhDSKBsy7wKz; path=/; secure; HttpOnly
Strict-Transport-Security: max-age=15768000
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 58beab553a76fea5-MEL
alt-svc: h3-27=":443"; ma=86400, h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400
cf-request-id: 026b0969420000fea583bd8200000001
Content-Length: 0
VBA Response
date: Thu, 30 Apr 2020 13:47:02 GMT
content-type: text/html; charset=utf-8
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
last-modified: Thu, 30 Apr 2020 13:47:01 GMT
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: private, no-cache, max-age=0
strict-transport-security: max-age=15768000
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri=""https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct""
server: cloudflare
cf-ray: 58c1b6504d3dfe8d-MEL
alt-svc: h3-27="":443""; ma=86400, h3-25="":443""; ma=86400, h3-24="":443""; ma=86400, h3-23="":443""; ma=86400
cf-request-id: 026cf0462f0000fe8d47804200000001
Snippet From My Code
Set objXMLHTTPSearch = CreateObject("MSXML2.XMLHTTP")
objXMLHTTPSearch.Open "POST", "https://f95zone.to/login/login", False
objXMLHTTPSearch.setRequestHeader "Accept", "text/html, application/xhtml+xml, image/jxr, */*"
objXMLHTTPSearch.setRequestHeader "Accept -Language", "en -US"
objXMLHTTPSearch.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
objXMLHTTPSearch.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
objXMLHTTPSearch.setRequestHeader "Accept -Encoding", "gzip , deflate"
objXMLHTTPSearch.setRequestHeader "Host", "f95zone.to"
objXMLHTTPSearch.setRequestHeader "Content-Length", Len(dataSTR)
objXMLHTTPSearch.setRequestHeader "Connection", "Keep -Alive"
objXMLHTTPSearch.setRequestHeader "cache -Control", "no-cache"
objXMLHTTPSearch.withCredentials = True
objXMLHTTPSearch.send dataSTR
statusSearch = objXMLHTTPSearch.status
fetchHeader = objXMLHTTPSearch.getAllResponseHeaders()
