HTTPS NSE data request in VBA how to get all cookie values

Question

I'm trying to access https://www.nseindia.com/ to get the cookies in response headers. But using MSXML2.XMLHTTP returns the cookie value as empty string.

From, VBA Microsoft.XMLHTTP setRequestHeader not sending cookie - I tried using WinHTTP, which did not connect at all and kept timing out. Apparently that can only be used on HTTP requests. How can the same be done for a HTTPS request?

MSXML2.ServerXMLHTTP also does not seem to support HTTPS requests.

Please find the code snippets I used for the 2 methods below:

MSXML2.XMLHTTP

Sub Get_Web_Data()

Dim request As Object
Dim response As String
Dim website As String
   
' Website to go to.
website = "https://www.nseindia.com/"

' Create the object that will make the webpage request.
Set request = CreateObject("MSXML2.XMLHTTP")

' Where to go and how to go there.
request.Open "GET", website, False

' Set headers.
request.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
request.setRequestHeader "user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
request.setRequestHeader "accept-Encoding", "gzip , deflate"
request.setRequestHeader "accept-language", "en-US,en;q=0.9"
request.setRequestHeader "Accept", "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
request.setRequestHeader "Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7"

' Send the request for the webpage.
request.send

responseHeaders = request.getAllResponseHeaders
    MsgBox responseHeaders

End Sub

winHTTP

Sub Get_Web_Data_Cookie()
    
Dim website As String
Dim cookieString As String
Dim XMLHTTP As WinHttp.WinHttpRequest

' Website to go to.
website = "https://www.nseindia.com/"

'Initialize XMLHttp Object
 Set XMLHTTP = CreateObject("WinHttp.WinHttpRequest.5.1") ' needs Microsoft WinHTTP Services 5.1 reference
    
' XMLHTTP.Option(WinHttpRequestOption_EnableRedirects) = False ' WinHttpRequestOption_EnableRedirects=6
        
XMLHTTP.Open "GET", website, False
    
' Set headers.
   XMLHTTP.setRequestHeader "user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
XMLHTTP.setRequestHeader "accept-encoding", "gzip , deflate"
XMLHTTP.setRequestHeader "accept-language", "en-US,en;q=0.9"
  
' Send the request for the webpage.
XMLHTTP.send

responseHeaders = XMLHTTP.getAllResponseHeaders
MsgBox responseHeaders

   
End Sub

Maybe this site is trying to block web scraping... any other options you can use for `user-agent` ? — Robin Mackenzie, Mar 17 '21 at 09:46

Abraham · Accepted Answer · 2021-03-18T11:35:44.047

Okay, I have finally solved it using the same steps as mentioned in - How to get cookie information using excel vba

This was NOT a Certificate issue at all.

Here is my code.

Public Function NSEDataCall(website, setCookies) As String

Dim XMLHTTP As WinHttp.WinHttpRequest

'Initialize XMLHttp Object
'Use the best/proper XMLHttp object available on your system
Set XMLHTTP = CreateObject("WinHttp.WinHttpRequest.5.1") ' needs Microsoft WinHTTP Services 5.1 reference
    
' XMLHTTP.Option(WinHttpRequestOption_EnableRedirects) = False ' WinHttpRequestOption_EnableRedirects=6
        
XMLHTTP.Open "GET", website, False
    
' Set headers.
XMLHTTP.setRequestHeader "REFERER", website
XMLHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
XMLHTTP.setRequestHeader "Accept", "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
XMLHTTP.setRequestHeader "Accept-Language", "en-us,en;q=0.5"
XMLHTTP.setRequestHeader "Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7"

' Set cookie value - used for second call
If Len(setCookies) > 0 Then
   XMLHTTP.setRequestHeader "cookie", setCookies
Else

End If

XMLHTTP.send

If Len(setCookies) > 0 Then

    ' Get response headers
    response = XMLHTTP.getAllResponseHeaders
    ' Debug.Print response

    ' Split by new line
    responseArray = Split(response, vbCrLf)
    ' Debug.Print responseArray(7)

    ' Helps to identify dataType - output comes as code numbers
    ' MsgBox (VarType(Trim(Split(Split(responseArray(5), ";")(0), ":")(1)) & "; " & Trim(Split(Split(responseArray(6), ";")(0), ":")(1))

    ' Return the sv_bm cookie in response array from indices 7 (indices start from 0)
    NSEDataCall = setCookies & "; " & Trim(Split(Split(responseArray(7), ";")(0), ":")(1))
    

Else

    ' Get response headers
    response = XMLHTTP.getAllResponseHeaders
    ' Debug.Print response

    ' Split by new line
    responseArray = Split(response, vbCrLf)

    ' Helps to identify dataType - output comes as code numbers
    ' MsgBox (VarType(Trim(Split(Split(responseArray(5), ";")(0), ":")(1)) & "; " & Trim(Split(Split(responseArray(6), ";")(0), ":")(1))

    ' Return the cookies in response array from indices 5 to 9
    NSEDataCall = Trim(Split(Split(responseArray(5), ";")(0), ":")(1)) & "; " & Trim(Split(Split(responseArray(6), ";")(0), ":")(1)) & "; " & Trim(Split(Split(responseArray(7), ";")(0), ":")(1)) & "; " & Trim(Split(Split(responseArray(8), ";")(0), ":")(1)) & "; " & Trim(Split(Split(responseArray(9), ";")(0), ":")(1))
    'Debug.Print (responseArray(5) + responseArray(6))

End If

End Function


Sub GetNSECookies()

Dim website As String
Dim cookieValues As String
Dim website2 As String
Dim cookieValuesFinal As String

' First call
website = "https://www.nseindia.com/market-data/securities-lending-and-borrowing"
cookieValues = NSEDataCall(website, cookieValues)
' Debug.Print (cookieValues)

' Second call for sv_bm cookie
website = "https://www.nseindia.com/market-data/securities-lending-and-borrowing"
cookieValues = NSEDataCall(website, cookieValues)
Debug.Print (cookieValues)


End Sub

the output for this code snippet is as below.

nsit=p8XRMHoQSM5uEQUM7XIJdT8B; nseappid=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcGkubnNlIiwiYXVkIjoiYXBpLm5zZSIsImlhdCI6MTYxNjA0MDYwNCwiZXhwIjoxNjE2MDQ0MjA0fQ.BLTcAzB76DYfZI3rXUl2PkrpgUZp9w8r1UF-yXGo4Os; AKA_A2=A; ak_bmsc=520CE4F35658A3B15048CCCE60A4E7547D38DE9EEC3000009CD25260A2FB3E43~pl9bruXUs5nNKxK5S/bJRyN580uFS9ZtV8nwBP8Qm3XWDms7ASCi3ptfGjv0tdgor2Su0pgG13S0ZfheirbKwG8ckZjIrwFJekieseYkMEljA7MivwOo+izySca+cktj38v8dtutTScCfjVJGZOpaCwWBzr9v3JYpKVjdQCqnb4KB7v51O1ZmuqCtoK4b2sA0qQUytHgGGgXf8ZtFFv0VT7AjbZNKcjEdm4LTLidgmXX0=; bm_mi=C5AE845425DB55CAB9626B7A4DD0F7FD~D8U6FxMuj0HFHRdsgmxmpC2LOBFGEpHQgTAoFX8vOTeoNfTZ/KMz4NOdu52Ao8qbord9vfGA+KQ2/HN+8ILK5BSLxT//yRFIsPKmHCiA7bxUU14SqZO3gQo4BVW92dqJodsoqYpEXwXEjOtbDCZ1E2w9aERcEpCAQeLuQvzlAjiJwmIia0iAAuTc2eewlyBtb0oSaRCMTZXDcuvziSCCfzdo8N8lTtWaMICsVClYXQqaIeqGWFjh1xyKo+jj1cJxYULE3vXWBSYaRHnJLZX/+oyYWrkqvrwPn+/PDpN1iPY=; bm_sv=E2881456097AB72A45E379FB86952E6F~iCyF/Esh3Bj9L205JScGaCUgr9qyeITu91BXox9pKZCvOvYx6Rt5m2VTyNCSMmYIBqZkRD0R3n11Htgb0Ill5TX9TXgVk+kL++6CUS9o93j
LGwFnXVBbQ60xz0+iuscdi7T93Dlv7JL9Lw32gobE/R6NmwzDjaxT2WFmKf7nV/M=

score 0 · Answer 2 · answered Mar 17 '21 at 04:35

Adapting the solution found here:

This works for me with url of https://www.stackoverflow.com:

Sub test()

   Dim url As String
   url = "https://www.stackoverflow.com"

   Dim winHttpReq As Object
   Set winHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")

   'Ignoriere SSL-Errors
   winHttpReq.Option(4) = &H3300

   winHttpReq.Open "GET", url, False

   winHttpReq.Send ("")

   Debug.Print winHttpReq.getAllResponseHeaders

End Sub

Outputs:

Cache-Control: private
Connection: keep-alive
Date: Wed, 17 Mar 2021 04:32:59 GMT
Transfer-Encoding: chunked
Via: 1.1 varnish
Content-Type: text/html; charset=utf-8
Accept-Ranges: bytes
Server: Microsoft-IIS/10.0
Vary: Fastly-SSL
strict-transport-security: max-age=15552000
x-route-name: Home/Index
x-frame-options: SAMEORIGIN
x-flags: AA
x-aspnet-duration-ms: 3
x-request-guid: 4e8b7ffc-6609-4009-b59e-f1d72d598284
x-is-crawler: 0
x-providence-cookie: f34f05db-5db7-2f34-bb41-7c9c32fada8c
feature-policy: microphone 'none'; speaker 'none'
content-security-policy: upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com
x-page-view: 1
X-Served-By: cache-syd10151-SYD
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1615955579.318915,VS0,VE282
X-DNS-Prefetch-Control: off

But for https://www.nseindia.com it just times out. Perhaps you can update this method with the request headers which may prevent the timeout.

Hi Robin, thank you so much for your response. I tried the same, by adding the headers but it still times out. Please see answer below for my code snippet — Abraham, Mar 17 '21 at 09:23

HTTPS NSE data request in VBA how to get all cookie values

2 Answers2