Unable to connect to webserver at specific time of day

Question

I am trying to read contents from a webpage using .Net code inside a WinForm. This was a working code up until recently wherein I started receiving the below connection failure error.

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

URL being attempted

Now, upon investigating further, I find that the error from the code driven call is coming only between certain times of the day; from 9 AM IST to approximately 12 PM IST. However, during this time, the URL call fired from any browser returns the correct response. After 12 PM IST, the code driven call also starts working.

This has been now tested in multiple Windows environment - my personal desktop, laptop, amazon cloud based VPS. In each of these cases, the ISPs were different.

This makes me think that there may be a time-bound protocol switch at the server side which is not accepting requests during the specific time.

Option Strict On
Imports System.IO
Imports System.Net
Public Class frmMain
    Private Async Sub btnStart_Click(sender As Object, e As EventArgs) Handles btnStart.Click
        Await Task.Run(AddressOf StartProcessing)
    End Sub

    Private Async Function StartProcessing() As Task
        Try
            Dim responseCookies As New CookieContainer
            Dim openPositionDataURL As String = "https://www.nseindia.com/live_market/dynaContent/live_watch/option_chain/optionKeys.jsp?symbolCode=-0&symbol=NIFTY&symbol=NIFTY&instrument=OPTIDX&date=-&segmentLink=17&segmentLink=17"
            ServicePointManager.Expect100Continue = False
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
            ServicePointManager.ServerCertificateValidationCallback = Function(s, Ca, CaC, sslPE)
                                                                          Return True
                                                                      End Function
            Dim request As HttpWebRequest = WebRequest.CreateHttp(openPositionDataURL)
            request.AllowAutoRedirect = True
            request.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate Or DecompressionMethods.None
            request.Host = "www.nseindia.com"
            request.KeepAlive = True
            request.Headers.Add("Upgrade-Insecure-Requests", "1")
            request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv:11.0) like Gecko"
            request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
            request.Headers.Add("Sec-Fetch-Site", "none")
            request.Headers.Add("Sec-Fetch-Mode", "navigate")
            request.Headers.Add("Accept-Encoding", "gzip, deflate, br")
            request.Headers.Add("Accept-Language", "en-US,en;q=0.9,hi;q=0.8,ko;q=0.7")
            request.CookieContainer = responseCookies

            ''request.ServicePoint.UseNagleAlgorithm = True
            'request.ProtocolVersion = HttpVersion.Version10
            'request.ServicePoint.ConnectionLimit = 12
            'request.Proxy = Nothing
            'request.ContentType = "application/x-www-form-urlencoded"
            'request.Referer = "https://www.google.com/"
            'request.Headers.Add("Sec-Fetch-User", "?1")
            'request.ConnectionGroupName = Guid.NewGuid().ToString()
            'request.Timeout = -1

            Using wr As WebResponse = Await request.GetResponseAsync()
                Using sr = New StreamReader(wr.GetResponseStream)
                    Console.WriteLine("URL Response")
                End Using
            End Using

        Catch ex As Exception
            Console.WriteLine(ex.Message)
            Throw ex
        End Try
    End Function
End Class

The commented out code is what I have tried in addition after searching for similar problems in SO.

Here's the Wireshark dump when hit from Browser, vs when hit from the code.

https://1drv.ms/u/s!AvZ7cbmSuh_chfw3l142uZGjQrBt2A?e=iYzdEy

https://1drv.ms/u/s!AvZ7cbmSuh_chfw5DVMjckPdfBcBKA?e=pdWQ8q

How can I get this working between the times mentioned above?

Start setting `Option Strict On` + You have an async method, but you're not awating anything, so change this line: `Using sr = New StreamReader(request.GetResponseAsync().Result.GetResponseStream)` it's quite wrong. + Use the security protocol that the server requires (TLS1.2). + `Accept-Encoding gzip (...)` requires [AutomaticDecompression](https://learn.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.automaticdecompression). + Add a `CookieContainer` + Use the IE11 `User-Agent` header (there's a reason for that). — Jimi, Dec 18 '19 at 07:35
Thats why `.Result is used to wait for response and then return response. The function `StartProcessing is called using Await. I dont see anything wrong here. As I said, the code works fine after a specific time of the day. All of what you have mentioned above would have made a difference if the code was not working at all. Link to result vs await in SO https://stackoverflow.com/questions/32239661/await-vs-task-result-in-an-async-method — Kallol, Dec 18 '19 at 07:41
Using `.Result` is a blocking call. You never block on an async method request. I cannot know about the `time of the day` issue and I've never seen one. I can tell you what's wrong with the code that I can see here. The *time* issue may be related to the cookies. + `AutomaticDecompression` is off by default. Having that `Accept-Encoding` header, means that you're receiving a compressed stream that you have to decompress manually: I don't see where that happens. etc. — Jimi, Dec 18 '19 at 07:46
Edited the question based on your suggestion. The problem still remains. — Kallol, Dec 18 '19 at 08:01
You haven't set `Option Strict On`. Remove `.ConfigureAwait(False)`: you don't use that in WinForms. The `Using` block is missing a piece. It doesn't look like this is real code. + You didn't set a `CookieContainer`, + you didn't change the `User-Agent` header -- I tested the site response: all is fine. I will re-test when the time-zone datetime agrees with mine. — Jimi, Dec 18 '19 at 08:17
Add `request.AllowAutoRedirect = true` and `request.ServicePoint.Expect100Continue = false`. Fix the code. Set `Option Strict On`. — Jimi, Dec 18 '19 at 08:24
Cookiecontainer is not required. I fired the URL directly from the browser after deleting cookies during the restriction time. It worked. The request was seen in the Network developer console of Chrome and it was seen to be sent without cookies. So cookies is not needed during the restricted time to make this work. Rest will edit the code here. The URL hit now (2:03 PM IST) from the browsder will work, as well as via the code. The code here is a representable version of the actual code. But it definitely works and has been tested. Will upload a video if required tomorrow during block time. — Kallol, Dec 18 '19 at 08:34
You don't need a cookie **storage**, you need a CookieContainer; two different things. Deleting the Browser cache has nothing to do with the exchange of cookies. The Browser will exchange cookies anyway. Your WebRequest needs to do the same. — Jimi, Dec 18 '19 at 08:38
When I said *all is fine*, I meant of course that I cannot see specific time-related settings on the server side, no specific timestamps in the result, nothing *special* about the cookies expire timestamp etc. I will take a look when the *time is right*. — Jimi, Dec 18 '19 at 08:48
Now it's better. Change the WebRequest creation in `Dim request As HttpWebRequest = WebRequest.CreateHttp(openPositionDataURL)` and just `ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12`. Add a `ServerCertificateValidationCallback` that just returns `True` for testing purposes (e.g., `ServicePointManager.ServerCertificateValidationCallback = Function(s, Ca, CaC, sslPE) Return True End Function`), since the Server **will** ask to validate the certificates. Let's wait... — Jimi, Dec 18 '19 at 08:57
Just to clarify, I suggested to change the `User-Agent` header because a more recent `UserAgent` may *cause* the Server to activate [HSTS - HTTP Strict Transport Security](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security) (a Server *may* activate HSTS, or not). That Server doesn't have it active (I found it disabled). But a load balancer may change this condition. When this happens, the Server expects a form of *interaction* with the WebBrowser and awaits a response. HttpWebRequest doesn't support it and will await the Server response until a time-out stops the boring party. — Jimi, Dec 18 '19 at 09:19

score 0 · Answer 1 · edited Mar 08 '20 at 11:08

0

As suggested by Jimi the following change into an IE based more recent UA did the trick.

Changed the UA I was using from

request.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"

to

request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv:11.0) like Gecko"

edited Mar 08 '20 at 11:08

halfer

19,824
17
99
186

answered Dec 19 '19 at 04:13

Kallol

264
1
12

(I have removed from here a supplementary question, as I suspect it was not seen by the party you were addressing. This sort of discussion is best done in the comments anyway, and possibly a new question if it is substantial enough. Thanks!). – halfer Mar 08 '20 at 11:09

Unable to connect to webserver at specific time of day

1 Answers1