0

Using .NET Core 3.0 I am trying to get the httpResponseMessage.StatusCode of:

https://www.element14.com/community/community/designcenter/azure-sphere-starter-kits?ICID=azure-sphereCH-topbanner

var sw = Stopwatch.StartNew();
var httpClient = new HttpClient();

// latest Chrome user-agent doesn't work
const string userAgent = @"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36";

// Curl User-Agent works after 10s
//const string userAgent = "curl/7.55.1";

httpClient.DefaultRequestHeaders.Add("User-Agent", userAgent);

// Trying to get keep-alive header working
//httpClient.DefaultRequestHeaders.Add("Connection", "close");
//httpClient.DefaultRequestHeaders.Add("Connection", "keep-alive");
httpClient.DefaultRequestHeaders.Add("Connection", "Keep-Alive");

var url = "https://www.element14.com/community/community/designcenter/azure-sphere-starter-kits?ICID=azure-sphereCH-topbanner";
try
{
    var httpResponseMessage = await httpClient.GetAsync(url);
    var sc = (int)httpResponseMessage.StatusCode;
}
catch (HttpRequestException ex)
{
    // always fails with Unable to read data from the transport connection:
    // A connection attempt failed because the connected party did not properly respond after a period of time,
    // or established connection failed because connected host has failed to respond.. 

    sw.Stop();
    // always failing around 19.3 or 19.5ms
}

Using

curl -I http://www.element14.com/community/community/designcenter/azure-sphere-starter-kits?ICID=azure-sphereCH-topbanner

enter image description here

We can see it is behind an F5 Big-IP load balancer which has been an issue with HttpClient:

https://github.com/dotnet/corefx/issues/7812#issuecomment-305848835

and from the F5 side:

https://devcentral.f5.com/s/question/0D51T00006i7fhD/bigip-randomly-closing-connection-from-net-application

Error Update

Full error text is: "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."

Error screenshot in VS2019 is: enter image description here

Fiddler Update

enter image description here

The request working from Chrome.

enter image description here

The request failing from HttpClient. Notice the Connection: keep-alive difference

https://stackoverflow.com/a/38648237/26086 - setting the headers I couldn't see any difference from Fiddler.

Curl Update

Passing the curl User-Agent works albeit slowly (10s).

Dave Mateer
  • 6,588
  • 15
  • 76
  • 125
  • What's the error/exception? You posted part of the error as comments, from what seems to be a timeout exception.`curl` doesn't use it's own network stack, it uses the same stuff HttpClient does. Is there a cooooookkkkkk --- screen scraper protection at F5's site that found out you didn't post the expected headers? Perhaps it didn't like the `Agent` header? – Panagiotis Kanavos Nov 27 '19 at 11:58
  • So, F5 obviously doesn't like screen scraping without permission. One option would be to *ask* for that permission and use whatever API/URL they have for this. Another option is to use Fiddler or any other debugging proxy to find out what headers, agent string do work and use the same with HttpClient – Panagiotis Kanavos Nov 27 '19 at 12:02
  • Thank you both. I'm building a broken link checker and want to know the status code of the page so interested in the content. Good idea on asking permission. This is just an interesting edge case. – Dave Mateer Nov 27 '19 at 13:20
  • Have updated code to refect user-agent passing. – Dave Mateer Nov 27 '19 at 13:22
  • The domain `www.element14.com` is resolved as an _AKAMAI_ address. The scraping protection, if any, is more likely to be there rather on the F5 (if it's running ASM, and it doesn't look so at first sight). Try your client on another computer which uses another IP or proxy. – Eugène Adell Nov 29 '19 at 11:48
  • Thanks Eugene - am getting consistent behaviour across multiple machines on different networks. Good thoughts on investigating domain resolution to AKAMAI! – Dave Mateer Nov 30 '19 at 06:40

0 Answers0