0

I work on a C#/.NET application that has functionality to integrate with external web services. For the communications, we utilize a class that we built that inherits from the .NET System.Net.WebClient class.

While attempting to retrieve the WSDL from the Bloomberg site for testing, we seem to be getting a dropped connection with no other indications as to why. It appears as though it gets a successful HTTPS connection, but the remote site simply disconnects when an attempt is made to read the response. We are simply calling the System.Net.WebClient.DownloadString() method and passing in the WSDL URL: https://service.bloomberg.com/assets/dl/dlws.wsdl?WSDL. 

webClient = new HttpWebClient();
UriBuilder uriBuilder = new UriBuilder(this.ServiceUrl);
uriBuilder.Query = "WSDL";
webClient.Headers.Add("Content-Type", "text/xml; charset=utf-8");
this.RawWsdlData = webClient.DownloadString(uriBuilder.Uri);

We have been able to verify that the URL is correct by putting in the same URL into a browser, which results in us obtaining the requested WSDL.

I have turned on System.Net tracing and have been able to obtain an internal trace. Note that HttpWebClient is our class that is inherited from System.Net.WebClient.

    Log started for PID = 6564, TID = 26
    Getting WSDL from service (https://service.bloomberg.com/assets/dl/dlws.wsdl?WSDL)
    System.Net Verbose: 0 : [2072] Entering HttpWebClient#21999770::DownloadString(https://service.bloomberg.com/assets/dl/dlws.wsdl?WSDL#-917497223)
    System.Net Verbose: 0 : [2072] Entering HttpWebClient#21999770::DownloadData(https://service.bloomberg.com/assets/dl/dlws.wsdl?WSDL#-917497223)
    System.Net Verbose: 0 : [2072] Entering WebRequest::Create(https://service.bloomberg.com/assets/dl/dlws.wsdl?WSDL)
    System.Net Verbose: 0 : [2072] Entering HttpWebRequest#49164730::HttpWebRequest(https://service.bloomberg.com/assets/dl/dlws.wsdl?WSDL#-917497223)
    System.Net Information: 0 : [2072] Current OS installation type is 'Client'.
    System.Net Information: 0 : [2072] RAS supported: True
    System.Net Verbose: 0 : [2072] Exiting HttpWebRequest#49164730::HttpWebRequest() 
    System.Net Verbose: 0 : [2072] Exiting WebRequest::Create()        -> HttpWebRequest#49164730
    System.Net Verbose: 0 : [2072] Entering HttpWebRequest#49164730::GetResponse()
    System.Net Error: 0 : [2072] Can't retrieve proxy settings for Uri 'https://service.bloomberg.com/assets/dl/dlws.wsdl?WSDL'. Error code: 12180.
    System.Net Verbose: 0 : [2072] Entering ServicePoint#10615457::ServicePoint(service.bloomberg.com:443)
    System.Net Information: 0 : [2072] Associating HttpWebRequest#49164730 with ServicePoint#10615457
    System.Net Information: 0 : [2072] Associating Connection#16234150 with HttpWebRequest#49164730
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#24691947::Socket(AddressFamily#2)
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#24691947::Socket() 
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#7067698::Socket(AddressFamily#23)
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#7067698::Socket() 
    System.Net.Sockets Verbose: 0 : [2072] Entering DNS::TryInternalResolve(service.bloomberg.com)
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#24691947::Connect(69.191.242.179:443#-1275937026)
    System.Net.Sockets Information: 0 : [2072] Socket#24691947 - Created connection from 10.2.7.102:52684 to 69.191.242.179:443.
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#24691947::Connect() 
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#7067698::Close()
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#7067698::Dispose()
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#7067698::Close() 
    System.Net Information: 0 : [2072] Connection#16234150 - Created connection from 10.2.7.102:52684 to 69.191.242.179:443.
    System.Net Information: 0 : [2072] TlsStream#32181663::.ctor(host=service.bloomberg.com, #certs=0, checkCertificateRevocationList=False, sslProtocols=Default)
    System.Net Information: 0 : [2072] Associating HttpWebRequest#49164730 with ConnectStream#18140357
    System.Net Information: 0 : [2072] HttpWebRequest#49164730 - Request: GET /assets/dl/dlws.wsdl?WSDL HTTP/1.1

    System.Net Information: 0 : [2072] ConnectStream#18140357 - Sending headers
    {
    Content-Type: text/xml; charset=utf-8
    Host: service.bloomberg.com
    Connection: Keep-Alive
    }.
    System.Net Information: 0 : [2072] SecureChannel#31141938::.ctor(hostname=service.bloomberg.com, #clientCertificates=0, encryptionPolicy=RequireEncryption)
    System.Net Information: 0 : [2072] Enumerating security packages:
    System.Net Information: 0 : [2072]     Negotiate
    System.Net Information: 0 : [2072]     NegoExtender
    System.Net Information: 0 : [2072]     Kerberos
    System.Net Information: 0 : [2072]     NTLM
    System.Net Information: 0 : [2072]     TSSSP
    System.Net Information: 0 : [2072]     pku2u
    System.Net Information: 0 : [2072]     WDigest
    System.Net Information: 0 : [2072]     Schannel
    System.Net Information: 0 : [2072]     Microsoft Unified Security Protocol Provider
    System.Net Information: 0 : [2072]     Default TLS SSP
    System.Net Information: 0 : [2072]     CREDSSP
    System.Net Information: 0 : [2072] SecureChannel#31141938 - Left with 0 client certificates to choose from.
    System.Net Information: 0 : [2072] SecureChannel#31141938::.AcquireClientCredentials, new SecureCredential() (flags=(ValidateManual, NoDefaultCred, SendAuxRecord), m_ProtocolFlags=(Ssl3Client, Tls10Client), m_EncryptionPolicy=RequireEncryption)
    System.Net Information: 0 : [2072] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
    System.Net Information: 0 : [2072] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = service.bloomberg.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [2072] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=125, returned code=ContinueNeeded).
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#24691947::Send()
    System.Net.Sockets Verbose: 0 : [2072] Data from Socket#24691947::Send
    System.Net.Sockets Verbose: 0 : [2072] 00000000 : 16 03 01 00 78 01 00 00-74 03 01 5C C2 03 9A 43 : ....x...t..\...C
    System.Net.Sockets Verbose: 0 : [2072] 00000010 : 54 67 9D 98 2C 4E 6B B9-A4 4C D5 8D 08 BC 03 36 : Tg..,Nk..L.....6
    System.Net.Sockets Verbose: 0 : [2072] 00000020 : 03 5F E6 D2 DA 82 2E AC-C4 C8 15 00 00 0E C0 0A : ._..............
    System.Net.Sockets Verbose: 0 : [2072] 00000030 : C0 09 C0 14 C0 13 00 35-00 2F 00 0A 01 00 00 3D : .......5./.....=
    System.Net.Sockets Verbose: 0 : [2072] 00000040 : 00 00 00 1A 00 18 00 00-15 73 65 72 76 69 63 65 : .........service
    System.Net.Sockets Verbose: 0 : [2072] 00000050 : 2E 62 6C 6F 6F 6D 62 65-72 67 2E 63 6F 6D 00 0A : .bloomberg.com..
    System.Net.Sockets Verbose: 0 : [2072] 00000060 : 00 08 00 06 00 1D 00 17-00 18 00 0B 00 02 01 00 : ................
    System.Net.Sockets Verbose: 0 : [2072] 00000070 : 00 23 00 00 00 17 00 00-FF 01 00 01 00          : .#...........
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#24691947::Send()   -> Int32#125
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#24691947::Receive()
    System.Net.Sockets Error: 0 : [2072] Socket#24691947::UpdateStatusAfterSocketError() - ConnectionReset
    System.Net.Sockets Error: 0 : [2072] Exception in Socket#24691947::Receive - An existing connection was forcibly closed by the remote host.
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#24691947::Receive()       -> Int32#0
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#24691947::Dispose()
    System.Net Error: 0 : [2072] Exception in HttpWebRequest#49164730:: - The underlying connection was closed: An unexpected error occurred on a send..
    System.Net Information: 0 : [2072] Associating HttpWebRequest#49164730 with ServicePoint#10615457
    System.Net Information: 0 : [2072] Associating Connection#39028125 with HttpWebRequest#49164730
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#26207244::Socket(AddressFamily#2)
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#26207244::Socket() 
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#38017319::Socket(AddressFamily#23)
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#38017319::Socket() 
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#26207244::Connect(69.191.242.179:443#-1275937026)
    System.Net.Sockets Information: 0 : [2072] Socket#26207244 - Created connection from 10.2.7.102:52685 to 69.191.242.179:443.
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#26207244::Connect() 
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#38017319::Close()
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#38017319::Dispose()
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#38017319::Close() 
    System.Net Information: 0 : [2072] Connection#39028125 - Created connection from 10.2.7.102:52685 to 69.191.242.179:443.
    System.Net Information: 0 : [2072] TlsStream#50247169::.ctor(host=service.bloomberg.com, #certs=0, checkCertificateRevocationList=False, sslProtocols=Default)
    System.Net Information: 0 : [2072] Associating HttpWebRequest#49164730 with ConnectStream#41123459
    System.Net Information: 0 : [2072] HttpWebRequest#49164730 - Request: GET /assets/dl/dlws.wsdl?WSDL HTTP/1.1

    System.Net Information: 0 : [2072] ConnectStream#41123459 - Sending headers
    {
    Content-Type: text/xml; charset=utf-8
    Host: service.bloomberg.com
    Connection: Keep-Alive
    }.
    System.Net Information: 0 : [2072] SecureChannel#53272024::.ctor(hostname=service.bloomberg.com, #clientCertificates=0, encryptionPolicy=RequireEncryption)
    System.Net Information: 0 : [2072] SecureChannel#53272024 - Left with 0 client certificates to choose from.
    System.Net Information: 0 : [2072] Using the cached credential handle.
    System.Net Information: 0 : [2072] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = service.bloomberg.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [2072] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=125, returned code=ContinueNeeded).
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#26207244::Send()
    System.Net.Sockets Verbose: 0 : [2072] Data from Socket#26207244::Send
    System.Net.Sockets Verbose: 0 : [2072] 00000000 : 16 03 01 00 78 01 00 00-74 03 01 5C C2 03 9A D8 : ....x...t..\....
    System.Net.Sockets Verbose: 0 : [2072] 00000010 : 38 E3 3A 39 65 29 89 69-19 11 28 40 22 A3 87 9F : 8.:9e).i..(@"...
    System.Net.Sockets Verbose: 0 : [2072] 00000020 : 0C 7A 2D C6 9F 0A 67 D9-F1 78 CA 00 00 0E C0 0A : .z-...g..x......
    System.Net.Sockets Verbose: 0 : [2072] 00000030 : C0 09 C0 14 C0 13 00 35-00 2F 00 0A 01 00 00 3D : .......5./.....=
    System.Net.Sockets Verbose: 0 : [2072] 00000040 : 00 00 00 1A 00 18 00 00-15 73 65 72 76 69 63 65 : .........service
    System.Net.Sockets Verbose: 0 : [2072] 00000050 : 2E 62 6C 6F 6F 6D 62 65-72 67 2E 63 6F 6D 00 0A : .bloomberg.com..
    System.Net.Sockets Verbose: 0 : [2072] 00000060 : 00 08 00 06 00 1D 00 17-00 18 00 0B 00 02 01 00 : ................
    System.Net.Sockets Verbose: 0 : [2072] 00000070 : 00 23 00 00 00 17 00 00-FF 01 00 01 00          : .#...........
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#26207244::Send()   -> Int32#125
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#26207244::Receive()
    System.Net.Sockets Error: 0 : [2072] Socket#26207244::UpdateStatusAfterSocketError() - ConnectionReset
    System.Net.Sockets Error: 0 : [2072] Exception in Socket#26207244::Receive - An existing connection was forcibly closed by the remote host.
    System.Net.Sockets Verbose: 0 : [2072] Exiting Socket#26207244::Receive()       -> Int32#0
    System.Net.Sockets Verbose: 0 : [2072] Entering Socket#26207244::Dispose()
    System.Net Error: 0 : [2072] Exception in HttpWebRequest#49164730:: - The underlying connection was closed: An unexpected error occurred on a send..
    System.Net Error: 0 : [2072] Exception in HttpWebRequest#49164730::GetResponse - The underlying connection was closed: An unexpected error occurred on a send..
    System.Net Verbose: 0 : [2072] Entering HttpWebRequest#49164730::Abort()
    System.Net Error: 0 : [2072] Exception in HttpWebRequest#49164730:: - The request was aborted: The request was canceled..
    System.Net Verbose: 0 : [2072] Exiting HttpWebRequest#49164730::Abort()

Being an HTTPS connection, I would have expected an SSL/TLS type error, but this just shows a remote disconnect and nothing else.

Anyone have any ideas on what the cause could be or what can be done to debug this further? I can reproduce this at will, so it is not a random issue.

ke4ktz
  • 1,152
  • 7
  • 18
  • 1
    Exception says following : Left with 0 client certificates to choose from. Code is trying to use a certificate which either is expired or not loaded. I had similar issue on another posting the server returned error 404 while TCP transport layer closed while attempting to go secure. In Net library SSL and HTTP are not well connected and errors in SSL do not get reported to HTTP. – jdweng Apr 26 '19 at 15:56
  • Exception also shows : "Receive - An existing connection was forcibly closed by the remote host". In Net Library this just means the connection did not complete. – jdweng Apr 26 '19 at 15:57
  • 1
    The SSL handshake fails. You need this `ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;` and this: `ServicePointManager.ServerCertificateValidationCallback = (s, cert, ch, sec) => { return true; };`. The final download it's an XML file of `89,155` bytes. – Jimi Apr 26 '19 at 15:59
  • You can test the results with this code: [WebClient hangs until timeout](https://stackoverflow.com/a/53873685/7444103). – Jimi Apr 26 '19 at 16:07
  • @Jimi, Thanks, that did the trick. Now, it's on to making this option user-controllable from the UI. – ke4ktz Apr 26 '19 at 17:53
  • What *option* are you referring to? – Jimi Apr 26 '19 at 17:54
  • @Jimi Forcing TLS 1.2. Although it's becoming more standard, we still have users in other parts of the world that aren't there. We need to make sure we don't force TLS 1.2 on those that don't have it available. – ke4ktz Apr 26 '19 at 17:57
  • `ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;`. The SSL handshake will pick the best one. You may want to limit these options or reject the connection if you don't like the protocol that has been agreed upon. See here: [Which TLS version was negotiated?](https://stackoverflow.com/a/48675492/7444103) how to do it. BTW, TLS 1.2 is not *becoming*, it's the standard by now. Also TLS 1.3 is out and .Net-available soon. – Jimi Apr 26 '19 at 18:02
  • @Jimi I work with a lot of large organizations and they are just now implementing policies that force the usage of TLS 1.2. Most that I deal with are only at the TLS 1.1 stage and are migrating to TLS 1.2. (And this is in the USA!!) In other parts of the world, we're lucky if they have even gotten out of using SSL3!! :( – ke4ktz Apr 26 '19 at 18:17
  • I was referring to Web (public interfaces). Other, private/limited access use whatever the actual security requirements/budget are. Enable all protocols, see what comes back and decide what to do. Or enable one or more protocols on-demand. Those are just flags, you can enable whatever you want. – Jimi Apr 26 '19 at 18:26
  • 1
    1. NEVER just return true from `ServerCertificateValidationCallback`. You've just turned off security. 2. Hardcoding TLS versions is not ideal, try using SchUseStrongCrypto first https://learn.microsoft.com/en-us/dotnet/framework/network-programming/tls#schusestrongcrypto – Steve Apr 26 '19 at 20:22
  • @Steve I doesn't apply here. I agree that, as a general rule, for generic connections, you have to validate the server certificate. But this relates to known sources. Also, adding a certificate check is very simple, it requires a few line of code that one can add whenever needed: the above is just an example, required to close the *current* problem. `SchUseStrongCrypto` depends on required system updates and, while it's a choice using HttpClient, it doesn't apply to WebRequest: if you don't explicitly enable Tls1.2, it won't be used. + the OP clearly states that they may need SSL3. – Jimi Apr 26 '19 at 20:49
  • What do you mean 'known sources'? If you remove the ability to validate the source, it can't be known anymore. The only way you can guarantee it's a known source is by checking the server certificate is a certificate you know. Otherwise anyone can drop in with their own cert that you've happily accepted. – Steve Apr 26 '19 at 20:58
  • @Steve a `Known source` is an already known (you have a local certificate in storage) EndPoint (verifiable response), from which you download known data in a known format. And, as already mentioned, this is an example. I linked this: [Which TLS version was negotiated?](https://stackoverflow.com/a/48675492/7444103) for a reason. Go there and inspect the code (about the server certificate validation, specifically). Another example, but much more detailed. So I provided an actual reference of how the code should look like in the end. – Jimi Apr 26 '19 at 21:48
  • Right, but the recommendation was to just `return true` in the first place. People will see code and copy it. They will not see links and follow them. Being a bit pedantic here, but this is important. – Steve Apr 29 '19 at 16:34
  • @Steve I only needed to set the SecurityProtocol value...no need to add the callback definition. – ke4ktz Apr 29 '19 at 18:00
  • @Steve Not much of a problem either (also pretty common), since you're not sending credentials nor uploading data. You're downloading a file that you then have to validate. Anyway, these are comments about a specific question made by a specific person. If lazy guys don't read further and get the first thing they see and use it in any random context, there isn't much you can do about it. A more detailed answer is linked. – Jimi Apr 29 '19 at 22:50

0 Answers0