WebClient.DownloadString with HTTPS "The authentication or decryption has failed"

Question

I am creating a game using the Unity Engine(2017.2.0f3) on Windows 10 in which I am trying to get the HTML from a webpage. To my knowledge, Unity uses the "Mono" C# compiler and runtime v2.0 (according to the results of running mono --version in Unity\Editor\Data\Mono\bin), but also v5.0.1 (according to the results of running mono --version in Unity\Editor\Data\MonoBleedingEdge\bin). I am currently using the HTML Agility Pack for parsing. When attempting to access an HTTP secured website, such as dictionary.com, everything works as expected, but no matter what I do, I cannot access an HTTPS secured website, such as urbandictionary.com(Edit: Apparently, the problem is specific to this site, because after importing certificates using mozroots, github.com can be accessed), and always receive the exception: TlsException: The authentication or decryption has failed. I am aware that the HTML Agility Pack does not support HTTPS, and have wrote my code according to this answer. I've tried the solution described here and here, but to no avail. I've tried running mozroots, which I located in PathTo/Unity/Editor/Data/MonoBleedingEdge/lib/mono/4.5/mozroots, with the --import, --sync, and --quiet flags, but the same exception is thrown (if this did work, I'm skeptical about its portability, but perhaps I could implement my own version of mozroots, as suggested by the comments). Also, I've been informed that the version of Mono that Unity uses does not support TLS 1.2, which is problematic. If there is no solution to this problem, are there any workarounds?

Note: abridged for clarity

class MyWebClient : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
        HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
        request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
        return request;
    }
}

class GetHtml
{
    public static void Get(string url)
    {
        string documentString = new MyWebClient().DownloadString(url);
        ...
    }
}

class CallingClass
{
     public void CallingMethod()
     {
         ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => true;
         //With or without the above line ^, the same TlsException is thrown. 
         GetHtml.Get("https://www.urbandictionary.com/define.php?term=example");
         //HTTP works properly, but HTTPS, as in this example, does not.
     }
}

The log is as follows:

TlsException: The authentication or decryption has failed.
Mono.Security.Protocol.Tls.RecordProtocol.ProcessAlert (AlertLevel alertLevel, AlertDescription alertDesc)
Mono.Security.Protocol.Tls.RecordProtocol.InternalReceiveRecordCallback (IAsyncResult asyncResult)
Rethrow as IOException: The authentication or decryption has failed.
Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (IAsyncResult asyncResult)
Rethrow as WebException: Error getting response stream (Write: The authentication or decryption has failed.): SendFailure
System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult)
System.Net.HttpWebRequest.GetResponse ()
System.Net.WebClient.GetWebResponse (System.Net.WebRequest request)
System.Net.WebClient.DownloadDataCore (System.Uri address, System.Object userToken)

Edit: I've tried updating the runtime version. The only option was to change from .Net 3.5 to 4.6 in the Project Settings->Player menu, which is experimental. I still get an exception when the code is run, but the output log is a little different.

TlsException: The authentication or decryption has failed.
Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord (System.IAsyncResult asyncResult) (at <eb1224ae7b184cd09343d47f8a05481b>:0)
Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord (System.IAsyncResult ar, System.Boolean ignoreEmpty) (at <eb1224ae7b184cd09343d47f8a05481b>:0)
Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker (System.IAsyncResult result) (at <eb1224ae7b184cd09343d47f8a05481b>:0)
Rethrow as IOException: The authentication or decryption has failed.
Mono.Security.Protocol.Tls.SslClientStream.EndNegotiateHandshake (System.IAsyncResult result) (at <eb1224ae7b184cd09343d47f8a05481b>:0)
Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (System.IAsyncResult asyncResult) (at <eb1224ae7b184cd09343d47f8a05481b>:0)
Rethrow as IOException: The authentication or decryption has failed.
Mono.Security.Protocol.Tls.SslStreamBase.EndRead (System.IAsyncResult asyncResult) (at <eb1224ae7b184cd09343d47f8a05481b>:0)
Mono.Net.Security.Private.LegacySslStream.EndAuthenticateAsClient (System.IAsyncResult asyncResult) (at <344dc4d3f1ad41809df78607b6121a41>:0)
Mono.Net.Security.Private.LegacySslStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) (at <344dc4d3f1ad41809df78607b6121a41>:0)
Mono.Net.Security.MonoTlsStream.CreateStream (System.Byte[] buffer) (at <344dc4d3f1ad41809df78607b6121a41>:0)
System.Net.WebConnection.CreateStream (System.Net.HttpWebRequest request) (at <344dc4d3f1ad41809df78607b6121a41>:0)
Rethrow as WebException: Error: SecureChannelFailure (The authentication or decryption has failed.)
System.Net.WebClient.DownloadDataInternal (System.Uri address, System.Net.WebRequest& request) (at <344dc4d3f1ad41809df78607b6121a41>:0)
System.Net.WebClient.DownloadString (System.Uri address) (at <344dc4d3f1ad41809df78607b6121a41>:0)
System.Net.WebClient.DownloadString (System.String address) (at <344dc4d3f1ad41809df78607b6121a41>:0)

Answer 1

This problem count be caused by the HTTP protocol. If the server uses secure HTTP (actually HTTPS), the error may be created when the underlying protocol fails to handle the X509 Certificate.

Try to change this line:

HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;

to:

HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest; 
request.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => true;

Or, alternatively, apply the validation callback to the global filter using:

ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => true;

before any HttpWebRequest object is created within your code.

If you want to handle the certificate validation more carefully, implement the callback as showin here: https://msdn.microsoft.com/en-us/library/office/dd633677

This should fix your problem.

Answer 2

After some rigorous research it turns out that the version of Mono that Unity(2017.2.0f3) currently uses does not support TLS 1.2, as explained by a developer here. Also, a careful examination of the exception log shows that internally, the legacy backend is being used: Mono.Net.Security.Private.LegacySslStream as opposed to Mono.Net.Security.Private.SslStream. As of right now, the only solution would involve third-party libraries, or workarounds.

Thanks to @Evk for pointing me in the right direction.