I am trying to perform a simple web request for a https webpage using mono and F# under Linux.
I've managed to get a solution working yesterday by disabling the certificate validation done by ServerCertificateValidationCallback
, however now it suddenly doesn't work anymore.
The relevant parts of my script looks like this
#r "../../packages/FSharp.Data/lib/net40/FSharp.Data.dll"
#r "System.Xml.Linq.dll"
open FSharp.Data
open System
open System.IO
open System.Net
open System.Net.Security
open System.Threading
open System.Security.Cryptography.X509Certificates
// disabling certificate validation
ServicePointManager.ServerCertificateValidationCallback <-
new RemoteCertificateValidationCallback(fun _ _ _ _ -> true)
type Either<'a,'b> = Left of 'a
| Right of 'b
let web_reader (url: string) =
try
Thread.Sleep 500
let wr = WebRequest.CreateHttp url
wr.UserAgent <- "test.bot" //I've also tried "safari" without luck
let stream = wr.GetResponse().GetResponseStream()
Right( (new StreamReader(stream)).ReadToEnd() )
with exp -> Left (sprintf "ERROR IN 'web_reader' FOR URL %s\n\n%A" url exp)
web_reader "https://www.jobfinder.dk/"
and calling the web_reader
function applied to the https url results in following exception
System.Net.WebException: Error: SendFailure (Error writing headers) ---> System.Net.WebException: Error writing headers ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
at Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord (IAsyncResult asyncResult) <0x41d866e0 + 0x0010b> in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord (IAsyncResult ar, Boolean ignoreEmpty) <0x41d86620 + 0x0002b> in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker (IAsyncResult result) <0x41d838e0 + 0x00227> in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.WebConnection.EndWrite (System.Net.HttpWebRequest request, Boolean throwOnError, IAsyncResult result) <0x41d87f00 + 0x00207> in <filename unknown>:0
at System.Net.WebConnectionStream+<SetHeadersAsync>c__AnonStorey1.<>m__0 (IAsyncResult r) <0x41d87800 + 0x0013b> in <filename unknown>:0
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x41d74070 + 0x0019f> in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () <0x41d6ee90 + 0x00053> in <filename unknown>:0
at FSI_0001.web_reader (System.String url) <0x41d8a4a0 + 0x00073> in <filename unknown>:0
I really don't get what is wrong. What am I missing?
I am using:
- F# 4.0
- mono 4.4.1
- Debian Jessie
I have tried to be very careful not to perform too many requests to that url and I've even added some delay with Thread.Sleep
. I am still able to ping the server and access the given website, so I doubt that they blocked my IP or something like that. The exception is thrown at the very first attempt.
EDIT:
I just tested the code in Visual Studio with .Net, and there it seemed that the request stalled forever. So I debugged and investigated, and ended up finding a solution The request was aborted: Could not create SSL/TLS secure channel
adding following to my script:
ServicePointManager.Expect100Continue <- true;
ServicePointManager.SecurityProtocol <- SecurityProtocolType.Tls12
And then it works for .Net.
I also updated my version of mono from 4.2.1 to 4.4.1 which seems to have support for Tls12 as well, however I can still not get it working with mono.