Power Shell Web Scraping SSL/TSL Issue

Question

I am wanting to run a web scraping script on a server.

The current script collects the html on the specified page.

$url = "http://websms"
 [net.httpWebRequest] $request = [net.webRequest]::create($url)
 [net.httpWebResponse] $response = $request.getResponse()
 $responseStream = $response.getResponseStream()
 $sr = new-object IO.StreamReader($responseStream)
 $result = $sr.ReadToEnd()

 $result

This works fine on a typical web page. However I am wanting to run it on a servers admin page which of course requires a login.

I thought before I attempt to login I will try and scrape the login page of the server. Running the above script I get the following result.

   Exception calling "GetResponse" with "0" argument(s): "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel."
At C:\temp\web3.ps1:3 char:56
+  [net.httpWebResponse] $response = $request.getResponse <<<< ()
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : DotNetMethodException

Any idea of how to get around this issue or perhaps if you could point me a different direction so I can scrape elements from the admin html page of the sever.

Thanks Guys!

score 25 · Accepted Answer · answered Mar 29 '12 at 01:40

25

This one liner will ignore SSL certificate errors:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}

Errors regarding self-signed untrusted certificates, mismatching names or expiration will be ignored after this is executed.

answered Mar 29 '12 at 01:40

Andy Arismendi

50,577
16
107
124

7

And to restore, just execute `[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null` – mousio Dec 20 '12 at 14:17
1

This is answer no longer works, see reason and workaround [here](http://stackoverflow.com/a/15841856/1254222) – 79E09796 Nov 16 '15 at 16:02

score 0 · Answer 2 · edited May 23 '17 at 12:16

0

Use this brilliant answer:

public bool AcceptAllCertifications(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certification, System.Security.Cryptography.X509Certificates.X509Chain chain, System.Net.Security.SslPolicyErrors sslPolicyErrors)
{
    return true;
}
ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(AcceptAllCertifications);

edited May 23 '17 at 12:16

Community

1
1

answered Mar 29 '12 at 01:20

Mathias R. Jessen

157,619
12
148
206

Thanks Mathias, Can that be converted to a Power Shell Script format? – Samuel Meddows Mar 29 '12 at 01:33
Just compile it from source when running in PoSH, as here: http://poshcode.org/624 – Mathias R. Jessen Mar 29 '12 at 01:39

Power Shell Web Scraping SSL/TSL Issue

2 Answers2

Linked