Access web using Powershell and Proxy

Question

I can't seem to get access to a webpage using Powershell. I keep getting a "(407) Proxy Authentication Required". I've tried many things. I don't know what the proxy is or what kind of authentication it requires. The only thing I have access to is in IE it uses a script for configuring. I tried using some IPs from that, but no luck. Any ideas?

Here is one example of what I tried:

$wc = New-Object System.Net.WebClient
$wc.Headers.Add("User-Agent","Mozilla/4.0+")        
$wc.Proxy = [System.Net.WebRequest]::DefaultWebProxy
$wc.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$wc.DownloadString("http://stackoverflow.com")

Answer 1

I had a similar issue and resolved it with just two lines of powershell:

$browser = New-Object System.Net.WebClient
$browser.Proxy.Credentials =[System.Net.CredentialCache]::DefaultNetworkCredentials

Answer 2

I haven't seen anyone doing something like this but there is a way to do this as a "global setting" in your Powershell script (I remember doing this in C# before for local dev builds).

[System.Net.WebRequest]::DefaultWebProxy = [System.Net.WebRequest]::GetSystemWebProxy()
[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials

This way if you don't want to update all your WebClients with proxy details, you can just override the global setting (have to be done every time you run the script). But this assumes that the current logged in Windows user is valid for the system-defined proxy server.

NOTE: I would say that this is only useful as a quick and dirty way to get a PS script working that wasn't proxy aware before (like Cake build).

Answer 3

If the proxy answers "407", "proxy authentication required", then the authentication is required:

$Username="Hugo"
$Password="abcdefgh"
$WebProxy = New-Object System.Net.WebProxy("http://webproxy:8080",$true)
$url="http://aaa.bbb.ccc.ddd/rss.xml"

$WebClient = New-Object net.webclient

$WebClient.Proxy=$webproxy
$WebClient.proxy.Credentials = New-Object System.Net.NetworkCredential($Username, $Password)
$path="C:\Users\hugo\xml\test.xml"
$WebClient.DownloadFile($url, $path)

Content now resides in "test.xml"

Answer 4

I know the question is specifically about Powershell 2.0, but I would like to share information on setting up a proxy server in Powershell Core (6+) because it's extremely hard to find elsewhere.

I agree with Dandré, that the best solution is to configure the default proxy server. I just had an issue with Powershell Core (7.1). I was trying exactly what Dandré suggests, but it didn't have any effect. After several hours of research and investigation, I have found out that Powershell Core is probably not using System.Net.WebRequest to make web requests anymore, but rather System.Net.Nett.HttpClient.

When I have configured the HttpClient's default proxy server, all web connections made by Powershell (Invoke-WebRequest, PowerShellGet's Find-Module, Install-Module, etc.) finally started to work.

Configure Default Proxy Server for Powershell Core

If you need to make the configuration permanent, just add the commands below to your Powershell profile.

Configure a specific default proxy server

[System.Net.Http.HttpClient]::DefaultProxy = New-Object System.Net.WebProxy('http://your-proxy')

If you need to set a specific port, add it to the proxy server URI: http://proxy:1234.

Configure credentials for authenticating proxy

In case it's an authenticating proxy, you need to set up the credentials to be used for proxy authentication. Use the following command to set the credentials of your domain account under which you're currently logged in to Windows:

[System.Net.Http.HttpClient]::DefaultProxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials

If you need different credentials, you can use the Get-Credential cmdlet, but it's interactive so it's not an ideal solution to have in your Powershell profile. But I'm sure there're other ways.

[System.Net.Http.HttpClient]::DefaultProxy.Credentials = Get-Credential

Configure proxy server bypassing

If you just need to bypass the proxy server and use the direct connection, but Powershell is using the default system-wide proxy server, simply set the HttpClient's default proxy to null:

[System.Net.Http.HttpClient]::DefaultProxy = New-Object System.Net.WebProxy($null)

Add Proxy Configuration to Powershell Profile

To make the configuration permanent, simply add the commands you need to your Powershell profile. There're four of them (All users, all hosts; All users, current host; Current user, all hosts; Current user, current host), but I would use the "Current user, all hosts". To find out the location of a specific profile, use the $Profile variable with a profile name:

$Profile.CurrentUserAllHosts

It should print the path $Home\Documents\Powershell\profile.ps1.

If the profile doesn't exist yet, just create it and put the configuration there. It will be executed every time you execute a new Powershell Core (pwsh.exe) instance.

Configure Default Proxy using Environment Variable

An alternative solution is to use an environment variable. According to the documentation, HttpClient is actually using the following variables to initialize the default proxy if they're defined:

• HTTP_PROXY for HTTP requests
• HTTPS_PROXY for HTTPS requests
• ALL_PROXY for both HTTP and HTTPS
• NO_PROXY may contain a comma-separated list of hostnames excluded from proxying

An example usage:

ALL_PROXY='http://proxy:1234'

And if you need to pass credentials:

ALL_PROXY='http://username:password@proxy:1234'

This solution is supported by a wider range of applications, so if it's better or worse than the previous solution depends on what exactly you want to configure, just Powershell or also other applications.

Answer 5

If you use the following you'll receive a prompt to enter your credentials:

$client.Credentials = Get-Credential

Answer 6

I solved this with just three lines:

1. $proxy='http://username:password@IP:PORT'
2. $ENV:HTTP_PROXY=$proxy
3. $ENV:HTTPS_PROXY=$proxy

Answer 7

If you know the script - just download it, open with Notepad and find IP and port of your proxy server. As for authentication - most probably your windows credentials are used, so in theory you should be able to keep it empty, unless there's something suspicious in the script.

Answer 8

try adding cache credentials....

$domain = 'ChDom'
$Client = new-object System.Net.WebClient
$cc = New-object System.Net.CredentialCache
$urlObj = New-object System.Uri($url)

#these variables must be plaintext strings
$creds = New-object System.Net.NetworkCredential($Username,$Password)

#your auth might be different than mine
$cc.add($urlobj,"NTLM",$creds)
$client.Credentials = $cc
$Client.Downloadfile($url, C:\Temp\TestPage.html)

Answer 9

Option 1 - Setting Proxy Authentication via Code.

• Another Authenticated Proxy Example With Credentials

    # 1). Set your http proxy address and port
  $proxy = New-Object System.Net.WebProxy("http://p.weshare.net:80",$true)
  
   # 2). Set your http proxy credentials
  $proxy.Credentials = New-Object System.Net.NetworkCredential("username", "password")
  
   # 3). Set the global proxy
  [System.Net.WebRequest]::DefaultWebProxy = $proxy
  
   # 4). sets a value that indicates whether to bypass the proxy server for local addresses.
  [System.Net.WebRequest]::DefaultWebProxy.BypassProxyOnLocal = $true
  
   # 5). Call Invoke-WebRequest without -Proxy Tag ( as it pulls it from what you defined )
  $reposnseText = Invoke-WebRequest https://www.oracle.com
  

---

1. Just wrapping it as a Function

   Function Set-My-Proxy { param($proxy,$username,$password)

     $proxy = New-Object System.Net.WebProxy($proxy,$true)
     $proxy.Credentials = New-Object System.Net.NetworkCredential($username, $password)
     [System.Net.WebRequest]::DefaultWebProxy = $proxy
     [System.Net.WebRequest]::DefaultWebProxy.BypassProxyOnLocal = $true
   

   }

2. Call the Function

   Set-My-Proxy 'http://p.weshare.net:80' 'username' 'password'

---

Option 2 - Setting Proxy Authentication via Another Proxy

Credits & Limitations of solution

1. Install local proxy, e.g : squid for windows

2. After installing, open squid.conf file ( right mouse click on squid icon on task bar -> Open Squid Configuration

3. Put the following code please provide authenticated proxy ip only ( in our example : 10.1.2.3 - not domain one )

   http_access allow all
   http_port 3128
   
   coredump_dir /var/spool/squid3
   refresh_pattern ^ftp:       1440    20% 10080
   refresh_pattern ^gopher:    1440    0%  1440
   refresh_pattern -i (/cgi-bin/|\?) 0 0%  0
   refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
   refresh_pattern .       0   20% 4320
   
   cache_peer 10.1.2.3 parent 80 0 no-query default login=my_username:my_password
   
   never_direct allow all
   access_log none
   cache_log none
   

4. Restart Squid Service

5. You now may call your local proxy, which will forward request to authenticated proxy and response back

   Invoke-WebRequest -Uri 'https://stackoverflow.com/' -Proxy 'http://127.0.0.1:3128'