Download URL content using PowerShell

Question

I am working in a script, where I am able to browse the web content or the 'url' but I am not able to copy the web content in it & download as a file. This is what I have made so far:

$url = "http://sp-fin/sites/arindam-sites/_layouts/xlviewer.aspx?listguid={05DA1D91-F934-4419-8AEF-B297DB81A31D}&itemid=4&DefaultItemOpen=1"
$ie=new-object -com internetexplorer.application
$ie.visible=$true
$ie.navigate($url)
while($ie.busy) {start-sleep 1} 

How can I copy the content of $url and save it to local drive as a file?

Update:

I got these errors:

Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (401) Unauthorized." At :line:6 char:47 + (New-Object system.net.webclient).DownloadFile( <<<< "$url/download-url-content", 'save.html' )

Missing ')' in method call. At :line:6 char:68 + (New-Object system.net.webclient).DownloadFile( "$url", 'save.html' <<<<

Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (401) Unauthorized." At :line:6 char:47 + (New-Object system.net.webclient).DownloadFile( <<<< "$url", 'save.html' )

Ok, let me explain more, on what I am trying to do: I have a excel file in our share point site & this is the file I am trying to download locally(any format), which is a part of the script, so that for the later part of the script, I can compare this file with other data & get an output.

Now if I can somehow map "my documents" from the site & able to download the file, that will also work for me.

Answer 1

Update Jan 2014: With Powershell v3, released with Windows 8, you can do this:

 (Invoke-webrequest -URI "http://www.kernel.org").Content

Original Post, valid for Powershell Version 2

This solution is very similar to the other answers from stej, Jay Bazusi and Marco Shaw. It is a bit more general, by installing a new module into your module directory, psurl. The module psurl adds new commands in case you have to do a lot of html-fetching (and POSTing) with powershell.

(new-object Net.WebClient).DownloadString("http://psget.net/GetPsGet.ps1") | iex

See the homepage of the code-sharing website http://psget.net/.

This nice line of PowerShell script will dowload GetPsGet.ps1 and send it to Invoke-Expression to install PsGet Module.

Then install PsUrl, a Powershell Module inspired by curl:

To install something (in our case PsUrl) from central directory just type:

install-module PsUrl

get-module -name psurl

Output:

ModuleType Name                      ExportedCommands
---------- ----                      ----------------
Script     psurl                     {Get-Url, Send-WebContent, Write-Url, Get-WebContent}

Command:

get-command -module psurl

Output:

CommandType     Name                                                Definition
-----------     ----                                                ----------
Function        Get-Url                                             ...
Function        Get-WebContent                                      ...
Alias           gwc                                                 Get-WebContent
Function        Send-WebContent                                     ...
Alias           swc                                                 Send-WebContent
Function        Write-Url                                           ...

You need to do this only once.

Note that this error might occur:

Q: Error "File xxx cannot be loaded because the execution of scripts is disabled on this system. Please see "get-help about_signing" for more details."

A: By default, PowerShell restricts execution of all scripts. This is all about security. To "fix" this run PowerShell as Administrator and call

Set-ExecutionPolicy RemoteSigned

From now on, in your new powershell sessions/scripts, do this:

import-module psurl
get-url "http://www.google.com"

To download and save to a file, do this:

get-url "http://www.google.com" | out-file -filepath "myfile.html"

Answer 2

As I understand it, you try to use IE because if automatically sends your credentials (or maybe you didn't know of any other option).

Why the above answers don't work is because you try to download file from SharePoint and you send an unauthenticated request. The response is 401.

This works:

PS>$wc=new-object system.net.webclient
PS>$wc.UseDefaultCredentials = $true
PS>$wc.downloadfile("your_url","your_file")

if the the current user of Posh has rights to download the file (is the same as the logged one in IE).

If not, try this:

PS>$wc=new-object system.net.webclient
PS>$wc.Credentials = Get-Credential
PS>$wc.downloadfile("your_url","your_file")

Answer 3

If you just want to download web content, use

(New-Object System.Net.WebClient).DownloadFile( 'download url content', 'save.html' )

Answer 4

I'm not aware of any way to save using that interface.

Does this render the page properly:

PS>$wc=new-object system.net.webclient
PS>$wc.downloadfile("your_url","your_file")

Answer 5

As already answered in https://stackoverflow.com/a/35202299/4636579, but with a mandatory Proxy and the credentials. Without proxy, it would be:

$url="http://aaa.bbb.ccc.ddd/rss.xml"

$WebClient = New-Object net.webclient

$path="C:\Users\hugo\xml\test.xml"

$WebClient.DownloadFile($url, $path)

Answer 6

$web = New-Object Net.WebClient

$web | Get-Member

$content=$web.DownloadString("http://www.bing.com")

Answer 7

If you're truly only concerned with the raw string content, the best route, as mentioned by a few others, is using the constructs within .NET to do this. However, I think in the previous answers a few opportunities are missed.

• It's often best to use WebRequest over WebClient as it provides better control over the entire request cycle
• Response buffering via System.IO.StreamReader, made possible by using WebRequest
• Creating a testable, reusable tool. Which is the very nature and purpose of PowerShell

function Get-UrlContent {
    <#
    .SYNOPSIS
        High performance url fetch

    .DESCRIPTION
        Given a url, will return raw content as string.

        Uses: 
        System.Net.HttpRequest
        System.IO.Stream
        System.IO.StreamReader

    .PARAMETER Url
        Defines the url to download

    .OUTPUTS
        System.String

    .EXAMPLE
        PS C:\> Get-UrlContent "https://www.google.com"
        "<!doctype html>..."
    #>

    [cmdletbinding()]
    [OutputType([String])]
    param(
        [Parameter(Mandatory, ValueFromPipeline)]
        [ValidateNotNullOrEmpty()]
        [string] $Url)

    Write-Debug "`n----- [Get-UrlContent]`n$url`n------`n`n"

    $req = [System.Net.WebRequest]::CreateHttp($url)

    try {
        $resp = $req.GetResponse()    
    }
    catch {
        Write-Debug "`n------ [Get-UrlContent]`nDownload failed: $url`n------`n"
    }
    finally {
        if ($resp) {
            $st = $resp.GetResponseStream()
            $rd = [System.IO.StreamReader]$st

            $rd.ReadToEnd()     
        }

        if ($rd) { $rd.Close() }
        if ($st) { $st.Close() }
        if ($resp) { $resp.Close() }   
    }
}