Upload multiple files from Powershell script

Question

I have a webapplication that can process POSTing of a html form like this:

<form action="x" method="post" enctype="multipart/form-data">
  <input name="xfa" type="file">
  <input name="pdf" type="file">
  <input type="submit" value="Submit">
</form>

Note that there are two type="file" <input> elements.

How can I script POSTing this from a Powershell script? I plan to do that to create a simple test-framework for the service.

I found WebClient.UploadFile(), but that can only handle a single file.

Thank you for taking your time.

Answer 1

I've been crafting multipart HTTP POST with PowerShell today. I hope the code below is helpful to you.

• PowerShell itself cannot do multipart form uploads.
• There are not many sample about it either. I built the code based on this and this.
• Sure, Invoke-RestMethod requires PowerShell 3.0 but the code in the latter of the above links shows how to do HTTP POST with .NET directly, allowing you to have this running in Windows XP as well.

Good luck! Please tell if you got it to work.

function Send-Results {
    param (
        [parameter(Mandatory=$True,Position=1)] [ValidateScript({ Test-Path -PathType Leaf $_ })] [String] $ResultFilePath,
        [parameter(Mandatory=$True,Position=2)] [System.URI] $ResultURL
    )
    $fileBin = [IO.File]::ReadAllBytes($ResultFilePath)
    $computer= $env:COMPUTERNAME

    # Convert byte-array to string (without changing anything)
    #
    $enc = [System.Text.Encoding]::GetEncoding("iso-8859-1")
    $fileEnc = $enc.GetString($fileBin)

    <#
    # PowerShell does not (yet) have built-in support for making 'multipart' (i.e. binary file upload compatible)
    # form uploads. So we have to craft one...
    #
    # This is doing similar to: 
    # $ curl -i -F "file=@file.any" -F "computer=MYPC" http://url
    #
    # Boundary is anything that is guaranteed not to exist in the sent data (i.e. string long enough)
    #    
    # Note: The protocol is very precise about getting the number of line feeds correct (both CRLF or LF work).
    #>
    $boundary = [System.Guid]::NewGuid().ToString()    # 

    $LF = "`n"
    $bodyLines = (
        "--$boundary",
        "Content-Disposition: form-data; name=`"file`"$LF",   # filename= is optional
        $fileEnc,
        "--$boundary",
        "Content-Disposition: form-data; name=`"computer`"$LF",
        $computer,
        "--$boundary--$LF"
        ) -join $LF

    try {
        # Returns the response gotten from the server (we pass it on).
        #
        Invoke-RestMethod -Uri $URL -Method Post -ContentType "multipart/form-data; boundary=`"$boundary`"" -TimeoutSec 20 -Body $bodyLines
    }
    catch [System.Net.WebException] {
        Write-Error( "FAILED to reach '$URL': $_" )
        throw $_
    }
}

Answer 2

I was bothered by this thing and haven't found a satisfactory solution. Although the gist here proposed can do the yob, it is not efficient in case of large files transmittal. I wrote a blog post proposing a solution for it, basing my cmdlet on HttpClient class present in .NET 4.5. If that is not a problem for you, you can check my solution at the following address http://blog.majcica.com/2016/01/13/powershell-tips-and-tricks-multipartform-data-requests/

EDIT:

function Invoke-MultipartFormDataUpload
{
    [CmdletBinding()]
    PARAM
    (
        [string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$InFile,
        [string]$ContentType,
        [Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$Uri,
        [System.Management.Automation.PSCredential]$Credential
    )
    BEGIN
    {
        if (-not (Test-Path $InFile))
        {
            $errorMessage = ("File {0} missing or unable to read." -f $InFile)
            $exception =  New-Object System.Exception $errorMessage
            $errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'MultipartFormDataUpload', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $InFile
            $PSCmdlet.ThrowTerminatingError($errorRecord)
        }

        if (-not $ContentType)
        {
            Add-Type -AssemblyName System.Web

            $mimeType = [System.Web.MimeMapping]::GetMimeMapping($InFile)

            if ($mimeType)
            {
                $ContentType = $mimeType
            }
            else
            {
                $ContentType = "application/octet-stream"
            }
        }
    }
    PROCESS
    {
        Add-Type -AssemblyName System.Net.Http

        $httpClientHandler = New-Object System.Net.Http.HttpClientHandler

        if ($Credential)
        {
            $networkCredential = New-Object System.Net.NetworkCredential @($Credential.UserName, $Credential.Password)
            $httpClientHandler.Credentials = $networkCredential
        }

        $httpClient = New-Object System.Net.Http.Httpclient $httpClientHandler

        $packageFileStream = New-Object System.IO.FileStream @($InFile, [System.IO.FileMode]::Open)

        $contentDispositionHeaderValue = New-Object System.Net.Http.Headers.ContentDispositionHeaderValue "form-data"
        $contentDispositionHeaderValue.Name = "fileData"
        $contentDispositionHeaderValue.FileName = (Split-Path $InFile -leaf)

        $streamContent = New-Object System.Net.Http.StreamContent $packageFileStream
        $streamContent.Headers.ContentDisposition = $contentDispositionHeaderValue
        $streamContent.Headers.ContentType = New-Object System.Net.Http.Headers.MediaTypeHeaderValue $ContentType

        $content = New-Object System.Net.Http.MultipartFormDataContent
        $content.Add($streamContent)

        try
        {
            $response = $httpClient.PostAsync($Uri, $content).Result

            if (!$response.IsSuccessStatusCode)
            {
                $responseBody = $response.Content.ReadAsStringAsync().Result
                $errorMessage = "Status code {0}. Reason {1}. Server reported the following message: {2}." -f $response.StatusCode, $response.ReasonPhrase, $responseBody

                throw [System.Net.Http.HttpRequestException] $errorMessage
            }

            $responseBody = [xml]$response.Content.ReadAsStringAsync().Result

            return $responseBody
        }
        catch [Exception]
        {
            $PSCmdlet.ThrowTerminatingError($_)
        }
        finally
        {
            if($null -ne $httpClient)
            {
                $httpClient.Dispose()
            }

            if($null -ne $response)
            {
                $response.Dispose()
            }
        }
    }
    END { }
}

Cheers

Answer 3

I have found a solution to my problem after studying how multipart/form-data is built. A lot of help came in the form of http://www.paraesthesia.com/archive/2009/12/16/posting-multipartform-data-using-.net-webrequest.aspx.

The solution then is to build the body of the request up manually according to that convention. I have left of niceties like correct Content-Lengths etc.

Here is an excerpt of what I am using now:

    $path = "/Some/path/to/data/"

    $boundary_id = Get-Date -Format yyyyMMddhhmmssfffffff
    $boundary = "------------------------------" + $boundary_id

    $url = "http://..."
    [System.Net.HttpWebRequest] $req = [System.Net.WebRequest]::create($url)
    $req.Method = "POST"
    $req.ContentType = "multipart/form-data; boundary=$boundary"
    $ContentLength = 0
    $req.TimeOut = 50000

    $reqst = $req.getRequestStream()

    <#
    Any time you write a file to the request stream (for upload), you'll write:
        Two dashes.
        Your boundary.
        One CRLF (\r\n).
        A content-disposition header that tells the name of the form field corresponding to the file and the name of the file. That looks like:
        Content-Disposition: form-data; name="yourformfieldname"; filename="somefile.jpg" 
        One CRLF.
        A content-type header that says what the MIME type of the file is. That looks like:
        Content-Type: image/jpg
        Two CRLFs.
        The entire contents of the file, byte for byte. It's OK to include binary content here. Don't base-64 encode it or anything, just stream it on in.
        One CRLF.
    #>

    <# Upload #1: XFA #> 
    $xfabuffer = [System.IO.File]::ReadAllBytes("$path\P7-T.xml")

    <# part-header #>
    $header = "--$boundary`r`nContent-Disposition: form-data; name=`"xfa`"; filename=`"xfa`"`r`nContent-Type: text/xml`r`n`r`n"
    $buffer = [Text.Encoding]::ascii.getbytes($header)        
    $reqst.write($buffer, 0, $buffer.length)
    $ContentLength = $ContentLength + $buffer.length

    <# part-data #>
    $reqst.write($xfabuffer, 0, $xfabuffer.length)
    $ContentLength = $ContentLength + $xfabuffer.length

    <# part-separator "One CRLF" #>
    $terminal = "`r`n"
    $buffer = [Text.Encoding]::ascii.getbytes($terminal)        
    $reqst.write($buffer, 0, $buffer.length)
    $ContentLength = $ContentLength + $buffer.length

    <# Upload #1: PDF template #>
    $pdfbuffer = [System.IO.File]::ReadAllBytes("$path\P7-T.pdf")

    <# part-header #>
    $header = "--$boundary`r`nContent-Disposition: form-data; name=`"pdf`"; filename=`"pdf`"`r`nContent-Type: application/pdf`r`n`r`n"
    $buffer = [Text.Encoding]::ascii.getbytes($header)        
    $reqst.write($buffer, 0, $buffer.length)
    $ContentLength = $ContentLength + $buffer.length

    <# part-data #>
    $reqst.write($pdfbuffer, 0, $pdfbuffer.length)
    $ContentLength = $ContentLength + $pdfbuffer.length

    <# part-separator "One CRLF" #>
    $terminal = "`r`n"
    $buffer = [Text.Encoding]::ascii.getbytes($terminal)        
    $reqst.write($buffer, 0, $buffer.length)
    $ContentLength = $ContentLength + $buffer.length

    <#
    At the end of your request, after writing all of your fields and files to the request, you'll write:

    Two dashes.
    Your boundary.
    Two more dashes.
    #>
    $terminal = "--$boundary--"
    $buffer = [Text.Encoding]::ascii.getbytes($terminal)        
    $reqst.write($buffer, 0, $buffer.length)
    $ContentLength = $ContentLength + $buffer.length

    $reqst.flush()
    $reqst.close()

    # Dump request to console
    #$req

    [net.httpWebResponse] $res = $req.getResponse()

    # Dump result to console
    #$res

    # Dump result-body to filesystem
<#    
    $resst = $res.getResponseStream()
    $sr = New-Object IO.StreamReader($resst)
    $result = $sr.ReadToEnd()
    $res.close()
#>

    $null = New-Item -ItemType Directory -Force -Path "$path\result"
    $target = "$path\result\P7-T.pdf"

    # Create a stream to write to the file system.
    $targetfile = [System.IO.File]::Create($target)

    # Create the buffer for copying data.
    $buffer = New-Object Byte[] 1024

    # Get a reference to the response stream (System.IO.Stream).
    $resst = $res.GetResponseStream()

    # In an iteration...
    Do {
        # ...attemt to read one kilobyte of data from the web response stream.
        $read = $resst.Read($buffer, 0, $buffer.Length)

        # Write the just-read bytes to the target file.
        $targetfile.Write($buffer, 0, $read)

        # Iterate while there's still data on the web response stream.
    } While ($read -gt 0)

    # Close the stream.
    $resst.Close()
    $resst.Dispose()

    # Flush and close the writer.
    $targetfile.Flush()
    $targetfile.Close()
    $targetfile.Dispose()

Answer 4

I've remixed @akauppi's answer into a more generic solution, a cmdlet that:

• Can take pipeline input from Get-ChildItem for files to upload
• Takes an URL as a positional parameter
• Takes a dictionary as a positional parameter, which it sends as additional form data
• Takes an (optional) -Credential parameter
• Takes an (optional) -FilesKey parameter to specify the formdata key for the files upload part
• Supports -WhatIf
• Has -Verbose logging
• Exits with an error if something goes wrong

It can be called like this:

$url ="http://localhost:12345/home/upload"
$form = @{ description = "Test 123." }
$pwd = ConvertTo-SecureString "s3cr3t" -AsPlainText -Force
$creds = New-Object System.Management.Automation.PSCredential ("john", $pwd)

Get-ChildItem *.txt | Send-MultiPartFormToApi $url $form $creds -Verbose -WhatIf

Here's the code to the full cmdlet:

function Send-MultiPartFormToApi {
    # Attribution: [@akauppi's post](https://stackoverflow.com/a/25083745/419956)
    # Remixed in: [@jeroen's post](https://stackoverflow.com/a/41343705/419956)
    [CmdletBinding(SupportsShouldProcess = $true)] 
    param (
        [Parameter(Position = 0)]
        [string]
        $Uri,

        [Parameter(Position = 1)]
        [HashTable]
        $FormEntries,

        [Parameter(Position = 2, Mandatory = $false)]
        [System.Management.Automation.Credential()]
        [System.Management.Automation.PSCredential]
        $Credential,

        [Parameter(
            ParameterSetName = "FilePath",
            Mandatory = $true,
            ValueFromPipeline = $true,
            ValueFromPipelineByPropertyName = $true
        )]
        [Alias("Path")]
        [string[]]
        $FilePath,

        [Parameter()]
        [string]
        $FilesKey = "files"
    );

    begin {
        $LF = "`n"
        $boundary = [System.Guid]::NewGuid().ToString()

        Write-Verbose "Setting up body with boundary $boundary"

        $bodyArray = @()

        foreach ($key in $FormEntries.Keys) {
            $bodyArray += "--$boundary"
            $bodyArray += "Content-Disposition: form-data; name=`"$key`""
            $bodyArray += ""
            $bodyArray += $FormEntries.Item($key)
        }

        Write-Verbose "------ Composed multipart form (excl files) -----"
        Write-Verbose ""
        foreach($x in $bodyArray) { Write-Verbose "> $x"; }
        Write-Verbose ""
        Write-Verbose "------ ------------------------------------ -----"

        $i = 0
    }

    process {
        $fileName = (Split-Path -Path $FilePath -Leaf)

        Write-Verbose "Processing $fileName"

        $fileBytes = [IO.File]::ReadAllBytes($FilePath)
        $fileDataAsString = ([System.Text.Encoding]::GetEncoding("iso-8859-1")).GetString($fileBytes)

        $bodyArray += "--$boundary"
        $bodyArray += "Content-Disposition: form-data; name=`"$FilesKey[$i]`"; filename=`"$fileName`""
        $bodyArray += "Content-Type: application/x-msdownload"
        $bodyArray += ""
        $bodyArray += $fileDataAsString

        $i += 1
    }

    end {
        Write-Verbose "Finalizing and invoking rest method after adding $i file(s)."

        if ($i -eq 0) { throw "No files were provided from pipeline." }

        $bodyArray += "--$boundary--"

        $bodyLines = $bodyArray -join $LF

        # $bodyLines | Out-File data.txt # Uncomment for extra debugging...

        try {
            if (!$WhatIfPreference) {
                Invoke-RestMethod `
                    -Uri $Uri `
                    -Method Post `
                    -ContentType "multipart/form-data; boundary=`"$boundary`"" `
                    -Credential $Credential `
                    -Body $bodyLines
            } else {
                Write-Host "WHAT IF: Would've posted to $Uri body of length " + $bodyLines.Length
            }
        } catch [Exception] {
            throw $_ # Terminate CmdLet on this situation.
        }

        Write-Verbose "Finished!"
    }
}

Answer 5

The -Form option

With Power Shell 7.2+ there was a neat feature added for multipart/form data. From MS Learn Example 6:

$Uri = 'https://api.contoso.com/v2/profile'
$Form = @{
firstName  = 'John'
lastName   = 'Doe'
email      = 'john.doe@contoso.com'
avatar     = Get-Item -Path 'c:\Pictures\jdoe.png'
birthday   = '1980-10-15'
hobbies    = 'Hiking','Fishing','Jogging'
}
$Result = Invoke-WebRequest -Uri $Uri -Method Post -Form $Form

But this feature comes with some downsides. For me the lack of an option to control the MIME-TYPE of the file made it useless. I have to send a PDF as application/pdf, but PS will send it as octet-stream.

flexible script using .NET

Inspired by MS Learn Example 5 and answers found in various questions, I crafted my own Power Shell Script using .NET classes. It results in a quite flexible and powerful script, I think:

#
# Assembles a POST WebRequest
# 

Add-Type -AssemblyName System.Net.Http, System.Collections

$url = "http://localhost:14783"

$Uri = "$url/public/system_under_test.php"

$cookiedomain = "localhost"

# collect the streams here to dispose them at the end
$OpenStreams = [System.Collections.Generic.List[System.IO.FileStream]]::new()

# returns a .NET ByteArrayContent, useful for text, numbers and others
function Get-Text-Part {
    param ( $FieldName, $FieldValue )

    $Header = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new('form-data')
    $Header.Name = $FieldName
    
    $FieldValueAsBytes =[System.Text.Encoding]::UTF8.GetBytes($FieldValue)
    $FieldContent = [System.Net.Http.ByteArrayContent]::new($FieldValueAsBytes)
    $FieldContent.Headers.ContentDisposition = $Header
    
    return $FieldContent
}

# returns a .NET StreamContent object , useful for arbitrary files
function Get-File-Part {
    param ( $FieldName, $FilePath, $MimeType )

    $Header = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new('form-data')
    $Header.Name = $FieldName
    $Header.FileName = Split-Path -leaf $FilePath

    $FileStream = [System.IO.FileStream]::new($FilePath, [System.IO.FileMode]::Open)
    $FileContent = [System.Net.Http.StreamContent]::new($FileStream)

    $OpenStreams.Add($FileStream)

    $FileContent.Headers.ContentDisposition = $Header
    $FileContent.Headers.ContentType = [System.Net.Http.Headers.MediaTypeHeaderValue]::Parse($MimeType)

    return $FileContent
}

$MultipartContent = [System.Net.Http.MultipartFormDataContent]::new()
$MultipartContent.Add((Get-Text-Part -FieldName 'captcha' -FieldValue 'so what'))
$MultipartContent.Add((Get-Text-Part -FieldName 'company' -FieldValue 'ACME'))
$MultipartContent.Add((Get-Text-Part -FieldName 'name' -FieldValue 'Jon Doe'))
$MultipartContent.Add((Get-Text-Part -FieldName 'phone' -FieldValue '123'))

$FullFilePath = Join-Path -Path (Get-Location) -ChildPath 'HelloWorld.pdf'
$MultipartContent.Add((Get-File-Part -FieldName 'pdf1' -FilePath $FullFilePath -MimeType 'application/pdf'))

$FullFilePath = Join-Path -Path (Get-Location) -ChildPath 'myImage.jpg'
$MultipartContent.Add((Get-File-Part -FieldName 'image1' -FilePath $FullFilePath -MimeType 'image/jpeg'))

$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
$session.Cookies.Add((New-Object System.Net.Cookie("session name", "m5k4ek685786586997097746av", "/", "$cookiedomain")))

# you probably do not need -WebSession and exhaustive -Headers
Invoke-WebRequest -Uri $Uri -Body $MultipartContent -Method 'POST' -WebSession $session -ContentType 'multipart/form-data;'-Headers @{
"Accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
  "Accept-Encoding"="gzip, deflate, br"
  "Accept-Language"="de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
  "Cache-Control"="max-age=0"
  "Origin"="$url"
  "Referer"="$url/public/system_under_test.php"
  "Sec-Fetch-Dest"="document"
  "Sec-Fetch-Mode"="navigate"
  "Sec-Fetch-Site"="same-origin"
  "Sec-Fetch-User"="?1"
  "Upgrade-Insecure-Requests"="1"
  "sec-ch-ua"="`"Chromium`";v=`"112`", `"Google Chrome`";v=`"112`", `"Not:A-Brand`";v=`"99`""
  "sec-ch-ua-mobile"="?0"
  "sec-ch-ua-platform"="`"Windows`""
}

# Dispose streams otherwise they could stay locked
foreach ( $stream in $OpenStreams){
  $stream.Dispose()
}