How to pass an array via web-request

Question

Situation:

We collect automatically many reports from some web services (PowerShell script running every night), and every day in manual mode (drag and drop on web-form) this reports are loaded in our DB.

Now our IT department gave us an API that can handle this job without user interaction.

Problem:

As was written in covering letter (about this API) it waits for reports[n] array with file. It can be done with PHP and curl:

$report = 'report_20160825.{8302F59C-E1E4-410F-BE37-A24CCD7E515E}.zip';
$cfile = new CURLFile(realpath($report),'application/zip',$report);
$PostData = array("reports[0]"=>$cfile);

But how to send array named reports[n] via PowerShell?

What I have tried:

$url = "https://test.example.com/uploadAPI/upload.php"
$Source =  "D:\report_20160825.{8302F59C-E1E4-410F-BE37-A24CCD7E515E}.zip"
$contentType = "multipart/form-data"

$Username = "ApiUploadKey"
$Headers = @{Authorization="Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:" -f $Username)))}

$FileContens = get-content $Source
$PostData = @{"reports[0]" = $FileContens;} 
#$reports = @($FileContens,'application/zip',$Source)

(Invoke-WebRequest -uri $url -Method POST -Headers $Headers -Body $PostData -ContentType $contentType).Content

#Invoke-RestMethod -uri $url -Method POST -Headers $Headers -Body $PostData -ContentType $contentType

That gives me a response that I am passing not-a-report.

EDIT 2016-10-11

Further investigation bring me to this answer and this article. I tried to use boundary:

clear
$url = "https://test.example.com/uploadAPI/upload.php"
$filename = "report_20160825.{8302F59C-E1E4-410F-BE37-A24CCD7E515E}.zip"
$Source =  "D:\"+$filename

$Username = "ApiUploadKey"
$Headers = @{Authorization="Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:" -f $Username)))}

$FileContens = get-content $Source
$enc = [System.Text.Encoding]::GetEncoding("iso-8859-1")
$fileBin = [IO.File]::ReadAllBytes($Source)
$fileEnc = $enc.GetString($fileBin)


$boundary = [System.Guid]::NewGuid().ToString()

$LF = "`n"

$contentType = "multipart/form-data; boundary=--$boundary"
#$bodyLines = "--"+$boundary+$LF+"Content-Disposition: form-data; name=`"reports[]`"; filename=`""+$filename+"`""+$LF+$LF+"Content-Type: application/zip"+$LF+"--"+$boundary+"--"+$LF+$LF+$FileContens+$LF+"--"+$boundary

$bodyLines = (
    "--$boundary",   #I have tried reports[0] here too
    "Content-Disposition: form-data; name=`"reports[]`"; filename=`"$filename`"",   # filename= is optional
    "Content-Type: application/zip",
    "",
    #$FileContens,
    $fileEnc,
    "--$boundary--"
    ) -join $LF


try {

    #Invoke-WebRequest -Uri "https://asrp.cntd.ru/uploadAPI/" -Headers $Headers -WebSession $ws
    Invoke-RestMethod  -Uri $url -Body $bodyLines -Method POST -Headers $Headers -ContentType $contentType -TimeoutSec 50
}
catch [System.Net.WebException] {
    Write-Error( "FAILED to reach '$url': $_" )
    throw $_
}

But with same results.

Also I tried this:

$wc = new-object System.Net.WebClient
$wc.Credentials = new-object System.Net.NetworkCredential("ApiUploadKey","")

ls "D:\*.zip" | foreach { 
    $wc.UploadFile('https://test.example.com/uploadAPI/upload.php', $_.FullName )
    write-host $_.FullName
}

And one more solution from this answer:

Invoke-RestMethod -Uri $url -InFile $Source -ContentType "multipart/form-data" -Method POST -Headers $Headers

Always same response - not a report

EDIT 2016-10-17

curl

I have downloaded curl for windows. And use it like:

curl.exe https://test.example.com/uploadAPI/upload.php --user ApiUploadKey: --form "reports[0]=@d:\report_746_226255_20161010_1635.zip;type=application/zip"

And that gave me:

[{"code" : 102 , "guid" : "{23CE9F7F-BEC8-4D4C-8AC3-2865CFA94FBD}" , "id" : "5804902bc73a2475177464", "filename" : "report_746_226255_20161010_1635.zip"}]

So with curl it works fine!

fiddler

Don't know exactly what log to post.

When I send file like this:

POST https://test.example.com/uploadAPI/upload.php

Content-Type: multipart/form-data; boundary=-------------------------acebdf13572468
User-Agent: Fiddler
Host: test.example.com
Authorization: Basic ...
Content-Length: 21075175

Request body:

---------------------------acebdf13572468
Content-Disposition: form-data; name="reports[]"; filename="report_746_226254_20161010_1320.{B67A9D89-368B-4665-96AC-77C2CA0F4766}.zip"
Content-Type: application/zip

<@INCLUDE *D:\report_746_226254_20161010_1320.{B67A9D89-368B-4665-96AC-77C2CA0F4766}.zip*@>
---------------------------acebdf13572468--

I got

HTTP/1.1 200 OK
Date: Mon, 17 Oct 2016 10:04:43 GMT
Server: Apache/2.4.20 (Win64) OpenSSL/1.0.2h PHP/7.0.6
X-Powered-By: PHP/7.0.6
Set-Cookie: SESSION_UPLOAD_ID=.....; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Connection: close
Content-Length: 193
Content-Encoding: none
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8

[{"code" : 102 , "guid" : "{B67A9D89-368B-4665-96AC-77C2CA0F4766}" , "id" : "5804a23be4152532018928", "filename" : "report_746_226254_20161010_1320.{B67A9D89-368B-4665-96AC-77C2CA0F4766}.zip"}]

Answer 1

The PHP code use Curl class and set $PostData as:

 $PostData = array("reports[0]"=>$cfile);

$PostData is an array of key/value pair (in PHP), and the key is named Reports[0]. it's just a string name represent the key not an array element.

in curl commandLine tool we can upload file with the command:

 curl.exe   -F Reports[0]=@"I:\test\report_20160825.{8302F59C-E1E4-410F-BE37-A24CCD7E515E}.zip"  http://MyServer/uploader/api/upload

or, it may be (note the key Reports[0][2][3] ):

   curl.exe   -F Reports[0][2][3]=@"I:\test\report_20160825.{8302F59C-E1E4-410F-BE37-A24CCD7E515E}.zip"  http://asd-pc/uploader/api/upload

-F is for entering Form key/value pair Note @ before filename Try it in your server and view the session in Fiddler to know how curl and PHP send the request to the server.

You can pass other parameters like -H (header) -v (verbose) -F for other files with other key name e.g Reports1.

I tested this code with My web server running Web Api2 service that upload file to the server and it's running fine.

So, the problem is: how to convert that code to work with Powershell.

In curl either PHP or commandLine tool , it auto prepare the Header and the Request Body for file uploading that support Multipart/form-data.

Both Powershell commands: Invoke-WebRequest and Invoke-RestMethod are unaware on how to format the request body in order to comply to the standard of Multipart/form-data as given in the RFC Forms: multipart/form-data

You have to manually set the message body and then invoke your call

The following script Upload file to web server using powershell.

I set Content-Type: application/octet-stream to support any type including zip files.

    function Upload-File ( $InFile,$Uri    )
    {    
        $Username = "ApiUploadKey"
        $Headers = @{Authorization="Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:" -f $Username)))}
        $LF = "`n"   
        $fileName = Split-Path $InFile -leaf
        $boundary = [System.Datetime]::Now.Ticks.ToString()
        $binaryData = [System.IO.File]::ReadAllBytes($InFile)   
        $binaryEncoded=[System.Text.Encoding]::GetEncoding("iso-8859-1").GetString($binaryData) 
        $ContentType = "application/octet-stream" 

        $body = @"
    --$boundary
    Content-Disposition: form-data; name=`"Reports[0]`"; filename=`"$fileName`"
    Content-Type: $ContentType
    $LF
    $binaryEncoded
    --$boundary--
    $LF
    "@  

        try
        { 

          return Invoke-RestMethod -Uri $Uri -Method Post -ContentType "multipart/form-data; boundary=$boundary"   -Body $body -Headers $Headers                             

        }
        catch [Exception]
        {
            $PSCmdlet.ThrowTerminatingError($_)
        }
    }


    #--   test the function ---------------------------------
    cls
    $uri = "http://MyServer/uploader/api/upload"        
    $filePath = "I:\test\report_20160825.{8302F59C-E1E4-410F-BE37-A24CCD7E515E}.zip"
    $response = Upload-File -InFile $filePath -Uri $uri  #-Credential $credentials
    $response

The output session from fiddler

    POST http://MyServer/uploader/api/upload HTTP/1.1
    Authorization: Basic QXBpVXBsb2FkS2V5Og==
    User-Agent: Mozilla/5.0 (Windows NT; Windows NT 6.1; en-US) WindowsPowerShell/5.0.10586.117
    Content-Type: multipart/form-data; boundary=636123339963709320
    Host: MyServer
    Content-Length: 379
    Connection: Keep-Alive

    --636123339963709320
    Content-Disposition: form-data; name="Reports[0]"; filename="report_20160825.{8302F59C-E1E4-410F-BE37-A24CCD7E515E}.zip"
    Content-Type: application/octet-stream


    xxxxxxxxxxxxxxxxxxxxxxxxxxxStream of byte sxxxxxxxxxxxxxxx

In the session data above you see Reports[0] is titled name

You can use the same code with the command: Invoke-WebRequest