Passing POST data from file to curl.exe in powershell

Question

I have a HUGE request whose contents are in a file. I am getting an error, and I do not know why; I have this code:

$creds = "UN" + ":" + "PW"
$strCusips = $cusips -join ","
$req1 =  -join("GET,(", $strCusips, "),(stuff),,, ,Titles=SHORT,DATEFORM=YMD")
$request = "Request=" + [System.Net.WebUtility]::UrlEncode($req1)
$request > "temp.txt"
$response = curl.exe POST --silent --user $creds --data '@temp.txt' http://url/cgi/stuff

But this produces the following:

Post Method invoked, but no request found!

To be clear - the file contains something like this... (But is actually much bigger)

Request=GET%2C(0003128%2C0005588%2C0016308%2C0021216%2C0028262%2C0045614%2C0047245%2C0053673%2C0056650%2C0059585)%2C(stuff)%2C%2C%2C+%2CTitles%3DSHORT%2CDATEFORM%3DYMD

What gives? Is this an encoding issue? Is it not supposed to be URL encoded? Or is it?

EDIT:

To be a little more clear, this works perfectly:

$creds = "UN" + ":" + "PW"
$strCusips = $cusips -join ","
$req1 =  -join("GET,(", $strCusips, "),(stuff),,, ,Titles=SHORT,DATEFORM=YMD")
$request = "Request=" + [System.Net.WebUtility]::UrlEncode($req1)

$response = curl.exe POST --silent --user $creds --data $request http://url/cgi/stuff

(Please note all I did was pass the string to the curl request, as opposed to saving it to a file and then passing the file.)

ALSO, this works, so I know I can read the file, and its contents exist and are correct:

Get-Content temp.txt

EDIT2:

I have tried every --data-* flag that curl has, and they all produce the same result.

EDIT3:

This is what I run, with results:

$strCusips1 = $cusips[0..9] -join ","
$req1 =  -join("GET,(", $strCusips1, "),(CNTYRISKtxt),,,,Titles=SHORT,")
$request1 = "Request=" + [System.Net.WebUtility]::UrlEncode($req1)
$request1 > temp.txt
Get-Content temp.txt
curl.exe -v --user $creds --data-raw '@temp.txt' http://url/cgi/stuff


Request=GET%2C(0003128%2C0005588%2C0016308%2C0021216%2C0028262%2C0045614%2C0047245%2C0053673%2C0056650%2C0059585)%2C(CNTYRISKtxt)%2C%2C%2C%2
CTitles%3DSHORT%2C
curl.exe :   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
At line:26 char:1
+ curl.exe -v --user $creds --data-raw '@temp.txt' http:// ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (  % Total    % ...  Time  Current:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
*   Trying x.x.x.x...
* TCP_NODELAY set
* Connected to url (x.x.x.x) port 80 (#0)
* Server auth using Basic with user 'user'
> POST /cgi/stuff HTTP/1.1
> Host: http://url/cgi/stuff
> Authorization: Basic xxxxxxxxxxxxx
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Length: 9
> Content-Type: application/x-www-form-urlencoded
> 
} [9 bytes data]
* upload completely sent off: 9 out of 9 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 200 Script results follow
< Content-type: text/plain
< 
{ [43 bytes data]
* Curl_http_done: called premature == 0
100    52    0    43  100     9    551    115 --:--:-- --:--:-- --:--:--   551
Post Method invoked, but no request found!
* Closing connection 0

PS C:\Users\lhawk> 

Answer 1

In your last example,

curl.exe -v --user $creds --data-raw '@temp.txt' http://url/cgi/stuff

produces

> Content-Length: 9

 @temp.txt
01234567890123456789
         ^ nine bytes            

So switch the command back to -d to grab the file contents.

curl.exe -v -u $creds -d '@temp.txt' http://url/cgi/stuff
                      ^^                                 

That should be fine since you are encoding the response.

$request = "Request=" + [System.Net.WebUtility]::UrlEncode($req1)

curl -v further shows that Post Method invoked, but no request found! is coming from the server.

< HTTP/1.0 200 Script results follow
< Content-type: text/plain
< 
{ [43 bytes data]
* Curl_http_done: called premature == 0
100    52    0    43  100     9    551    115 --:--:-- --:--:-- --:--:--   551
Post Method invoked, but no request found!
* Closing connection 0

That will be true regardless of the different syntax errors.

Twice you mentioned that strings are working.

EDIT:

To be a little more clear, this works perfectly:

$creds = "UN" + ":" + "PW"
$strCusips = $cusips -join ","
$req1 =  -join("GET,(", $strCusips, "),(stuff),,, ,Titles=SHORT,DATEFORM=YMD")
$request = "Request=" + [System.Net.WebUtility]::UrlEncode($req1)

$response = curl.exe POST --silent --user $creds --data $request http://url/cgi/stuff

(Please note all I did was pass the string to the curl request, as opposed to saving it to a file and then passing the file.)

and

The request is good, since I can pass it as a string just fine, and get the desired results.

So, I think the problem is the encoding done by >.

"a happy string" > temp.txt

produced a text file encoded in UCS-2 LE BOM. And the file ends in a newline.

Though, the newline does not matter.

-d, --data <data>

...

If you start the data with the letter @, the rest should be a file name to read the data from, or - if you want curl to read the data from stdin. Multiple files can also be specified. Posting data from a file named 'foobar' would thus be done with -d, --data @foobar. When --data is told to read from a file like that, carriage returns and newlines will be stripped out. If you don't want the @ character to have a special interpretation use --data-raw instead.

Previously, I wrote a function to work around PowerShell's default encoding. So, let's use the function to see if the file's encoding is the issue.

function giveBinaryEqualFile ([string] $myInput, [string] $fileName)
{
    $Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
    [System.IO.File]::WriteAllText($fileName, $myInput, $Utf8NoBomEncoding)
}

# $request > "temp.txt" becomes 
$myFile = "C:\full\path\temp.txt" 
giveBinaryEqualFile $request $myFile

# $response = curl.exe -v --user $creds --data-raw '@temp.txt' http://url/cgi/stuff becomes 
$response = curl.exe -v --user $creds --data '@temp.txt' http://url/cgi/stuff
# since you are referencing a file. 
# --data-raw
# https://curl.haxx.se/docs/manpage.html

If this starts working then switch -v to --silent.