I am in the process of working on updating some PowerShell code that usesd System.Net.HttpWebRequest to handle file transfers to and from a web service. The System.IO.FileStream class has a major problem with handling file streams being transferred to a web service using HttpWebRequest that Microsoft isn't going to fix or address within DotNetCore, and thus PowerShellCore (MS PowerShell GitHub Issue 4129). Essentially, one runs into a Stream was too long exception in .NetCore's legacy API implementation for FileStream and PowerShellCore/PowerShell7. So, I have the following verified and working PowerShell code I am trying to convert to C# async class method using HttpClient to asynchronously upload files to an Apache/Spring-based REST API service:
$Hostname = "somewebserver"
$Method = "POST"
$File = Get-ChildItem C:\Directory\file.iso
$FSOpenMode = [System.IO.FileMode]::Open
$FSRead = [System.IO.FileAccess]::Read
# Read file using FileStream
$fs = [IO.FileStream]::new($File.FullName, $FSOpenMode, $FSRead)
[void]$fs.FlushAsync()
try
{
# Set the URL that will be the upload destination
$url = "https://{0}/upload?uploadfilename={1}" -f $Hostname, $File.Name
$_DispositionContentType = "application/octet-stream"
[System.Net.httpWebRequest]$uploadRequest = [System.Net.WebRequest]::Create($uri)
$uploadRequest.Method = $Method
$boundary = "---------------------------" + [DateTime]::Now.Ticks.ToString("x")
[byte[]]$BoundaryBytes = [System.Text.Encoding]::UTF8.GetBytes("`r`n--" + $boundary + "`r`n");
$disposition = "Content-Disposition: form-data; name=`"file`"; filename=`"{0}`";`r`nContent-Type: {1}`r`n`r`n" -f $File.Name, $_DispositionContentType
[byte[]]$ContentDispBytes = [System.Text.Encoding]::UTF8.GetBytes($disposition);
[byte[]]$EndBoundaryBytes = [System.Text.Encoding]::UTF8.GetBytes("`r`n--" + $boundary + "--`r`n")
$uploadRequest.Timeout = 1200000
$uploadRequest.ContentType = "multipart/form-data; boundary={0}" -f $boundary
$uploadRequest.Headers.Item("auth") = "SessionID"
$uploadRequest.Headers.Item("uploadfilename") = $File.Name
$uploadRequest.AllowWriteStreamBuffering = $true
$uploadRequest.SendChunked = $true
$uploadRequest.ContentLength = $BoundaryBytes.length + $ContentDispBytes.length + $File.Length + $EndBoundaryBytes.Length
$uploadRequest.Headers.Item("ContentLength") = $BoundaryBytes.length + $ContentDispBytes.length + $File.Length + $EndBoundaryBytes.Length
$rs = $uploadRequest.GetRequestStream()
[void]$rs.FlushAsync()
[byte[]]$readbuffer = [byte[]]::new(4096 * 1024)
$rs.write($BoundaryBytes, 0, $BoundaryBytes.Length);
$rs.write($ContentDispBytes, 0, $ContentDispBytes.Length);
# This is used to keep track of the file upload progress.
$numBytesToRead = $fs.Length
[int64]$numBytesRead = 0
$_sw = [System.Diagnostics.Stopwatch]::StartNew()
$_progresssw = [System.Diagnostics.Stopwatch]::StartNew()
while ($bytesRead = $fs.Read($readbuffer, 0, $readbuffer.length))
{
[void]$fs.Flush()
$rs.write($readbuffer, 0, $bytesRead)
[void]$fs.Flush()
[void]$rs.Flush()
# Keep track of where we are at clearduring the read operation
$_numBytesRead += $bytesRead
# Flush the buffer every 200ms and 1MB written
if ($_progresssw.Elapsed.TotalMilliseconds -ge 200 -and $_numBytesRead % 100mb -eq 0)
{
[void]$rs.flush()
}
# Use the Write-Progress cmd-let to show the progress of uploading the file.
[Int]$_percent = [math]::floor(($_numBytesRead / $fs.Length) * 100)
# Elapsed time to calculat throughput
[Int]$_elapsed = $_sw.ElapsedMilliseconds / 1000
if ($_elapsed -ne 0 )
{
[single]$_transferrate = [Math]::Round(($_numBytesRead / $_elapsed) / 1mb)
}
else
{
[single]$_transferrate = 0.0
}
$status = "({0:0}MB of {1:0}MB transferred @ {2}MB/s) Completed {3}%" -f ($_numBytesRead / 1MB), ($numBytesToRead / 1MB), $_transferrate, $_percent
# Handle how poorly Write-Progress adds latency to the file transfer process by only refreshing the progress at a specifc interval
if ($_progresssw.Elapsed.TotalMilliseconds -ge 500)
{
if ($_numBytesRead % 1mb -eq 0)
{
Write-Progress -activity "Upload File" -status ("Uploading '{0}'" -f $File.Name) -CurrentOperation $status -PercentComplete $_percent
}
}
}
$fs.close()
# Write the endboundary to the file's binary upload stream
$rs.write($EndBoundaryBytes, 0, $EndBoundaryBytes.Length)
$rs.close()
$_sw.stop()
$_sw.Reset()
Write-Progress -activity "Upload File" -status ("Uploading '{0}'" -f $File.Name) -Complete
}
catch [System.Exception]
{
if ($fs)
{
$fs.close()
}
if ($_sw.IsRunning)
{
$_sw.Stop()
$_sw.Reset()
}
# Dispose if still exist
if ($rs)
{
$rs.close()
}
Throw $_
}
try
{
# Indicate we are waiting for the API to process the uploaded file in the write progress display
Write-Progress -activity "Upload File" -status ("Uploading '{0}'" -f $File.Name) -CurrentOperation "Waiting for completion response from appliance." -percentComplete $_percent
# Get the response from the API on the progress of identifying the file
[Net.httpWebResponse]$WebResponse = $uploadRequest.getResponse()
$uploadResponseStream = $WebResponse.GetResponseStream()
# Read the response & convert to JSON
$reader = [System.IO.StreamReader]::new($uploadResponseStream)
$responseJson = $reader.ReadToEnd()
$uploadResponse = ConvertFrom-Json $responseJson
$uploadResponseStream.Close()
$uploadRequest = $Null
# Finalize write progress display
Write-Progress -activity "Upload File" -CurrentOperation "Uploading $Filename " -Completed
}
catch [Net.WebException]
{
if ($null -ne $_.Exception.Response)
{
Try
{
# Need to see if Response is not empty
$sr = [IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
}
Catch
{
$PSCmdlet.ThrowTerminatingError($_)
}
$errorObject = $sr.readtoEnd() | ConvertFrom-Json
# dispose if still exist
if ($rs)
{
$rs.close()
}
if ($fs)
{
$fs.close()
}
$sr.close()
# This New-ErrorRecord is an internal function that generates a new System.Management.Automation.ErrorRecord object to then throw
$ErrorRecord = New-ErrorRecord HPEOneview.Appliance.UploadFileException $errorObject.ErrorCode InvalidResult 'Upload-File' -Message $errorObject.Message -InnerException $_.Exception
Throw $ErrorRecord
}
else
{
Throw $_
}
}
The following working C# code is the result of some research I have done to try to understand how HttpClient works with multipart/form-data uploads:
public class HttpClientUpload : IDisposable
{
private readonly string _uploadUrl;
private readonly string _sourceFilePath;
private readonly string _authToken;
// THIS IS ONLY HERE FOR TESTING
private static HttpClientHandler _handler = new HttpClientHandler() { ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => { return true; } };
private HttpClient _httpClient = new HttpClient(_handler)
{
Timeout = TimeSpan.FromDays(1)
};
public delegate void ProgressChangedHandler(long? totalFileSize, long totalBytesDownloaded, double? progressPercentage);
public event ProgressChangedHandler ProgressChanged;
public HttpClientUpload(string uploadUrl, string sourceFilePath, string authToken)
{
_uploadUrl = uploadUrl;
_sourceFilePath = sourceFilePath;
_authToken = authToken;
}
// Borrowed code from https://stackoverflow.com/questions/16416601/c-sharp-httpclient-4-5-multipart-form-data-upload
public void UploadAsync()
{
var Path = _sourceFilePath;
var Appliance = _uploadUrl;
var AuthToken = _authToken;
var m_CancellationSource = new CancellationTokenSource();
var token = m_CancellationSource.Token;
var fileInfo = new System.IO.FileInfo(Path);
var uri = $"https://{Appliance}/rest/firmware-bundles?uploadfilename={fileInfo.Name}";
// If the file extension is CRL, we need to use a different disposition/mime type declaration.
var DispositionContentType = fileInfo.Extension == "crl" ? "application/pkix-crl" : "application/octet-stream";
_httpClient.DefaultRequestHeaders.Add("User-Agent", $"Custom User Agent ({Environment.OSVersion.ToString()})");
_httpClient.DefaultRequestHeaders.Add("X-API-Version", "1800");
_httpClient.DefaultRequestHeaders.Add("auth", AuthToken);
_httpClient.DefaultRequestHeaders.Add("uploadfilename", fileInfo.Name);
_httpClient.DefaultRequestHeaders.Add("accept-language", "en_US");
_httpClient.DefaultRequestHeaders.Add("accept-encoding", "gzip, deflate");
using (var content = new MultipartFormDataContent("---------------------------" + DateTime.Now.Ticks.ToString("x")))
{
content.Headers.Add("Content-Disposition", $"form-data; name=\"file\"; filename=\"{fileInfo.Name}\"");
content.Headers.ContentLength = fileInfo.Length;
FileStream fs = File.OpenRead(Path);
var streamContent = new StreamContent(fs);
streamContent.Headers.Add("Content-Type", $"{DispositionContentType}");
streamContent.Headers.Add("Content-Disposition", $"form-data; name=\"file\"; filename=\"{fileInfo.Name}\"");
content.Add(streamContent, "file", fileInfo.FullName);
Task<HttpResponseMessage> message = _httpClient.PostAsync(uri, content);
var input = message.Result.Content.ReadAsStringAsync();
Console.WriteLine(input.Result);
Console.WriteLine("Press enter to continue...");
Console.Read();
}
}
public void Stop()
{
_httpClient.CancelPendingRequests();
throw new OperationCanceledException("File upload cancelled.");
}
public void Dispose()
{
_httpClient?.Dispose();
}
}
The code above can be invoked from PowerShell as such:
$Hostname = "somewebserver"
$AuthToken = "SessionID"
$FileName = 'C:\directory\somefile.iso'
$Task = ([HttpClientUpload]::new($Hostname, $FileName, $AuthToken)).UploadAsync()
The file appears to be transferred to the target, but te API endpoint generates an error (not an HTTP error, but a JSON response that it couldn't parse the file) and I get this in our API application log file:
caught apache common fileuploadexception ex=Processing of multipart/form-data request failed. Stream ended unexpectedly while uploading file somefile.iso
I know that this is caused by the fact that the request from the client must append a byte array of an endboundary. Because without it, I would get the very same message with the original PowerShell code I currently use if I omit the following line(s):
# Write the endboundary to the file's binary upload stream
$rs.write($EndBoundaryBytes, 0, $EndBoundaryBytes.Length)
These are the following posts I have seen, but have only been a bit helpful in understanding the basic function of attaching a filestream object to an HttpClient object:
- Progress bar with HttpClient
- C# HttpClient 4.5 multipart/form-data upload (this one is close, but doesn’t offer tracking of progress, and am unsure how to attach an Event to it, let alone needing to chunk the upload for a FileStream, not MemoryStream)
- post multipart/form-data in c# HttpClient 4.5
- How to display upload progress using C# HttpClient PostAsync
- C#: HttpClient, File upload progress when uploading multiple file as MultipartFormDataContent (the post here marked as the Correct answer doesn’t appear right to me, especially when the code is documented for Download, not upload)
- How can I calculate progress with HttpClient PostAsync?
So, the TL;DR to my post. My questions are:
- How does one ensure that a multipart/form-data request that needs to include a binary stream of a file contains the start and end boundary for the
HttpClientrequest? - Any guidance on how to then chunk the file stream in order to report buffer transfer progress? I intend to read this from PowerShell after the async task has begun, in order to use
Write-Progressback to the user.
