streamReader.ReadToEndAsync() changing message content

Question

I have a .Net Core 3.1 Web API that needs to interact with an external service that is providing a multipart/related response from a WCF Service with attachments in the context as the below block.

HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 1790633
Content-Type: multipart/related; type="text/xml"; start="<8fb5b3efa6474add9032cb80870dc065>"; boundary="------=_Part_20200731120159.494997"
Server: Microsoft-IIS/10.0
Set-Cookie: ASP.NET_SessionId=jga2opbhoirgnhjhmmsuxi1k; path=/; HttpOnly; SameSite=Lax
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Set-Cookie: ARRAffinity=f7b4f7f5d8cbd8c8bc5d9a4bf49e8b5938412c4294f6e2c2b5c2d5cfc4b4d437;Path=/;HttpOnly;Domain=lparouterpoc.azurewebsites.net
Date: Fri, 31 Jul 2020 13:33:04 GMT


--------=_Part_20200731120159.494997
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: binary
Content-Id: <8fb5b3efa6474add9032cb80870dc065>

<?xml version="1.0" encoding="iso-8859-1"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
...
</SOAP-ENV:Envelope>
--------=_Part_20200731120159.494997
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: 7b1fff220df5@xxxxx

%PDF-1.5
%����
...
Additional characters added in here
...
%%EOF

--------=_Part_20200731120159.494997
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: 844eb478dc05@xxxxx

%PDF-1.5
...
Additional characters added here
...
%%EOF
--------=_Part_20200731120159.494997--

Using SOAP UI, I see that there are attachments, but each attachment is a blank PDF.

The code in the .Net Core API is

using (var response = await Client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead))
{
    var stream = await response.Content.ReadAsStreamAsync();
    using var streamReader = new StreamReader(stream, Encoding.UTF8);
    var message = await streamReader.ReadToEndAsync(); // This is where output is changed to add additional characters
    var headers = response.Headers;
    return new customResponse
    {
        ContentResult = new ContentResult
        {
            Content = message,
            ContentType = response.Content.Headers.ContentType?.ToString(),
            StatusCode = (int)response.StatusCode
        },
        ResponseHeaders = headers
    };
}

The image below shows an example of where the additional characters are coming in causing the PDF's to be invalid \ corrupt \ blank.

An additional characteristic of this is the content length going from 1021690 to 1790633.

I have been able to debug it this far and test that the files before they get into the message (string) are correct as I am able to read the multipart content and write the files out to disk when checking that the files are indeed correct using

var content = await response.Content.ReadAsMultipartAsync();
for (var i = 0; i < content.Contents.Count; i++)
{
    var item = content.Contents[i];
    if (item.Headers.ContentType.ToString() == "application/octet-stream")
    {
        var bytes = await item.ReadAsByteArrayAsync();
        File.WriteAllBytes(@$"c:\temp\{i}.pdf", bytes);
    }

}

and then iterating through the content.Contents.

I would be grateful if anyone could tell me how \ why this is occurring and how I can stop this from occuring

Further investigation and hypothesis is could this be down to encoding? The WCF service for the attachments is TransferEncoding = "binary"

Left and middle of the image are the same, the middle written to disk direct from my API (not what I need) shows me that the file can be read and written.

The right is the direct output from the service, which comes back into SOAP UI and downloads as expected.

When looking at the response in the watch window the transfer encoding is empty so am a little confused on how to resolve this.

That is to be expected. You are reading the entire response as a utf-8 encoded string, which it isn't (`Content-Transfer-Encoding: binary`). — GSerg, Aug 04 '20 at 09:10
looking at an existing service that is being replace, the content-transfer-encoding might need to change to 8bit — Simon Price, Aug 04 '20 at 09:29
That would still result in data corruption if you were again to read it as utf-8. — GSerg, Aug 04 '20 at 09:35
using 8bit as I need to match the service that is being replace, I have tried the other encoding options and each result in `Unexpected end of MIME multipart stream. MIME multipart message is not complete.` What else could \ would \ should work? — Simon Price, Aug 04 '20 at 10:18
Nothing. You cannot read this response as a string, in any encoding. It's a byte array. — GSerg, Aug 04 '20 at 10:21
even trying `var bytes = await response.Content.ReadAsByteArrayAsync(); var messageFromBytes = System.Text.Encoding.Default.GetString(bytes);` I still get the same issue when getting the response as a byte array — Simon Price, Aug 04 '20 at 10:23
Like I said, you cannot read this response as a string. Saving it in a temporary byte array before reading it as a string does not change anything. — GSerg, Aug 04 '20 at 10:26
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/219181/discussion-between-simon-price-and-gserg). — Simon Price, Aug 04 '20 at 10:27
Im currently at a loss of why I can read it and write it to disk from a byte array what comes back into the API, but when its returned from the API im not able to write a good file — Simon Price, Aug 04 '20 at 10:33
Because when you attempt to convert the byte array to a string, it corrupts it, regardless of the encoding you are trying to use. — GSerg, Aug 04 '20 at 10:39
Does this answer your question? [How to parse XHR multipart/form-data request with multiple files](https://stackoverflow.com/questions/39417912/how-to-parse-xhr-multipart-form-data-request-with-multiple-files) — GSerg, Aug 04 '20 at 10:54
its an incomplete response and the answer in which its been taken from, requires a delimiter, which from the result. I dont know what it is — Simon Price, Aug 04 '20 at 11:46
It shows the entire code you need to process that response. In that email you've shown, the delimiter is `--------=_Part_20200731120159.494997`. If you needed to be told that, then probably you are not yet up for this task. On a side note, are those actual valid session id and cookie for `lparouterpoc.azurewebsites.net` that you have exposed? — GSerg, Aug 04 '20 at 12:04

score 1 · Accepted Answer · answered Aug 04 '20 at 13:06

try this,

public async Task<HttpResponseMessage> Post(string requestBody, string action)
{
    using (var request = new HttpRequestMessage(HttpMethod.Post, ""))
    {
        request.Headers.Add("SOAPAction", action);
        request.Content = new StringContent(requestBody, Encoding.UTF8, "text/xml");
        var response = await Client.SendAsync(request);
        return response;
    }
}

and then in your controller return it like this

var result = await _service.Post(requestBody, action);

var stream = await result.Content.ReadAsStreamAsync();
await stream.CopyToAsync(HttpContext.Response.Body);

score 0 · Answer 2 · answered Mar 27 '23 at 14:41

In my case, response had this header:

content-encoding: br

So I had to use BrotliStream (.Net 5)

        response.Body.Seek(0, SeekOrigin.Begin); // MemoryStream

        var encoding = response.Headers["content-encoding"];
        Stream responseStream;
        
        if (encoding.Contains("br"))
            responseStream = new BrotliStream(response.Body, CompressionMode.Decompress);
        else
            responseStream = response.Body;
        
        var text = await new StreamReader(responseStream).ReadToEndAsync();
        response.Body.Seek(0, SeekOrigin.Begin);

see: How do I decode an encoded HttpWebResponse?

streamReader.ReadToEndAsync() changing message content

2 Answers2