Transferring large files: combining streamed transfer and content-length

Question

TL;DR: How can I stream large files with a known size using WCF, and still show progress (Content-Length) to the final user (a web browser)?

I have a WCF REST service which downloads and then serves very large files (1-20Gb) to a web browser. To simplify, think of my service as a proxy. This obliges me to set TransferMode = Streamed or TransferMode = StreamedResponse on the Binding, or the end client will have to wait for the source files to be downloaded to the webserver before the actual download begins. Also, buffered transfer mode kills the server for large files (RAM usage). Intermediate disk storage is not an option. From TransferMode man page:

(...) Buffered transfers hold the entire message in a memory buffer until the transfer is complete.

But when setting TransferMode to Streamed or StreamedResponse, WCF no longer returns the header Content-length to the client, and a new header Transfer-Encoding: chunked is added. This is consistent with wikipedias article on chunked transfer:

(...) uses the Transfer-Encoding HTTP header in place of the Content-Length header (...)

But I always know the size of the data to be transferred beforehand, and for the end user, it's very frustrating not to know the size of the download. So:

(How) can I configure the WCF binding to use a "streaming" Transfer mode (more specifically, not buffering the entire message before sending) and still use the Content-Length header?

Some leads:

• This q/a states that the http standard disallows both Transfer-Encoding and Content-length: 123456 in the same message, so I guess that's not an option?

• I have tried modifying the headers using an inspector in IDispatchMessage.BeforeSendReply but at this point the Content-Length header has not yet been removed, and Transfer-Encoding has not yet been set, so it's "too early". I have later read that chunked transfer encoding is on TCP level, so changing the header at this point probably wont work even if I could.

• I have tried setting aspNetCompatibilityEnabled="true", setting wcf transfer mode to buffered output (default) and then set System.Web.HttpContext.Current.Response.BufferOutput = false;. This is ignored by wcf though, the message is clearly buffered.

• It seems to be a missing feature, according to this link. But there may still be a quirky workaround somewhere..

Answer 1

This is a tested workaround, that only works for WCF under IIS - I have not found any solution for a self-hosted service.

In short - turn on aspNetCompatibility which gives you runtime access to System.Web.HttpContext.Current.

Web.config:

(...)
    <system.serviceModel>
      <bindings>
          <webHttpBinding>
              <binding transferMode="Streamed">
              </binding>
          </webHttpBinding>
      </bindings>
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
(...)
</system.serviceModel>

and in your service function that returns a Stream:

HttpContext.Current.Response.Headers.Add("Content-Length", 
contentLength.ToString());

Anything like the following will be silently ignored:

WebOperationContext.Current.OutgoingResponse.Headers["Content-Length"] = 
        contentLength.ToString();

Simple as that! Creds goes to Uffe Lausen's question on Msdn

Answer 2

If you have the control over the client code, you could write the data length as the first bytes in response stream, this is how i do it to transfer large files, but the client must know that the first bytes are only the size and not containing any data.

Example @server:

public Stream GetBigData()
{
    byte[] bigData = GetBigDataFromSomeWhere();
    var ms = new MemoryStream();
    var lengthInfo = BitConverter.GetBytes(bigData.Length);
    ms.Write(lengthInfo, 0, lengthInfo.Length);
    ms.Write(bigData , 0, bigData .Length);
    ms.Flush();
    ms.Position = 0;
    return ms;
}

Example @client:

using (var respStream = yourService.GetBigData())
using (var resultStream = new MemoryStream())
{
    //read first 4 bytes 
    var lengthBuffer = new byte[4];
    respStream.Read(lengthBuffer, 0, 4);
    var totalCount = BitConverter.ToInt32(lengthBuffer, 0);
    resultStream.Capacity = totalCount;

    //read rest
    var readcount = 0;
    var buffer = new byte[1024 * 32];
    while ((readcount = respStream.Read(buffer, 0, buffer.Length)) > 0)
    {
        resultStream.Write(buffer, 0, readcount);
        UpdateProgress(readcount, totalCount)
    }
}