Streaming large file uploads to ASP.NET MVC

Question

For an application I'm working on, I need to allow the user to upload very large files--i.e., potentially many gigabytes--via our website. Unfortunately, ASP.NET MVC appears to load the entire request into RAM before beginning to service it--not exactly ideal for such an application. Notably, trying to circumvent the issue via code such as the following:

if (request.Method == "POST")
{
    request.ContentLength = clientRequest.InputStream.Length;
    var rgbBody = new byte[32768];

    using (var requestStream = request.GetRequestStream())
    {
        int cbRead;
        while ((cbRead = clientRequest.InputStream.Read(rgbBody, 0, rgbBody.Length)) > 0)
        {
            fileStream.Write(rgbBody, 0, cbRead);
        }
    }
}

fails to circumvent the buffer-the-request-into-RAM mentality. Is there an easy way to work around this behavior?

Where does your code reside in your application? Controller? handler? module? — Ronnie Overby, Dec 20 '12 at 16:02
I know this question is old, but it's a really good question that I'm using as a reference for a project I'm working on. Quick question -- what is the "cb" supposed to denote in your variable cbRead? — KSwift87, Dec 21 '12 at 22:43
@KSwift87 `cb` stands for "Count of Bytes", which is standard Apps Hungarian notation at Fog Creek. — Benjamin Pollack, Dec 23 '12 at 21:41

score 25 · Accepted Answer · answered Aug 10 '09 at 21:36

25

It turns out that my initial code was basically correct; the only change required was to change

request.ContentLength = clientRequest.InputStream.Length;

to

request.ContentLength = clientRequest.ContentLength;

The former streams in the entire request to determine the content length; the latter merely checks the Content-Length header, which only requires that the headers have been sent in full. This allows IIS to begin streaming the request almost immediately, which completely eliminates the original problem.

answered Aug 10 '09 at 21:36

Benjamin Pollack

27,594
16
81
105

1

So, basically simply touching the InputStream property defeats your disk buffering. Is it possible to obtain a stream handle to the disk-buffered file? – Daniel Crenna Jan 15 '10 at 04:34
2

Not that I've been able to determine, but I stopped seriously investigating this once I got something that worked. – Benjamin Pollack Feb 08 '12 at 14:53

D'Arcy Rittich · Answer 2 · 2009-06-25T23:08:44.997

Sure, you can do this. See RESTful file uploads with HttpWebRequest and IHttpHandler. I have been using this method for a few years and have a site that has been tested with files of at least several gigabytes. Essentially, you want to create your own IHttpHandler, which is easier than it sounds.

In a nutshell, you create a class that implements the IHttpHandler interface, meaning you have to support the IsReusable property and the ProcessRequest method. On top of that, there is a minor change to your web.config, and it works like a charm. At this stage in the life cycle of the request, the entire file being uploaded does not get loaded into memory, so it neatly steps around out of memory issues.

Note that in the web.config,

<httpHandlers>
 <add verb="*" path="DocumentUploadService.upl" validate="false" type="TestUploadService.FileUploadHandler, TestUploadService"/>
</httpHandlers>

the file referenced, DocumentUploadService.upl, doesn't actually exist. That is just there to give an alternate extension so that the request is not intercepted by the standard handler. You point your file upload form to that path, but then your FileUploadHandler class kicks in and actually receives the file.

Update: Actually, the code I use is different from that article, and I think I stumbled on the reason it works. I use the HttpPostedFile class, in which "Files are uploaded in MIME multipart/form-data format. By default, all requests, including form fields and uploaded files, larger than 256 KB are buffered to disk, rather than held in server memory."

if (context.Request.Files.Count > 0)
{
    string tempFile = context.Request.PhysicalApplicationPath;
    for(int i = 0; i < context.Request.Files.Count; i++)
    {
        HttpPostedFile uploadFile = context.Request.Files[i];
        if (uploadFile.ContentLength > 0)
        {
            uploadFile.SaveAs(string.Format("{0}{1}{2}",
              tempFile,"Upload\\", uploadFile.FileName));
        }
    }
}

Your code looks nearly identical to mine; the only difference is that I'm running it in ASP.NET MVC, and you're running it directly in a new IHttpHandler. Do you happen to know why there's a difference? Is there something in my code that's forcing the stream to be pulled in? — Benjamin Pollack, Jun 25 '09 at 22:04
I uses a Flash file uploader, this is the best way to queue multiple files for upload and get upload status. (E.g., http://www.codeproject.com/KB/aspnet/FlashUpload.aspx) — D'Arcy Rittich, Jul 20 '09 at 18:26
The IHttpHandler approach was really useful. I've integrated a similar solution in my file uploader. I needed to use GetBufferlessInputStream() to prevent ASP.NET from waiting until the post data had been fully uploaded. Some extended request processing logic is included here http://blog.appsoftware.com/post/2013/03/08/ASPNET-File-Uploader-with-SignalR-Progress-Bar-and-Extended-HTTP-Request-Processing — gbro3n, Mar 09 '13 at 01:43

Streaming large file uploads to ASP.NET MVC

2 Answers2

Linked

Related