Most efficient way to write large files to HttpResponse in ASP.NET

Question

I am creating a ZIP file on the fly. I am using dotnetzip for doing this. The size of my ZIP file can go up to 500 MB, with many users trying to download it at the same time.

What is the most efficient way in which I can serve up the files? If given a choice, I would rather not save the files on the disk either, since might cause severe disk space constraints.

Edit:More about my use case:

We are hosting our files in a SharePoint 2010 asset library hosted in an Intranet site which has users all around the world. The files typically range from 10-80 MBs. The users would like the ability to download multiple files at once.

Can you tell us more about the use case? Use the web for these sizes of files with out caching smells like a design flaw - but of course your use case may be different to a more common one. — Preet Sangha, Apr 28 '11 at 23:06

score 3 · Accepted Answer · edited May 23 '17 at 12:08

Well, in theory:

For a traditional ASP.Net application you should just be able to write the response data (bytes) to the HttpContext.Response.OutputStream (how you get the http response context will depend on how you handle the download request e.g. if you are implementing IHttpHandler then you get the http context passed to you).

Looking at the DotNetZip examples it looks like the Save method takes a stream, so this would be as simple as

zip.Save(context.Response.OutputStream);

If the zip file is re-used and downloaded by many users then you can instead write the zip to a MemoryStream when creating the zip which allows you to later copy the contents of this memory stream to individual responses:

MemoryStream stream = new MemoryStream()
zip.Save(stream);
// Save data somewhere common (e.g. cache it)
byte[] data = stream.ToArray();

To write this data back to a response:

MemoryStream reader = new MemoryStream(data);
CopyStream(reader, context.Response.OutputStream);

See Best way to copy between two Stream instances - C# for an implementation of CopyStream.

However in reality:

Thinking about this for a second, this means that if the zip file is 500MB we are storing 500MB of data in memory - this might be fine if this is the only zip file ever in existance but if there are 3 or 4 of these we are quickly going to run out of "memory" (i.e. virtual address space).

The solution? I'm afraid the easiest way is to save your zip to a file (even if it is a temporary file not directly served up by IIS) instead of to a memory stream:

// To save the zip
string filename = Path.GetTempFileName();
zip.Save(filename);

// To write the file
context.Response.TransmitFile(filename);

You should probably also delete the file when you are done.

Note that you only need to bother with this if you are determined to share the same zip between multiple users - if you just construct the zip on a per-user basis and write it directly to the output stream using zip.Save(OutputStream) then things are a lot less hassle. My advice would be to first just do it the simple way and then test to see if you get performance problems that can be solved by only creating the zip once.

You may find this link useful when you come to implementing the bit that says "You should probably also delete the file when you are done." http://stackoverflow.com/q/2688282/150342 — Colin, Feb 27 '13 at 10:46

score 1 · Answer 2 · answered Apr 28 '11 at 23:19

For this, you'd probably do better by doing it asynchronously using a message queue. This way, the zipping is added to the queue and you get either get the one server to do it, or multiple servers. I don't know if this meets your requirements though.

I have used Rabbit MQ (http://www.rabbitmq.com/) before.

Most efficient way to write large files to HttpResponse in ASP.NET

2 Answers2