3

I'm currently planing an update mechanism via web api. The transferred files can be up to 1 GB in size and there can be 20 clients that are simultaneously trying to get files.

Whenever I looked at examples I found something like this (simplified):

public HttpResponseMessage GetFile(string name)
{
     var reqFile = @"C:\updates\" + name;
     var dataBytes = File.readAllBytes(reqFile)
     var dataStream new MemoryStream (dataBytes);
     HttpResponseMessage httprm = Request.CreateResponse(HttpStatusCode.OK);
     httprm.Content = new StreamContent(dataStream);
     httprm.Content.Headers.ContentDisposition = = new ContentDispositionHeaderValue("attachment");
     httprm.Content.Headers.ContentDisposition.FileName = name;
     httprm.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octetstream");

     return httprm;
}

Now though in these examples they load the whole file into memory (up to 20 GB total thus for a single file that the clients need to download in my case.....which is not optimal, even if I only load each specific file only once I run into similar problems as they can be at differnt update steps).

An Option I'm seeing is that beforehanded I split the file into 10 MB chunks and let the client download it and then put it together again. That way the maximum memory footprint for the files alone would be 200 MB which is in a more acceptable area.

Still though I'm wondering if there is another way to accomplish the download without having to use 20 GB of memory for 20 concurrent clients (aside from the splitup)?

Thomas
  • 2,886
  • 3
  • 34
  • 78
  • That's just bad example, don't use it. Use `FileStream` directly instead of copying the whole file into `MemoryStream` for absolutely no reason. – Evk Mar 27 '18 at 09:49
  • @evk I only found examples like the above with memorystream. So I could "Just" use filestream there for the streamcontent and it does not put the file into my Memory or the Clients, and just Transfers "chunks" at a time while the Client loads the data and does whatever he wants with it (if into a filestream there then directly onto the hard drive)? – Thomas Mar 27 '18 at 09:53
  • Yes, just pass `FileStream` there and you will be fine. It will read file with small chunks (depends on buffer size, usually 1MB) and write directly to http response (so basically to network socket). Ensure to not lock that stream (so open for reading and with `Read` share) so that multple requests can read from it at the same time. – Evk Mar 27 '18 at 09:58
  • And don't forget to ensure that you download this file correctly from client. For example if you use something like `DownloadData` or any similar api which pulls whole response from server into memory - you will bloat clients memory this time for no reason. Instead copy response stream directly to file on client (`responseStream.CopyTo(localFileStream)` – Evk Mar 27 '18 at 10:11
  • @evk tnx btw I would daresay that would also be an answer to my question – Thomas Mar 27 '18 at 10:14

1 Answers1

3

Try to use PushStreamContent. It pushes the stream back to the client without need to load it in the memory.

Here's excellent article from Stephen Cleary that uses PushStreamContent, the only difference is it zips the file before sending it, so you should modify it, or even better, send them zipped version to save bandwidth.

Gvozden Kuzman
  • 74
  • 4
  • isnt pushstreamcontent mostly used for Video Streaming though? (https://stackoverflow.com/questions/16168683/webapi-streamcontent-vs-pushstreamcontent) or am I overlooking something? – Thomas Mar 27 '18 at 10:11
  • It is, but it also works in the scenario you've described. I've managed to do it flawlessly even for files bigger than 2 gbs – Gvozden Kuzman Mar 27 '18 at 10:19
  • You can use `PushStreamContent`, but why? Regular `StreamContent` can do the same job with less code. – Evk Mar 27 '18 at 10:27