1

I have large file (usually 300 MB to 2 GB) which consists of smaller files. I want to unpack them to folder and I want to do that efficiently (zero-copy).

I found, that .NET Framework have TransmitFile method for that

using (StreamWriter writer = File.CreateText(outputPath))
{
    HttpResponse response = new HttpResponse(writer);

    response.TransmitFile(inputPath, offset, length);
}

but what about .NET Core, that I'm targeting...?

IHttpSendFileFeature Interface looks promising and it is probably implemented in OwinFeatureCollection class, but how can I use it?

Matěj Pokorný
  • 16,977
  • 5
  • 39
  • 48
  • How would sending a file 'unpack' it? – Neil Jan 26 '19 at 18:23
  • Is P/Invoking an option(Windows only solution?) or you want your solution to be cross platform dependent? – Hasan Emrah Süngü Jan 26 '19 at 18:26
  • @Neil, OP wants to read and transmit it without creating copy of the data – Hasan Emrah Süngü Jan 26 '19 at 18:27
  • Why would transmitting a file make a copy of it? – Neil Jan 26 '19 at 18:28
  • Possible duplicate of [Efficiently sending files from asp.net core](https://stackoverflow.com/questions/44979668/efficiently-sending-files-from-asp-net-core) – Neil Jan 26 '19 at 18:29
  • @Neil, I do not have in depth detail but if you search for Zero-Copy then you will see that usual way the sending file is implemented such that "it will buffer the file in memory". OP does not want to. WinAPI offers a method "TransferFile" for that – Hasan Emrah Süngü Jan 26 '19 at 18:32
  • @HasanEmrahSüngü This sounds to me like 'premature optimization'. Unless it is running in exceptionally small amounts of ram (like a few 100 MB), then this really is not an issue. There are many different parts of Windows that will 'buffer' data, that you really can't control from a C#. Don't worry about it until you have a working system, that you can measure the 'issues', and THEN if it's a problem, work around it. – Neil Jan 26 '19 at 18:36
  • @Neil I want to copy part of the large file to the new one, without copying data to user-space. Look at that zero-copy reference ;-). – Matěj Pokorný Jan 26 '19 at 18:40
  • @Neil, Yeah you are right. It may very well be early optimization. However, on wikipedia page it mentions that `Zero-copy protocols are especially important for high-speed networks in which the capacity of a network link approaches or exceeds the CPU's processing capacity.` – Hasan Emrah Süngü Jan 26 '19 at 18:40
  • @HasanEmrahSüngü If P/Invoking will be necessary, I will probably stay with [Stream.CopyToAsync](https://learn.microsoft.com/en-us/dotnet/api/system.io.stream.copytoasync)... – Matěj Pokorný Jan 26 '19 at 18:43
  • You can easily use `TransmitFile` from here ;) https://learn.microsoft.com/en-ca/windows/desktop/api/mswsock/nf-mswsock-transmitfile – Hasan Emrah Süngü Jan 26 '19 at 18:45
  • To answer your inquiry about the `IHttpSendFileFeature`, something like this should work, though note it falls back to slower variations if the feature is unsupporter. `var sendFileFeature = HttpContext.Features.Get(); await sendFileFeature.SendFileAsync("filename", offset, byteCount, cancellationToken);` – pinkfloydx33 Jan 26 '19 at 20:26
  • @pinkfloydx33 Thanks, this could work! Just to figure out, how to use it properly ([first attempt](https://i.imgur.com/OiMd5Gg.png)). – Matěj Pokorný Jan 26 '19 at 20:45
  • Yah that's because the feature is null. I misunderstood and thought you were working inside of an asp application. You're not going to use the feature (at least this way) – pinkfloydx33 Jan 26 '19 at 20:57
  • Ah, okay. So it seems impossible to do zero-copy in .NET Core without running server like Kestrel... – Matěj Pokorný Jan 26 '19 at 21:11
  • 1
    I dug through the asp source... Seems like https://github.com/aspnet/AspNetCore/blob/85e2147ff037cc9950249fbb9d095ad47ec4184a/src/Servers/HttpSys/src/RequestProcessing/ResponseBody.cs does part of the work (look for SendFileAsync) but it ultimately calls out to https://github.com/aspnet/AspNetCore/blob/85e2147ff037cc9950249fbb9d095ad47ec4184a/src/Servers/HttpSys/src/RequestProcessing/ResponseBody.cs and uses `IOCompletionPorts` and `Overlapped` perhaps you can review and get some ideas. Overlapped io seems promising. – pinkfloydx33 Jan 27 '19 at 00:04
  • @HasanEmrahSüngü If you are worried about >high-speed networks, then you don't write your code in C# :-) – Neil Jan 27 '19 at 12:44
  • @Neil, I do not believd this is not a good argument. .Net or Java for that matter is not as fast as other other native languages such as c or c++ is quite old – Hasan Emrah Süngü Jan 28 '19 at 00:33
  • In general you might be correct but if you are worried about memory use and high speed network transfers then you don't use c# or java. See the many jobs for c++ on for the stock markets where microseconds mean money. – Neil Jan 28 '19 at 07:32

0 Answers0