3

Background:

I'm using an API that for a certain call returns a System.IO.Stream. The content is humongous and cannot be contained in RAM, and needs to be persisted. The size ranges from a couple of hundred MiB to a few GiB. The api is not something I can access other than the call I am making, but it has to be used. I cannot increase the amount of memory available to the program or switch to 64 bit.

What I've tried:

Simply doing

var output = api.getStream();
System.IO.StreamWriter streamWriter = new System.IO.StreamWriter("C:\\temp\\myfile.extension");
streamWriter.Write(output);

fails not because I'm writing an arbitrary binary as a string, but rather due to that when the first line is run (where output is assigned) the program crashes with a System.InsufficientMemoryException.

When trying the approaches outlined in the most upvoted answers to this post: How do I save a stream to a file in C#? I also get an out of memory error once the line where I get the stream from the api is called.

I've also tried approaches such as api.getStream.CopyTo(myFileStream) as suggested in an answer to this post: Save and load MemoryStream to/from a file , but still get a system out of memory exception.

The resulting question:

Does anyone know how to write an arbitrarily large IO stream to file in C# when the stream is supplied from an api-call to an external library?

Community
  • 1
  • 1
simonra
  • 245
  • 2
  • 8
  • 5
    If API you're calling (`getStream()`) fails because it exhausts all available memory then there isn't much you can do in your own code. I don't know its prototype but I guess it's returning a `byte[]`, in this case it **must** be changed to - let's say - `IEnumerable` or `IEnumerable` - or provide a callback function to process data chunks, no workarounds. Oh, well, yes you _may_ call it in another process and access shared memory to read it chunk by chunk but this will just move your issue little bit forward, it won't solve it. – Adriano Repetti Mar 30 '16 at 07:37
  • 2
    It sounds like this is more of a bug in the underlying getStream API but you've haven't told us what that is so any answers are going to be speculative at best. – Mike Zboray Mar 30 '16 at 07:49

0 Answers0