0

I'm sent large files from an API as base64 encoded strings which I convert to a byte[] array (in one go) and then return to the client via controller action, example:

byte[] fileBytes = Convert.FromBase64String(base64File);

return this.File(fileBytes);

FileContentResult - MSDN

In some cases, these files end up being very large which I believe is causing the fileBytes object to be allocated to the LOH where it wont be immediately freed once out of scope. This is happening enough that it is causing out of memory exceptions and the application to restart.

My question is, how can I read these large base64 strings without allocating a byte[] to the LOH? I thought about reading it into a stream and then returning a FileStreamResult instead e.g:

using(var ms = new MemoryStream(fileBytes))
{
    // return stream from action
}

But I'd still need to convert the base64 to byte[] first. Is it possible to read the the base64 itself in smaller chunks, therefore creating smaller byte[] which wouldn't end up in LOH?

DGibbs
  • 14,316
  • 7
  • 44
  • 83
  • Is it safe to assume that the string data in the stream would be UTF8? – Matthew Watson Feb 23 '22 at 14:17
  • @MatthewWatson Thats correct, it would be UTF8 – DGibbs Feb 23 '22 at 14:20
  • Can you receive the base64 as a stream also? Then you can stream it straight from the API out to a `FileStream` – Charlieface Feb 23 '22 at 14:41
  • 1
    Does this answer your question? [Is there a Base64Stream for .NET?](https://stackoverflow.com/questions/2525533/is-there-a-base64stream-for-net) You need `FromBase64Transform` rather than `ToBase64Transform` – Charlieface Feb 23 '22 at 14:42
  • @Charlieface No it doesn't answer my question it still allocates the `byte[]` and it's concerned with _outputting_ base64, I already have the string, I need to turn it into a stream without allocating large blocks of memory. Secondly, no I can't recieve it as a stream. – DGibbs Feb 23 '22 at 14:52
  • 1
    You don't need to output it to `byte[]`, you can pass the `CryptoStream` directly to `FileStreamResult`. As I said, you can switch it to `FromBase64Transform` for readong rather than writing Base64. And if you already have it as a string then you anyway have LOH problems: the string will be 33% larger than the `byte[]` array. You *need* to fully stream this to avoid allocations. Use `StreamReader` and `CryptoStream` and pass the whole thing through without any copying – Charlieface Feb 23 '22 at 14:56
  • 1
    Not sure if there's something like that built into C#, but you could create your own `Stream` that takes your string as parameter and implements the read methods. Should be relatively easy using `var size = Encoding.UTF8.GetBytes(input.AsSpan(offset, count), buffer);` to convert chunks. – NotFound Feb 23 '22 at 15:06
  • @Charlieface I must be missing something; I don't see how it's possible to create a `CryptoStream` directly from a base64 string? Do you have an example? The post you linked demonstrates creating one via stream which itself is created by allocating the `byte[]`. – DGibbs Feb 23 '22 at 15:09
  • Where are you getting the string from, you aren't showing that? If you want to avoid LOH allocations, you are also going to need to receive the Base64 from your API as a `Stream`, not a `string`. Then you can do `new CryptoStream(yourApiStream, new FromBase64Transform(), CryptoStreamMode.Read)` and pass that to `FileStreamResult`. If you show where you get `base64File` from perhaps i can write a proper answer – Charlieface Feb 23 '22 at 15:16
  • @Charlieface Yes that would be the ideal scenario, unfortunately we recieve the base64 from an API which we have no control over, it comes down as a large JSON object with one of the fields being the file base64. – DGibbs Feb 23 '22 at 15:20
  • Then you aren't going to avoid a LOH allocation for the string. You can avoid the `byte[]` allocation at least, by using [this answer](https://stackoverflow.com/a/57100948/14868997) to create a stream over the string, and returning `new FileStreamResult(new CryptoStream(yourStringStream, new FromBase64Transform(), CryptoStreamMode.Read)` – Charlieface Feb 23 '22 at 15:29
  • @Charlieface Thank you. I'm aware that the string will still likely be added to the LOH but unfortunately there isn't a lot we can do to address that, but we can look at the `byte[]`. – DGibbs Feb 23 '22 at 15:40
  • 1
    Make sure you use that answer for string to stream, and not any other, as the others use `Encoding.GetBytes` which also causes allocation – Charlieface Feb 23 '22 at 15:41
  • @Charlieface There's a complete `StringStream` implementation here: https://github.com/koszeggy/KGySoft.CoreLibraries/blob/master/KGySoft.CoreLibraries/IO/StringStream.cs unfortunately though, this doesn't seem to work. It seems to think the base64 is provided to it is invalid (I've validated the string using online tools). `The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.` – DGibbs Feb 23 '22 at 16:09

0 Answers0