IndexOutOfRangeException on decoding base64 bytes with LF characters included (>= .NET 5 only)

Question

I have upgraded runtime from netcoreapp3.1 to NET 5 and the code which was transforming base64 bytes containing LF characters to string without errors started to crash with IndexOutOfRangeException. Behavior is the same for Windows and Linux platforms.

I've already submitted a bug report but decided to ask anyway if there is something wrong or bug-prone with this code.

For now, a single workaround I can think of is to add a middleware stream, which will remove all LF characters from the input, because whitespaces are excessive in base64 anyway. It is worth mentioning that exception is not happening with CRLF delimiter.

[TestFixture]
public class Fixture
{
    [Test]
    public void Crashes_on_runtime_greater_or_equal_to_NET_5()
    {
        var txt = "YWJj\nZGVm"; // abc\ndef
        var base64Bytes = Encoding.UTF8.GetBytes(txt);
        var stream = new MemoryStream(base64Bytes);
        var base64Transform = new FromBase64Transform();
        var cryptoStream = new CryptoStream(stream, base64Transform, CryptoStreamMode.Read);

        var result = new MemoryStream();
        cryptoStream.CopyTo(result);

        Console.WriteLine(Encoding.UTF8.GetString(result.ToArray()));
    }
}

System.IndexOutOfRangeException : Index was outside the bounds of the array.
   at System.Security.Cryptography.FromBase64Transform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken, Boolean useAsync)
   at System.Security.Cryptography.CryptoStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Security.Cryptography.CryptoStream.CopyTo(Stream destination, Int32 bufferSize)
   at System.IO.Stream.CopyTo(Stream destination)
   at ClassLibrary1.Fixture.Crashes_on_runtime_greater_or_equal_to_NET_5() in E:\cm_1\drive\ClassLibrary1\Class1.cs:line 20

This is wrong!!!! You are missing base64Str = Convert.ToBase64String(base64Bytes) — jdweng, Oct 12 '22 at 09:54
@CodeCaster I think they are genuinely trying to help, but it's staggering how bad that help is. — DavidG, Oct 12 '22 at 09:56
This example is simplified for the purpose of showing minimal repro. In real service I have base64 input of a lot of bytes, being read from a stream, so, I can not just convert them in place. I have a high-loaded service, so the memory will just die if I won't decode base64 in a streaming manner. — idementia, Oct 12 '22 at 09:59
Yeah, the problem is I do not control producing side, so the only option is to modify the input in a way that it would not contain LF delimiters. Or to wait for some runtime patch or any other more simple workaround. — idementia, Oct 12 '22 at 11:26

score 1 · Accepted Answer · edited Feb 14 '23 at 08:12

CryptoStream calls:

int TransformBlock(byte[] inputBuffer, int inputOffset, int inputCount, ...)

With the input buffer, offset 0, count 8 (two whole blocks have been read, 2*4).

The FromBase64Transform keeps some internal state and returns 6 (six bytes of output have been written), then receives a final call:

byte[] TransformFinalBlock(byte[] inputBuffer, int inputOffset, int inputCount)

With ('m\0\0\0', 0, 1). This triggers the bug in FromBase64Transform, specifically in the aggressively inlined method:

private static int GetOutputSize(int bytesToTransform, Span<byte> tmpBuffer)
{
    // ...
    if (tmpBuffer[len - 2] == padding)
}

The buffer has a length of 1, going -2 on that isn't appreciated by the runtime.

Either:

Pad your input to be of length % 4 (including whitespace) with extraneous = characters
Alter your input using a stream wrapper
Wait for a bugfix

The first option looks simpler than the second one. Will probably use it, thx — idementia, Oct 12 '22 at 12:21

IndexOutOfRangeException on decoding base64 bytes with LF characters included (>= .NET 5 only)

1 Answers1