AES_GCM in .net with streams

Question

In my previous question (RAM not being freed in c# after working with files) I asked about a way to clear RAM. someone suggested using streams instead of reading it into a variable. I found Encrypting/Decrypting large files (.NET) which uses streams but it is not using AesGcm. The problem is that I can't find how to use AesGcm with streams. AesGcm.decrypt only accepts Byte[] in the ciphertext field, and AesManaged doesn't have CihperMode.GCM.

Currently, decryption takes 4GB of ram when decrypting an 800MB file. How can I decrypt a file with AesGcm without filling the RAM?

Thanks.

Answer 1

I'll say that AesGcm (and probably AesCcm) in .NET don't support "streaming" mode and it seems the consensus (https://crypto.stackexchange.com/questions/51537/delayed-tag-checks-in-aes-gcm-for-streaming-data) is that you shouldn't create a streaming mode AesGcm. I'll add another reference about this https://github.com/dotnet/runtime/issues/27348 . I'm not an expert in cryptography so it isn't clear for me what are the problems about streaming an encrypted document and checking for its authentication tags only at the end.

If possible you should change the algorithm. Otherwise other solutions can be found. The Bouncycastle library supports AesGcm.

Answer 2

I am posting a non-streaming answer because I have a fairly decent low-allocation implementation of AesGcm that may satisfy your needs. You can take the ArraySegment<byte> straight into a stream and write to disk with a FileStream. Memory allocation shouldn't be more than double the file itself (obviously x2 because you loaded the file into memory and have to store the encrypted bytes.) It's also quite performant but I take no credit for that, obviously just Net5.0 enhancements.

Byte structure is straight forward if you need to use an alternative decryption mechanism.

  12 bytes       16 bytes    n bytes up to int.IntMax - 28 bytes.
[ Nonce / IV ][ Tag / MAC ][               Ciphertext           ]

Link to my Github Repo.

Usage example)

// HashKey or PassKey or Passphrase in 16/24/32 byte format.
var encryptionProvider = new AesGcmEncryptionProvider(hashKey, "ARGON2ID");

// This is an ArraySegment<byte>, this allows a defer allocation of byte[]
var encryptedData = encryptionProvider.Encrypt(_data);

// When you are ready for a byte[]
encryptedData.ToArray()

// You can also use
encryptedData.Array
// but this is a buffer and often exceeds the actual size of your bytes.
// Use conscientiously but does prevent a copy / allocation of the bytes.

// To Decrypt - same return type in case you need to serialize / decompress etc.
var decryptedData = encryptionProvider.Decrypt(encryptedData);

decrypted.ToArray() // the proper decrypted bytes of data.
decrypted.Array // the buffer used.

// Convert to a string
Encoding.UTF8.GetString(_encryptedData.ToArray())

If you see any issues let me know, happy to make changes/fix - or even better submit an issue/PR on Github so I can keep the real code up to date.

Benchmarks

// * Summary *

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.18363.1556 (1909/November2019Update/19H2)
Intel Core i7-9850H CPU 2.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=5.0.203
  [Host]     : .NET 5.0.6 (5.0.621.22011), X64 RyuJIT
  Job-ADZLQM : .NET 5.0.6 (5.0.621.22011), X64 RyuJIT
  .NET 5.0   : .NET 5.0.6 (5.0.621.22011), X64 RyuJIT

Runtime=.NET 5.0

|                  Method |        Job | IterationCount |          Mean |         Error |        StdDev |        Median | Ratio | RatioSD |     Gen 0 |    Gen 1 |    Gen 2 | Allocated |
|------------------------ |----------- |--------------- |--------------:|--------------:|--------------:|--------------:|------:|--------:|----------:|---------:|---------:|----------:|
|          Encrypt1KBytes |   .NET 5.0 |        Default |      1.512 us |     0.0298 us |     0.0398 us |      1.504 us |  1.00 |    0.00 |    0.1926 |        - |        - |      1 KB |
|          Encrypt2KBytes |   .NET 5.0 |        Default |      1.965 us |     0.0382 us |     0.0408 us |      1.951 us |  1.30 |    0.04 |    0.3548 |        - |        - |      2 KB |
|          Encrypt4kBytes |   .NET 5.0 |        Default |      2.946 us |     0.0583 us |     0.0942 us |      2.948 us |  1.96 |    0.07 |    0.6828 |        - |        - |      4 KB |
|          Encrypt8KBytes |   .NET 5.0 |        Default |      4.630 us |     0.0826 us |     0.0733 us |      4.631 us |  3.09 |    0.08 |    1.3351 |        - |        - |      8 KB |
|          Decrypt1KBytes |   .NET 5.0 |        Default |      1.234 us |     0.0247 us |     0.0338 us |      1.216 us |  0.82 |    0.03 |    0.1869 |        - |        - |      1 KB |
|          Decrypt2KBytes |   .NET 5.0 |        Default |      1.644 us |     0.0328 us |     0.0378 us |      1.630 us |  1.09 |    0.04 |    0.3510 |        - |        - |      2 KB |
|          Decrypt4kBytes |   .NET 5.0 |        Default |      2.462 us |     0.0274 us |     0.0214 us |      2.460 us |  1.64 |    0.04 |    0.6752 |        - |        - |      4 KB |
|          Decrypt8KBytes |   .NET 5.0 |        Default |      4.167 us |     0.0828 us |     0.1016 us |      4.179 us |  2.76 |    0.12 |    1.3275 |        - |        - |      8 KB |

Snapshot of the code in time.

public class AesGcmEncryptionProvider : IEncryptionProvider
{
    /// <summary>
    /// Safer way of generating random bytes.
    /// https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.rngcryptoserviceprovider?redirectedfrom=MSDN&view=net-5.0
    /// </summary>
    private readonly RNGCryptoServiceProvider _rng = new RNGCryptoServiceProvider();
    private readonly ArrayPool<byte> _pool = ArrayPool<byte>.Create();

    private readonly byte[] _key;

    public string Type { get; private set; }

    public AesGcmEncryptionProvider(byte[] key, string hashType)
    {
        if (!Constants.Aes.ValidKeySizes.Contains(key.Length)) throw new ArgumentException("Keysize is an invalid length.");
        _key = key;

        switch (_key.Length)
        {
            case 16: Type = "AES128"; break;
            case 24: Type = "AES192"; break;
            case 32: Type = "AES256"; break;
        }

        if (!string.IsNullOrWhiteSpace(hashType)) { Type = $"{hashType}-{Type}"; }
    }

    public ArraySegment<byte> Encrypt(ReadOnlyMemory<byte> data)
    {
        using var aes = new AesGcm(_key);

        // Slicing Version
        // Rented arrays sizes are minimums, not guarantees.
        // Need to perform extra work managing slices to keep the byte sizes correct but the memory allocations are lower by 200%
        var encryptedBytes = _pool.Rent(data.Length);
        var tag = _pool.Rent(AesGcm.TagByteSizes.MaxSize); // MaxSize = 16
        var nonce = _pool.Rent(AesGcm.NonceByteSizes.MaxSize); // MaxSize = 12
        _rng.GetBytes(nonce, 0, AesGcm.NonceByteSizes.MaxSize);

        aes.Encrypt(
            nonce.AsSpan().Slice(0, AesGcm.NonceByteSizes.MaxSize),
            data.Span,
            encryptedBytes.AsSpan().Slice(0, data.Length),
            tag.AsSpan().Slice(0, AesGcm.TagByteSizes.MaxSize));

        // Prefix ciphertext with nonce and tag, since they are fixed length and it will simplify decryption.
        // Our pattern: Nonce Tag Cipher
        // Other patterns people use: Nonce Cipher Tag // couldn't find a solid source.
        var encryptedData = new byte[AesGcm.NonceByteSizes.MaxSize + AesGcm.TagByteSizes.MaxSize + data.Length];
        Buffer.BlockCopy(nonce, 0, encryptedData, 0, AesGcm.NonceByteSizes.MaxSize);
        Buffer.BlockCopy(tag, 0, encryptedData, AesGcm.NonceByteSizes.MaxSize, AesGcm.TagByteSizes.MaxSize);
        Buffer.BlockCopy(encryptedBytes, 0, encryptedData, AesGcm.NonceByteSizes.MaxSize + AesGcm.TagByteSizes.MaxSize, data.Length);

        _pool.Return(encryptedBytes);
        _pool.Return(tag);
        _pool.Return(nonce);

        return encryptedData;
    }

    public async Task<MemoryStream> EncryptAsync(Stream data)
    {
        using var aes = new AesGcm(_key);

        var buffer = _pool.Rent((int)data.Length);
        var bytesRead = await data
            .ReadAsync(buffer.AsMemory(0, (int)data.Length))
            .ConfigureAwait(false);

        if (bytesRead == 0) throw new InvalidDataException();

        // Slicing Version
        // Rented arrays sizes are minimums, not guarantees.
        // Need to perform extra work managing slices to keep the byte sizes correct but the memory allocations are lower by 200%
        var encryptedBytes = _pool.Rent((int)data.Length);
        var tag = _pool.Rent(AesGcm.TagByteSizes.MaxSize); // MaxSize = 16
        var nonce = _pool.Rent(AesGcm.NonceByteSizes.MaxSize); // MaxSize = 12
        _rng.GetBytes(nonce, 0, AesGcm.NonceByteSizes.MaxSize);

        aes.Encrypt(
            nonce.AsSpan().Slice(0, AesGcm.NonceByteSizes.MaxSize),
            buffer.AsSpan().Slice(0, (int)data.Length),
            encryptedBytes.AsSpan().Slice(0, (int)data.Length),
            tag.AsSpan().Slice(0, AesGcm.TagByteSizes.MaxSize));

        // Prefix ciphertext with nonce and tag, since they are fixed length and it will simplify decryption.
        // Our pattern: Nonce Tag Cipher
        // Other patterns people use: Nonce Cipher Tag // couldn't find a solid source.
        var encryptedStream = new MemoryStream(new byte[AesGcm.NonceByteSizes.MaxSize + AesGcm.TagByteSizes.MaxSize + (int)data.Length]);
        using (var binaryWriter = new BinaryWriter(encryptedStream, Encoding.UTF8, true))
        {
            binaryWriter.Write(nonce, 0, AesGcm.NonceByteSizes.MaxSize);
            binaryWriter.Write(tag, 0, AesGcm.TagByteSizes.MaxSize);
            binaryWriter.Write(encryptedBytes, 0, (int)data.Length);
        }

        _pool.Return(buffer);
        _pool.Return(encryptedBytes);
        _pool.Return(tag);
        _pool.Return(nonce);

        encryptedStream.Seek(0, SeekOrigin.Begin);
        return encryptedStream;
    }

    public MemoryStream EncryptToStream(ReadOnlyMemory<byte> data)
    {
        return new MemoryStream(Encrypt(data).ToArray());
    }

    public ArraySegment<byte> Decrypt(ReadOnlyMemory<byte> encryptedData)
    {
        using var aes = new AesGcm(_key);

        // Slicing Version
        var nonce = encryptedData
            .Slice(0, AesGcm.NonceByteSizes.MaxSize)
            .Span;

        var tag = encryptedData
            .Slice(AesGcm.NonceByteSizes.MaxSize, AesGcm.TagByteSizes.MaxSize)
            .Span;

        var encryptedBytes = encryptedData
            .Slice(AesGcm.NonceByteSizes.MaxSize + AesGcm.TagByteSizes.MaxSize)
            .Span;

        var decryptedBytes = new byte[encryptedBytes.Length];

        aes.Decrypt(nonce, encryptedBytes, tag, decryptedBytes);

        return decryptedBytes;
    }

    public MemoryStream Decrypt(Stream stream)
    {
        using var aes = new AesGcm(_key);
        using var binaryReader = new BinaryReader(stream);

        var nonce = binaryReader.ReadBytes(AesGcm.NonceByteSizes.MaxSize);
        var tag = binaryReader.ReadBytes(AesGcm.TagByteSizes.MaxSize);
        var encryptedBytes = binaryReader.ReadBytes((int)binaryReader.BaseStream.Length - AesGcm.NonceByteSizes.MaxSize - AesGcm.TagByteSizes.MaxSize);
        var decryptedBytes = new byte[encryptedBytes.Length];

        aes.Decrypt(nonce, encryptedBytes, tag, decryptedBytes);

        return new MemoryStream(decryptedBytes);
    }

    public MemoryStream DecryptToStream(ReadOnlyMemory<byte> data)
    {
        return new MemoryStream(Decrypt(data).ToArray());
    }
}

Answer 3

It goes like this:

using var aes = new AesGcm(_key);
using FileStream fs = new(<path to file>, FileMode.Open);
int bytesRead;
while ((bytesRead = fs.Read(buffer)) > 0)
{
     aes.Encrypt(nonce, buffer[..bytesRead], buffer[..bytesRead], tag);

     using var encfs = new FileStream($@"{path to output file}.enc", FileMode.Append);
     encfs.Write(_salt);
     encfs.Write(nonce);
     encfs.Write(buffer[..bytesRead]);
     encfs.Write(tag);
}

This opens a stream to the file, streams the file via the buffer depending on the buffer size, and writes the resulting ciphers into the same file. The memory at this stage should will be taking the buffer size + something very small coming from various objects in the program that are alive at the time.

I was having issues loading for example a 3 GB file, encrypt it and have the memory be nice and clear and the code above provides just that without any issues.