How create big stream .Net (like 2-5GB) using only memory

Question

I know few trick how you create file that can be as big as you want and then use File Stream to get quite large Streams, if I would use Memory Stream after 2GB it can't index that and it fails. Maybe there is more clever ways to create Streams in .Net like 2-5GB using only memory.

Tried: MemoryStream (Can handle like 2GB stream), Creating file (Limitation, that I can not create them in that particular environment)

Expecting: Create Stream that is capable to contain up to 5GB in memory (RAM in this case is not a problem) and be able to pass to other service that will use Stream Reader (for example).

`MemoryStream (Can handle like 1GB stream)` - never heard of that claim, are you positive this is correct? — Rand Random, Aug 08 '23 at 07:32
MemoryStream should be limited to about 2GB by default, since that is the limit of a byte-array. But there is nothing preventing you from creating your own memory stream that splits data between multiple arrays. But why specifically do you need such large memory streams? — JonasH, Aug 08 '23 at 07:41
This is not CLS-compliant but perhaps [`UnmanagedMemoryStream`](https://learn.microsoft.com/en-us/dotnet/api/system.io.unmanagedmemorystream?view=net-7.0) is an option? — Sweeper, Aug 08 '23 at 07:52
Why not save the file on a blob storage on the network if you cant store it locally? From where do you want to fill the Stream (what is the source)? — Magnus, Aug 08 '23 at 08:06
What is your goal? Why do you want to have a stream that holds gigabytes of memory? But you could like create a stream implementation that holds its data in multiple arrays? — CodeCaster, Aug 08 '23 at 08:43

score 1 · Answer 1 · answered Aug 08 '23 at 08:35

If you are fine with using non-CLS-compliant APIs and unsafe code, you can use a UnmanagedMemoryStream, which is also a subclass of Stream.

Here is an example of creating a stream containing 5GB.

unsafe
{
    const long fiveGB = 1024L * 1024L * 1024L * 5L;
    IntPtr memIntPtr = Marshal.AllocHGlobal(
        new nint(fiveGB) // assuming this is on a 64-bit system, this will not overflow
    );
    byte* memBytePtr = (byte*)memIntPtr.ToPointer();
    using (var writeStream = new UnmanagedMemoryStream(memBytePtr, fiveGB, fiveGB, FileAccess.ReadWrite))
    {
        // as an example, I'm writing the same byte to the entire block of memory
        // you can see the memory usage reach 5GB in Task Manager
        byte[] buffer = new byte[8];
        Array.Fill<byte>(buffer, 0x77);
        writeStream.Position = 0;
        while (writeStream.Position < writeStream.Length)
        {
            writeStream.Write(buffer);
        }
    }
    Marshal.FreeHGlobal(memIntPtr);
}

score 0 · Answer 2 · answered Aug 08 '23 at 07:47

0

You could implement your own Stream with long length and an Infinite IEnumerable data source yielding random numbers. No need to allocate a buffer in that case but the stream will be forward only.

answered Aug 08 '23 at 07:47

d-markey

163
1
7

Matthew Watson · Answer 3 · 2023-08-08T08:59:06.953

If you want to avoid using unsafe code (which is required for a Marshal.AllocHGlobal() solution) you can use a MemoryMappedFile.

(This assumes that you can use a non-CLS-compliant class.)

When doing this you will need to create a MemoryMappedFile of the right size and a MemoryMappedViewStream from that MMF. Because you will need to remember to Dispose() both those objects, it's probably safest to wrap them both in a class along these lines:

public sealed class LargeMemoryStream : IDisposable
{
    public LargeMemoryStream(long size)
    {
        _mmf  = MemoryMappedFile.CreateNew(mapName: null, size);
        _view = _mmf.CreateViewStream();
    }

    public void Dispose()
    {
        _view.Dispose();
        _mmf.Dispose();
    }

    public Stream Stream => _view;

    readonly MemoryMappedFile       _mmf;
    readonly MemoryMappedViewStream _view;
}

Then you can create a LargeMemoryStream, and dispose it when done. Sample code:

public static class Program
{
    const long ONE_GB = 1024L * 1024L * 1024L;

    public static void Main()
    {
        const long SIZE = 5L * ONE_GB;
    
        using var bigStream = new LargeMemoryStream(SIZE);

        testLargeStream(bigStream.Stream);
    }

    static void testLargeStream(Stream stream)
    {
        var oneGbBuffer = new byte[ONE_GB];

        for (int i = 0; i < 5; i++)
        {
            Console.WriteLine($"Writing block {i}");

            Array.Fill(oneGbBuffer, (byte)i);
            stream.Write(oneGbBuffer, 0, oneGbBuffer.Length);
        }

        stream.Position = 0;

        for (int i = 0; i < 5; i++)
        {
            Console.WriteLine($"Block {i} start = {stream.ReadByte()}");

            stream.Position += ONE_GB - 2; // -2 because previous read already advanced by 1.

            Console.WriteLine($"Block {i} end = {stream.ReadByte()}");
        }

        Console.WriteLine($"After reading all data ReadByte() returns {stream.ReadByte()}");
    }
}

Output:

Writing block 0
Writing block 1
Writing block 2
Writing block 3
Writing block 4
Block 0 start = 0
Block 0 end = 0
Block 1 start = 1
Block 1 end = 1
Block 2 start = 2
Block 2 end = 2
Block 3 start = 3
Block 3 end = 3
Block 4 start = 4
Block 4 end = 4
After reading all data ReadByte() returns -1

How create big stream .Net (like 2-5GB) using only memory

3 Answers3