Write large data into MemoryStream with C#

Question

I have a big stream (4Go), I need to replace some character (I need to replace one specific character with 2 or 3 ones) in that stream, i get the stream from à service à.d I have to return back a stream. This is what I'm doing

private static Stream UpdateStream(Stream stream, string oldCharacters, string newCharacters, int size = 2048)
{
    stream.Position = 0;

    StreamReader reader = new StreamReader(stream);

    MemoryStream outputStream = new MemoryStream();
    StreamWriter writer = new StreamWriter(outputStream);
    writer.AutoFlush = true;

    char[] buffer = new char[size];

    while (!reader.EndOfStream)
    {
        reader.Read(buffer, 0, buffer.Length);
        if (buffer != null)
        {
            string line = new string(buffer);
            if (!string.IsNullOrEmpty(line))
            {
                  string newLine = line.Replace(oldCharacters, newCharacters);
                writer.Write(newLine);
            }
        }
    }

    return outputStream;
}

But I'm getting an OutOfMemory exception at some point in this line but when looking at computer memery I still have planty available.

writer.Write(newLine);

Any advise ?

Have you tried streaming it to another file instead of a MemoryStream — jbriggs, Apr 02 '21 at 19:54
looks like you should look for `MemoryMappedFile` to process large files. — King King, Apr 02 '21 at 20:13
Completely unrelated to memory issues, this code has a fairly nasty bug. What happens when your `Read` from the input stream bisects a match that should be replaced? Only search strings of length 1 are safe. — Lee Hiles, Feb 04 '22 at 06:02

score 1 · Answer 1 · answered Apr 02 '21 at 21:26

This is not an answer, but I couldn't possibly fit it into a comment.

Currently your problem is not solvable without making some assumptions. The problem, as I hear it, is that you want to replace some parts of a large body of text saved in a file and save the modified text in the file again.

Some unknown variables:

How long are those strings you are replacing?
How long are those strings you are replacing it with? The same length as the replaced strings?
What kinds of strings are you looking to replace? A single word? A whole sentence? A whole paragraph?

A solution to your problem would be to read the file into memory in chunks, replace the necessary text and save the "updated" text in a new file and then finally rename the "new file" to the name of the old file. However, without knowing the answers to the above points, you could potentially be wanting to replace a string as long as all text in the file (unlikely, yes). This means in order to do the "replacing" I would have to read the whole file into memory before I can replace any of the text, which causes an OutOfMemoryException. (Yes, you could do some clever scanning to replace such large strings without reading it all into memory at once, but I doubt such a solution is necessary here).

Please edit your question to address the above points.

Thanks, I updated the post: I get a stream and have to return a stream, I'm replacing 1 character by 2 or 3 ones. — Khaled, Apr 02 '21 at 22:56

score 1 · Accepted Answer · answered Apr 03 '21 at 12:21

1

So to make it work I had to :

use the HugeMemoryStream class from this post Failed to write large amount of data to stream
and define the gcAllowVeryLargeObjects parameter to true

and set the build to 64 bit (Prefer 32-bit unchecked)

answered Apr 03 '21 at 12:21

Khaled

317
2
7

Is this the solution you want to go with? It's quite hacky and not very resource-friendly as it still requires a lot of available memory to do a very tiny operation. Meaning: you'll still get an `OutOfMemoryException` if the text is large enough or your application just doesn't have access to enough memory. I'd advice you to avoid keeping it all in memory at once. – Xerillio Apr 04 '21 at 13:31
That what worked so far and didn't find better. If you have any other approach I'll be happy to get it. The only thing is that : I receive à 4Go stream, and I need to return a stream after modifing some characters. – Khaled Apr 04 '21 at 19:06
I'd suggest you look at the accepted answer to the question you linked. You should avoid keeping all the data in memory. If you're working with files, use one file stream as input and another as output. Don't keep it in a memory stream. – Xerillio Apr 04 '21 at 20:45
Thanks, but the accepted answer of the link I posted (writing to a FileStream) doesn't help since I'm in a middleware that needs to get Stream as an input and return a modified Stream as an output and has no access to disk. – Khaled Apr 05 '21 at 07:57

Write large data into MemoryStream with C#

2 Answers2