Out of memory error archiving a log file

Question

I am having a problem with a console job that runs and creates a daily log file that I archive at midnight.

This creates a blank log file for the next day and an archived file with yesterdays date in the name and the contents of the old file for debugging issues I may have had and not known about until the day after.

However since I cranked up the BOT's job I have been hitting issues with System Out of Memory errors when I try and archive the file.

At first I was just not able to get an archived file at all then I worked out a way to get at least the last 100,000 lines which is not nearly enough.

I wrap everything in 3 try/catches

1. I/O
2. System out of memory
3. standard exception

However it's always the OutOfMemoryException that I get e.g

System.OutOfMemoryException Error: Exception of type 'System.OutOfMemoryException' was thrown.;

To give you an example of size 100,000 lines of log is about 11MB file

A standard full log file can be anything from 1/2 a GB to 2GB

What I need to know is this:

a) what size of a standard text file will throw an out of memory error when trying to use File.ReadAllText or a custom StreamReader function I call ReadFileString e.g

public static string ReadFileString(string path)
{
    // Use StreamReader to consume the entire text file.
using (StreamReader reader = new StreamReader(path))
{
    return reader.ReadToEnd();
    }
}

b) is it my computers memory (I have 16GB RAM - 8GB use at time of copying) or the objects I am using in C# that are failing with the opening and copying of files.

When archiving I first try with my custom ReadFileString function (see above), if that returns 0 bytes of content I try File.ReadAllText and then if that fails I try a custom function to get the last 100,000 lines, which is really not enough for debugging errors earlier in the day.

The log file starts at midnight when a new one is created and logs all day. I never used to have out of memory errors but since I have turned up the frequency of method calls the logging has expanded which means the file sizes have as well.

This is my custom function for getting the last 100,000 lines. I am wondering how many lines I could get without IT throwing an out of memory error and me not getting any contents of the last days log file at all.

What do people suggest for the maximum file size for various methods / memory needed to hold X lines, and what is the best method for obtaining as much of the log file as possible?

E.G some way of looping line by line until an exception is hit and then saving what I have.

This is my GetHundredThousandLines method and it logs to a very small debug file so I can see what errors happened during the archive process.

private bool GetHundredThousandLines(string logpath, string archivepath)
{
    bool success = false;

    int numberOfLines = 100000;


    if (!File.Exists(logpath))
    {
    this.LogDebug("GetHundredThousandLines - Cannot find path " + logpath + " to archive " + numberOfLines.ToString() + " lines");
    return false;
    }

    var queue = new Queue<string>(numberOfLines);

    using (FileStream fs = File.Open(logpath, FileMode.Open, FileAccess.Read, FileShare.Read))
    using (BufferedStream bs = new BufferedStream(fs))  // May not make much difference.
    using (StreamReader sr = new StreamReader(bs))
    {
    while (!sr.EndOfStream)
    {
        if (queue.Count == numberOfLines)
        {
        queue.Dequeue();
        }

        queue.Enqueue(sr.ReadLine() + "\r\n");
    }
    }

    // The queue now has our set of lines. So print to console, save to another file, etc.
    try
    {

    do
    {        
        File.AppendAllText(archivepath, queue.Dequeue(), Encoding.UTF8);
    } while (queue.Count > 0);


    }
    catch (IOException exception)
    {
    this.LogDebug("GetHundredThousandLines - I/O Error accessing daily log file with ReadFileString: " + exception.Message.ToString());
    }
    catch (System.OutOfMemoryException exception)
    {
    this.LogDebug("GetHundredThousandLines - Out of Memory Error accessing daily log file with ReadFileString: " + exception.Message.ToString());
    }
    catch (Exception exception)
    {
    this.LogDebug("GetHundredThousandLines - Exception accessing daily log file with ReadFileString: " + exception.Message.ToString());
    }


    if (File.Exists(archivepath))
    {
    this.LogDebug("GetHundredThousandLines - Log file exists at " + archivepath);
    success = true;
    }
    else
    {
    this.LogDebug("GetHundredThousandLines - Log file DOES NOT exist at " + archivepath);
    }

    return success;

}

Any help would be much appreciated.

Thanks

Answer 1

try: keep the queue and stream position in class scope, try GC.Collect() when getting out of memory exception and call function again. seek stream to last position and continue. or: use one database like sqlite and keep newest 100000 record in each table.