Application Maxing Memory

Question

I have an application that reads a file into memory. It then does a number of checks of this data (checks run concurrently in the background on multiple tasks) before presenting the results to the user.

Although the application doesn't crash due to OutOfMemory exception, I notice that even though the processing is happening in the background, the UI will hitch up and the window will suddenly be "Not Responding". The application will eventually complete the task and display the results but in the meantime the application looks broken and would prompt users to close the window.

After much Googling I'm unsure how to handle this. Is there something I should be doing in order to make sure the the app doesn't become unresponsive? Should I attempt to calculate the estimated memory availability vs usage on startup and alert the user if the machine doesn't have enough RAM? Should I monitor memory usage and do something if available memory drops below a certain amount?

Just to be clear, all of the processing is happening in the background. I'm using Task.Run. I think I can tell that the UI thread isn't being blocked by processing as when I run on a machine with enough memory the UI doesn't hitch.

Snippet

private void Window_Loaded(object sender, RoutedEventArgs e)
{
    TaskList();
}

---

private async Task TaskList()
{
var taskList = new List<Task>();
txbStatus.Text += "Processing...\n";

taskList.Add(Task.Run(async () =>
{
        await GetFileLists();

    Application.Current.Dispatcher.Invoke(() => { txbStatus.Text += "Get file list completed\n"; });
}));

await Task.WhenAll(taskList.ToArray());
}

---

private async Task GetFileLists()
{
    var taskList = new List<Task>();

    if (!string.IsNullOrEmpty(TextDirPath))
        taskList.Add(Task.Run(() =>
        {
            _textFilesInFolderPathList =
                FileList.GetFileList(TextDirPath);
        }));
    if (!string.IsNullOrEmpty(ImagesDirPath))
        taskList.Add(Task.Run(() =>
        {
            _imageFilesInFolderPathList =
                FileList.GetFileList(ImagesDirPath));
        }));

    if (!string.IsNullOrEmpty(NativesDirPath))
        taskList.Add(Task.Run(() =>
        {
            _nativesFilesInFolderPathList =
                FileList.GetFileList(NativesDirPath);
        }));

    await Task.WhenAll(taskList.ToArray());
}

Screen Shots

Answer 1

1) You have some IO-intensive processes that access the disk(s).
2) Your UI becomes non-responsive during this phase.
3) Your UI has another requirement, to show an animated image while those processes are running.
Update:
4) Tests states that the primary problem is the lack of physical memory, which forces the System to swap to page file, while a new sequential IO-bound quasi-concurrent request is made.
This causes stuttering.

I'm proposing the following method to detach the UI from the IO processes that renders it non-responsive.
Test it first as it is, then running your actual Tasks.
Update:
In the current context, Async methods alone can't solve the problem.
The UI stutters because the System does.

Thus, new proposed solution:
Access the large files using a segment (or segments) of virtual memory.
This is achieved using a MemoryMappedFile, which correlates the file and the memory space, letting the mapped portion(s) be treated as primary memory. File Read and Write operations are performed in the same way, but through a MemoryMappedViewAccessor (random access) or a MemoryMappedViewStream (sequential access). A Write operation is performed when the related Accessor is Flushed()
An Accessor can be shared. Processes can access the same Accessor without concurrency.

---

Class MyTaskResults is used to store results and to pass parameters to the async method responsible for the IO-bound processes.

Edit 1:
The returned value must be a List<string> (?)
(Why using .FirstOrDefault() on the returned List)?

Edit 3:
Added two shared MemoryMappedFile objects to the main class.

public class MyTaskResults
{
    public int TaskID { get; set; }
    public string TextDirPath { get; set; }
    public string ImagesDirPath { get; set; }
    public string NativesDirPath { get; set; }

    public List<string> TextDirPathResult { get; set; }
    public List<string> ImagesDirPathResult { get; set; }
    public List<string> NativesDirPathResult { get; set; }
}

//List of MyTaskResults Class, used to store all Tasks run and their results
List<MyTaskResults> _ListOfTasks = new List<MyTaskResults>();

private static MemoryMappedFile _mmfDatData; 
private static MemoryMappedFile _mmfOptData;

bool CriticalJobRunning = false;
int _TasksCounter = 0;

You can run TaskRunProxy() from any other method, which doesn't need to be an async one.

Edit2:
Moved the call to TaskRunProxy() in MainWindow.Loaded() event handler

private void wMain_Loaded(object sender, RoutedEventArgs e) { TaskRunProxy(); }

Added search directories for TextDirPath, ImagesDirPath, NativesDirPath.
Results:

TextFiles: 765 (*.txt) - ImageFiles: 697 (*.jpg) - NativeFiles: 28422 (*.dll)

Elapsed Time: 88428ms
Initial: Page File Size: 4096 Memory (Working Set): 60.907.520
Final: Page File Size: 4096 Memory (Working Set): 91.385.856

Total: 29884 files found that matched the supplied patterns.
Total Memory: 30.478.336 (~29Mb)

The UI didn't even notice it.
=> With a maxed out memory, the System may be swapping heavily. A test ought to be performed after zeroing-rebooting-rebuilding the system page file (and general clean-up/defrag).

Edit3:
Create 2 Memory Mapped Files correlating the large disk files to the Virtual Memory space:

public MainWindow()
{
    InitializeComponent();

    string _datFilePath = @"PATHTOLARGEFILE";//~200MB
    string _optFilePath = @"PATHTOLARGEFILE2";//~200MB

    Int64 _sizeDatData = new FileInfo(_datFilePath).Length;
    Int64 _sizeOptData = new FileInfo(_optFilePath).Length;

    //Capacity = 0 means a capacity equal to the full size of the file on disk. 
    //Or _sizeDatData and _sizeOptData can be used.
    _mmfDatData = MemoryMappedFile.CreateFromFile(_datFilePath, 
                                    FileMode.Open, 
                                    "DatData", 0, 
                                    MemoryMappedFileAccess.ReadWrite);
    _mmfOptData = MemoryMappedFile.CreateFromFile(_optFilePath, 
                                    FileMode.Open, 
                                    "OptData", 0, 
                                    MemoryMappedFileAccess.ReadWrite);
}

Read and write operations: (Random access)
[TYPE] can be a Reference Type or a Value Type (structs are of course included)
The example uses the first 128 MB for Read/Write operations.

    MemoryMappedViewAccessor _viewOptData = _mmfOptData.CreateViewAccessor(
                             0, 
                             0x8000000L, 
                             MemoryMappedFileAccess.ReadWrite);

    _viewOptData.Read<[TYPE]>([Position], out [TYPE]);
    _viewOptData.Write<[TYPE]>([Position], ref [TYPE]);

Run the IO-bound Tasks after the UI is presented.

private void wMain_Loaded(object sender, RoutedEventArgs e)
{
    TaskRunProxy();
}

private async void TaskRunProxy()
{
    _TasksCounter += 1;

    MyTaskResults _Task = new MyTaskResults
    {
        TaskID = _TasksCounter,
        TextDirPath = Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),
        ImagesDirPath = Environment.GetFolderPath(Environment.SpecialFolder.MyPictures),
        NativesDirPath = Environment.GetFolderPath(Environment.SpecialFolder.Windows)
    };

        Console.WriteLine("Page File Size: " + Environment.SystemPageSize.ToString());
        Console.WriteLine("Memory (Working Set): " + Environment.WorkingSet.ToString());

        Stopwatch _SW = new Stopwatch();
        _SW.Start();
        CriticalJobRunning = true;
        _ListOfTasks.Add(await GetFileListsAsync(_Task));
        CriticalJobRunning = false;
        _SW.Stop();

        Console.WriteLine("Time: " + _SW.ElapsedMilliseconds + Environment.NewLine);

        Console.WriteLine("TextFiles: " +  _Task.TextDirPathResult.Count + 
                        "  ImageFiles: " + _Task.ImagesDirPathResult.Count + 
                        "  NativeFiles: " + _Task.NativesDirPathResult.Count);

        Console.WriteLine("Page File Size: " + Environment.SystemPageSize.ToString());
        Console.WriteLine("Memory (Working Set): " + Environment.WorkingSet.ToString());
}

private void wMain_Closing(object sender, System.ComponentModel.CancelEventArgs e)
{
    if (this.CriticalJobRunning)
        e.Cancel = true;
    //Let the use know
}

The async method used to run the (probably synchronous) Tasks:

Edit1:
Return value changed to List<string>
Edit2:
Changed dummy Thread.Sleep(x) with "real-life" file enumeration + memory loading:

Directory.GetFiles(_Task.[PATH], "[PATTERN]", SearchOption.AllDirectories).ToList<string>();

private async Task<MyTaskResults> GetFileListsAsync(MyTaskResults _Task)
{
    if (!string.IsNullOrEmpty(_Task.TextDirPath))
        _Task.TextDirPathResult = await Task.Run(() =>
        {
            return Directory.GetFiles(_Task.TextDirPath, 
                                     "*.txt", 
                                      SearchOption.AllDirectories).ToList<string>();
            //Thread.Sleep(4000);
            //return new List<string> {TextDirPathResult Completed"};
            //return FileList.GetFileList(_Task.TextDirPath);
        });

    if (!string.IsNullOrEmpty(_Task.ImagesDirPath))
        _Task.ImagesDirPathResult = await Task.Run(() =>
        {
            return Directory.GetFiles(_Task.ImagesDirPath, 
                                     "*.jpg", 
                                      SearchOption.AllDirectories).ToList<string>();
            //Thread.Sleep(3000);
            //return new List<string> {"TextDirPathResult Completed"};
            //return FileList.GetFileList(_Task.ImagesDirPath);
        });

    if (!string.IsNullOrEmpty(_Task.NativesDirPath))
        _Task.NativesDirPathResult = await Task.Run(() =>
        {
            return Directory.GetFiles(_Task.NativesDirPath, 
                                     "*.dll", 
                                      SearchOption.AllDirectories).ToList<string>();
            //Thread.Sleep(3000);
            //return new List<string> {"TextDirPathResult Completed"};
            //return FileList.GetFileList(_Task.NativesDirPath);
        });

    return _Task;
}