How to read text file without locking the file on disk?

Question

Basically I have some code like this that reads the contents of ASCII text file into a List:

    List<string> lines = new List<string> ( );
    using ( StreamReader sr = File.OpenText ( textfile ) )
    {
        string s = String.Empty;
        while ( ( s = sr.ReadLine ( ) ) != null )
            lines.Add ( s );
    }

But the thing is when the file is being written by another thread, it throws an exception:

The process cannot access the file 'myfile.txt' because it is being used by another process.

The same thing happens for File.ReadAllLines. Why do these functions lock the file on disk or care that the file is being use by another process?

I just want to read the contents periodically so if it's being used this time, then the next time it will read the updated content. I do this so check if new entries have been added as the user can also add them manually.

What functions I can use to read this file into memory without throwing an exception, or should I use run this code inside try/catch.

This is the latest code:

        var fs = new FileStream ( filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite );
        using ( StreamReader sr = new StreamReader ( fs ) )
        {
            string s = String.Empty;
            while ( ( s = sr.ReadLine ( ) ) != null )
                lines.Add ( s );
        }

The code that modifies the file:

public static void RemoveCoinFromBuyOrderLogs ( string symbol )
{
    if ( !walletLocked )
    {
        walletLocked = true;

        string [ ] lines = File.ReadAllLines ( walletFilename );

        var newlines = lines.Where ( c => !c.StartsWith ( symbol + "USDT" ) && !c.StartsWith ( symbol + "BUSD" ) && !c.StartsWith ( symbol + "USDC" ) && !c.StartsWith ( symbol + "TUSD" ) ).Select ( c => c ).ToList ( );
        File.WriteAllLines ( walletFilename, newlines );

        using ( FileStream fs = File.Open ( walletFilename, FileMode.OpenOrCreate ) )
        {
            StreamWriter sw = new StreamWriter ( fs );
            sw.AutoFlush = true;
            newlines.ForEach ( r => sw.WriteLine ( r ) );
        }

        walletLocked = false;
    }
}

public static void AddCoinToOrderLogs ( string newOrder, long orderId )
{
    if ( !walletLocked )
    {
        var lines = Utility.ReadAllLines ( walletFilename );
        lines = lines.Select ( line => line.Replace ( "\r", "" ) ).ToList ( );
        lines = lines.Where ( line => line != "" ).Select ( line => line ).ToList ( );

        var fields = lines.Select ( line => line.Split ( '\t' ) ).ToList ( );

        bool duplicate = false;
        foreach ( var field in fields )
        {
            if ( field.Length >= 5 )
            {
                long id = Convert.ToInt64 ( field [ 4 ] );
                if ( id == orderId )
                    duplicate = true;
            }
        }

        if ( !duplicate )
        {
            lines.Add ( newOrder );
            lines.Sort ( );

            walletLocked = true;
            File.WriteAllLines ( walletFilename, lines );
            walletLocked = false;
        }
    }
}

Answer 1

Take a look at this overload of File.Open(). It allows you to specify additional parameters to avoid the locking. I think it should do the trick.

For example, you can do var stream = new FileStream(textfile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);

Answer 2

First, if your application is multi-threaded, you shouldn't be using a bool guard. You should be using thread synchronization tools such as locks, mutexes, events and/or semaphores.

Also, your read is opening for share, but your writes aren't.

You are also not wrapping streams in using blocks. This is another problem. You should never do this:

StreamWriter sw = new StreamWriter(fs);

You should do this:

using(var sw = new StreamWriter(fs))
{
    // ...
}

The cardinal rule with objects that implement Dispose is you should always wrap them in a using block.

That aside, you probably don't want to read while writing or write while reading. That is going to give you massive problems with race conditions that will be hard to recreate when you need to debug what is going on.

Since you aren't using async/await, I would suggest using a lock. This will only allow one thread at a time to do file operations. No race conditions, no "sharing" files.

private static readonly object _fileLock = new object();

public static void RemoveCoinFromBuyOrderLogs(string symbol)
{
    lock(_fileLock)
    {
        var newlines = File.ReadAllLines(walletFilename)
            .Where(c =>
                !c.StartsWith(symbol + "USDT") &&
                !c.StartsWith(symbol + "BUSD") &&
                !c.StartsWith(symbol + "USDC") &&
                !c.StartsWith(symbol + "TUSD"));

        File.WriteAllLines(walletFilename, newlines);
    }
}

public static void AddCoinToOrderLogs(string newOrder, long orderId)
{
    lock (_fileLock)
    {
        var lines = File.ReadAllLines(walletFilename).ToList();
        lines = lines.Select(line => line.Replace("\r", "")).ToList();
        lines = lines.Where(line => line != "").Select(line => line).ToList();

        var fields = lines.Select(line => line.Split('\t')).ToList();

        bool duplicate = false;
        foreach (var field in fields)
        {
            if (field.Length >= 5)
            {
                long id = Convert.ToInt64(field[4]);
                if (id == orderId)
                    duplicate = true;
            }
        }

        if (!duplicate)
        {
            lines.Add(newOrder);
            lines.Sort();
            File.WriteAllLines(walletFilename, lines);
        }
    }
}

I can't test this code as I do not have the data to test, but try to get it to look close to that.

And, in all honesty, it is my opinion that you should be using something like an SQLite database to do this type of work. Manipulating a single flat file with multiple threads is a tough thing to do properly and efficiently.

ETA

Here is an example of an async/await pattern using SemaphoreSlim for synchronization

private static readonly SemaphoreSlim _smph = new SemaphoreSlim(1, 1);

private static async Task<IEnumerable<string>> ReadAllLinesAsync(
    string fileName, bool removeEmptyLines = true)
{
    using (var s = File.OpenText(fileName))
    {
        var data = await s.ReadToEndAsync().ConfigureAwait(false);
        return await Task.Run(() =>
            data.Split(new[] { Environment.NewLine },
                removeEmptyLines ? StringSplitOptions.RemoveEmptyEntries : StringSplitOptions.None));
    }
}

private static async Task WriteAllLinesAsync(string fileName, IEnumerable<string> lines)
{
    using (var s = File.OpenWrite(fileName))
    using (var sr = new StreamWriter(s))
    {
        var data = await Task.Run(() => 
            string.Join(Environment.NewLine, lines)).ConfigureAwait(false);
        await sr.WriteAsync(data);
    }
}

public static async Task RemoveCoinFromBuyOrderLogsAsync(string symbol)
{
    await _smph.WaitAsync().ConfigureAwait(false);
    try
    {
        var lines = await ReadAllLinesAsync(walletFilename);
        lines = lines.Where(c =>
                !c.StartsWith(symbol + "USDT") &&
                !c.StartsWith(symbol + "BUSD") &&
                !c.StartsWith(symbol + "USDC") &&
                !c.StartsWith(symbol + "TUSD"));
        await WriteAllLinesAsync(walletFilename, lines);
    }
    finally
    {
        _smph.Release();
    }
}

public static async Task AddCoinToOrderLogsAsync(string newOrder, long orderId)
{
    await _smph.WaitAsync().ConfigureAwait(false);
    try
    {
        var lines = await ReadAllLinesAsync(walletFilename);

        var duplicate = lines.Select(line => line.Split('\t'))
                .Any(x => (x.Length >= 5) && Convert.ToInt64(x[4]) == orderId);

        if (!duplicate)
        {
            var newLines = await Task.Run(() =>
            {
                var newList = lines.ToList();
                newList.Add(newOrder);
                newList.Sort();
                return newList;
            });

            await WriteAllLinesAsync(walletFilename, newLines);
        }
    }
    finally
    {
        _smph.Release();
    }
}

I added Task.Run on parts I thought could be CPU intensive operations.

Answer 3

File.ReadAllLines(StringPath)

This function opens a text file, reads all lines of the file, and then closes the file. If file is less than 4G could be useful.

Cast the result string array to a list and do what ever you want with it. The file remains free and unlocked. Update the final result, using WriteAllLines(path,StringLines) at the end.

There is also asynchronous method if it is really necessary to do it real-time This article could be really helpful. The functions are

For reading

public async Task SimpleReadAsync()
{
    string filePath = "simple.txt";
    string text = await File.ReadAllTextAsync(filePath);

    Console.WriteLine(text);
}

For writing

    public async Task SimpleWriteAsync()
    {
        string filePath = "simple.txt";
        string text = $"Hello World";
    await File.WriteAllTextAsync(filePath, text);

}