Read text file line by line using timer

Question

StreamReader sr = new StreamReader("C:/CR EZ Test/Log.txt");    //use with IF
private void timer2_Tick(object sender, EventArgs e)
{
    if ((line = sr.ReadLine()) != null)
    {   
        //FileStream fs = File.Open("C:/CR EZ Test/Log.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
        //StreamReader sr = new StreamReader(fs); //use with While can't use with }else{
        //while ((line = sr.ReadLine()) != null) 
        //{
        string[] dataLog = line.Split(new[] { ',' }, StringSplitOptions.None);
        mpa = (dataLog[1]);
        ml  = (dataLog[2]);
        lph = (dataLog[3]);
        elapsedTime = float.Parse(dataLog[4]) / 1000;

        if (testStatus > 0) time = elapsedTime.ToString("0.0");
        tb2.Value = int.Parse(dataLog[6]);

        if (chart1.Series[0].Points.Count > tb1.Value && tb1.Value > 0)
        {
            chart1.Series[0].Points.RemoveAt(0);
            chart1.Series[1].Points.RemoveAt(0);
        }
        chart1.Series[0].Points.AddXY(dataLog[5], int.Parse(dataLog[1]));
        chart1.Series[1].Points.AddXY(dataLog[5], int.Parse(dataLog[6]));
        //}
    }
    else
    {
        sr.DiscardBufferedData();
        sr.BaseStream.Seek(0, SeekOrigin.Begin);
        sr.BaseStream.Position = 0;
        //sr.Close();
        //alertTB.Text = "";
        timer2.Enabled = false;
    }
    alertTB.ForeColor = Color.Red;
    alertTB.Text = "Data Log Viewing In Progress";
}

The issue is I am reading a text file full of variables back through a GUI, like replaying a video. As the code is shown, it works and I can control the timer tick to change the replay speed. The issue is the file is in use, so I can't write to or delete the text while the file is in use, without closing it first. I would like to either be able to find a workaround of the Streamreader, or use the Filestream to Streamreader code that will allow me to edit the file while it is in use. The issue there is, I can't figure out how to make it work with the timer, it just reads the entire file very quickly. Any help or ideas are greatly appreciated.

The issue here is how to have the commented out code to:

read a line of the text file,
have the timer to tick
then read the next line of the text file, and so on. Obviously handling the data as it arrives.

Would it be an option to read the entire file at once and then use the timer to read from the in-memory data? You would then only need to determine when to read the file again — Camilo Terevinto, Jun 30 '18 at 19:40
minimize the time reading. Give `System.IO.File.ReadAllLines` a go — Cleptus, Jun 30 '18 at 20:12
Open the file, read all of its contents at once into a `List or something, close it, then process ("play back") the contents of the list. When you get to the end of the list, reopen the file, read it, repeat. — 3Dave, Jun 30 '18 at 22:00

Svek · Answer 1 · 2018-07-01T10:31:07.443

Opening a file while it is in use

I think what you are looking for is FileStream with FileShare.ReadWrite for the instance of your StreamReader (not the instance you have commented out),

var fs = new FileStream("C:\foo.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
var sr = new StreamReader(fs);

Setting the position of the stream

It also seems like based on your comments, you are having trouble with positioning the stream, this is how you could do that...

fs.Position = 0; // note this is the FileStream not the StreamReader!
// alternatively, you could use Seek

Difference between sequential and random access

Lastly, you might want to take a look below to see the difference between sequential and random access

A Potential Solution

Here is a class called FileMonitor that will check the file and update the list whenever the file is changed / updated.

I understand that you want a timer to poll the data in the text file, but in case the timer is very fast, I have optimized the FileMonitor to watch the file for changes and only extract when there is a change.

Please note that this only continues to read where it was left off, based on the position of the stream. So, it will not work if lines are deleted or modified prior to getting "extracted". This means it only functions based on your requirements and is not improved to handle a lot of other scenarios, but it should adequately cover your requirements.

public class FileMonitor : IDisposable
{
    private readonly FileStream _file;
    private readonly StreamReader _reader;

    private long _position;
    private List<string> _lines;

    public FileMonitor(string file)
    {
        if (String.IsNullOrEmpty(nameof(file))) throw new ArgumentNullException(nameof(file));

        _lines = new List<string>();

        FileSystemWatcher watcher = new FileSystemWatcher();
        watcher.Path = Path.GetDirectoryName(file);
        watcher.Filter = Path.GetFileName(file);
        watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName;
        watcher.Changed += new FileSystemEventHandler(OnChanged);
        //watcher.Created += new FileSystemEventHandler(OnCreated);
        //watcher.Deleted += new FileSystemEventHandler(OnDeleted);
        //watcher.Renamed += new RenamedEventHandler(OnRenamed);

        // begin watching.
        watcher.EnableRaisingEvents = true;

        // begin reading
        _file = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
        _reader = new StreamReader(_file);
        _lines = ReadLines(_reader).ToList();
        _position = _file.Position;
    }

    private void OnChanged(object source, FileSystemEventArgs e)
    {
        List<string> update = ReadLines(_reader).ToList();
         // fix to remove the immidate newline
        if (update.Count() > 0 && String.IsNullOrEmpty(update[0])) update.RemoveAt(0);
        _lines.AddRange(update);
        _position = _file.Position;

        // just for debugging, you should remove this
        Console.WriteLine($"File: {e.FullPath} [{e.ChangeType}]");
    }

    public IEnumerable<string> Lines { get { return _lines; } }

    public void Reset()
    {
        _file.Position = 0;
        _position = _file.Position;
        _lines.Clear(); 
    }

    private static IEnumerable<string> ReadLines(StreamReader reader)
    {
        string line;
        while ((line = reader.ReadLine()) != null)
        {
            yield return line;
        }
    }

    public void Dispose()
    {
        _reader.Dispose();
        _file.Dispose();
    }
}

Here is how you could use it with your timer

private IEnumerable<string> _lines; // holds all the lines "extracted"

void Main()
{
    string file = @"C:\Data\foo.txt";

    using (var timer = new System.Timers.Timer())
    {
        timer.Interval = 2000; // 2 second interval
        timer.Elapsed += OnTimedEvent; // attach delegate
        timer.Enabled = true; // start the timer    

        // open the file
        using (var monitor = new FileMonitor(file))
        {
            _lines = monitor.Lines;

             // loop forever, remove this
            while (true) { }
        }
    }
}

public void OnTimedEvent(object sender, EventArgs e)
{
    // just for debugging, you should remove this
    Console.WriteLine($"current count: {_lines.Count()}");
}

If it isn't clear, the data extracted is held in a list of strings. Above, you can grab the "extracted" data from the monitor using the monitor.Line property.

The commented out code will allow me to edit the file, or delete all the data in the file.The issue with using that code is the string data just reads to the end without the timer delaying the the next line read. If I could declare it outside of the function, it may work correctly. The Code as is works because the Streamreader is declared outside the function, allowing the timer to tick. — RatherLogical, Jun 30 '18 at 19:48
I just tried that code, it makes no difference it just reads to the end of the file instantly. My greatest issue is implementing a variable delay time between reading the next line of the text file, without blocking the code from running. — RatherLogical, Jun 30 '18 at 20:03
@CRIMSON501 Why not use `Seek` or `Position` and go to the beginning of the stream? — Svek, Jun 30 '18 at 20:06
I do, after I read it to the end and want to view it again. Do you not see that in the code? — RatherLogical, Jun 30 '18 at 20:09
Can I use that in a loop increment like lineCount ++; fs.Position = lineCount; — RatherLogical, Jun 30 '18 at 20:13
@CRIMSON501 I am talking about setting the `Position` on the `FileStream` as it is not the same as a `StreamReader` — Svek, Jun 30 '18 at 20:14
How can I iterate through the file incrementally, instead of reading it concurrently? — RatherLogical, Jun 30 '18 at 20:20
@CRIMSON501 I've added a diagram for you, I think you're now asking about sequential access? — Svek, Jun 30 '18 at 20:23
I need to figure out how to implement a timer tick between the stream reading the next line, any ideas. — RatherLogical, Jun 30 '18 at 20:24
@CRIMSON501 I don't have the time at the moment to write the solution for you. Sorry. But there should be enough info here to get you started. — Svek, Jun 30 '18 at 20:52
@CRIMSON501 -- I've added a potential solution, that will update a list of strings whenever the file is updated -- which is probably a more efficient solution than based on timers. You can call it with your timer as frequent as you like, and it will only extract new data only when there is any new data to pull. — Svek, Jul 01 '18 at 10:16

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

A Proven Working Solution

 string line;
        if (!File.Exists(logFile))
        {
            viewLog.Text = "Play";
            alertTB.ForeColor = Color.Red;
            alertTB.Text = "File Does Not Exist | Log Data To Create File";
            chart.Text = "Scope On";
        }

        if (File.Exists(logFile))
        {
            var lineCount = File.ReadLines(logFile).Count();//read text file line count to establish length for array
            if (lineCount < 2)
            {
                viewLog.Text = "Play";
                alertTB.ForeColor = Color.Red;
                alertTB.Text = "File Exists | No Data Has Been Recorded";
                chart.Text = "Scope On";
            }

            if (counter < lineCount && lineCount > 0)//if counter is less than lineCount keep reading lines
            {
                line = File.ReadAllLines(logFile).Skip(counter).Take(lineCount).First();

                string[] dataLog = line.Split(new[] { ',' }, StringSplitOptions.None);
                //-----------------------------------------Handling my data 
                counter++;
            }
            else
            {
                counter = 0;
                timer2.Enabled = false;
            }
        }

This is the fix I arrived at, it allows editing the file or deleting the contents of the file. I get the line count before trying to load the file. I then use the counter to iterate through the lines. I can change the delay between the next line read based upon the timer tick interval, pause it, or stop it.