How to optimize "real-time" C# write-to-file & MATLAB read-from-file operation

Question

I am trying to find a good method to write data from a NetworkStream (via C#) to a text file while "quasi-simultaneously" reading the newly written data from the text file into Matlab.

Basically, is there a good method or technique for coordinating write/read operations (from separate programs) such that a read operation does not block a write operation (and vice-versa) and the lag between successive write/reads is minimized?

Currently I am just writing (appending) data from the network stream to a text file via a WriteLine loop, and reading the data by looping Matlab's fscanf function which also marks the last element read and repositions the file-pointer to that spot.

Relevant portions of C# code:

(Note: The loop conditions I'm using are arbitrary, I'm just trying to see what works right now.)

NetworkStream network_stream = tcp_client.GetStream();

string path = @"C:\Matlab\serial_data.txt";
            FileInfo file_info = new FileInfo(path);
            using (StreamWriter writer = file_info.CreateText())
            {
                string foo = "";
                writer.WriteLine(foo);
            }

            using (StreamWriter writer = File.AppendText(path))
            {
                byte[] buffer = new byte[1];
                int maxlines = 100000;
                int lines = 0;
                while (lines <= maxlines)
                {
                    network_stream.Read(buffer, 0, buffer.Length);
                    byte byte2string = buffer[0];
                    writer.WriteLine(byte2string);
                    lines++;
                }
            }

Relevant Matlab Code:

i=0;

while i<100;
    a = fopen('serial_data.txt');
    b = fscanf(a, '%g', [1000 1]);
    fclose(a);
    i=i+1;
end

When I look at the data read into Matlab there are large stretches of zeros in between the actual data, and the most disconcerting part is that number of consecutive data-points read between these "false zero" stretches varies drastically.

I was thinking about trying to insert some delays (Thread.sleep and wait(timerObject)) into C# and Matlab, respectively, but even then, I don't feel confident that will guarantee I always obtain the data received over the network stream, which is imperative.

Any advice/suggestions would be greatly appreciated.

Answer 1

Looks like there's an issue with how fscanf is being used in the reader on the Matlab side.

The reader code looks like it's going to reread the entire file each time through the loop, because it's re-opening it on each pass through the loop. Is this intentional? If you want to track the end of a file, you probably want to keep the file handle open, and just keep checking to see if you can read further data from it with repeated fscanf calls on the same open filehandle.

Also, that fscanf call looks like it might always return a zero-padded 1000-element array, regardless of how large the file it read was. Maybe that's where your "false zeros" are coming from. How many there are would vary with how much data is actually in the file and how often the Matlab code read it between writes. Grab the second argout of fscanf to see how many elements it actually read.

[b,nRead] = fscanf(a, '%g', [1000 1]);
fprintf('Read %d numbers\n', nRead);
b = b(1:nRead);

Check the doc page for fscanf. In the "Output Arguments" section: "If the input contains fewer than sizeA elements, MATLAB® pads A with zeros."

And then you may want to look at this question: How can I do an atomic write/append in C#, or how do I get files opened with the FILE_APPEND_DATA flag?. Keeping the writes shorter than the output stream's buffer (like they are now) will make them atomic, and flushing after each write will make them visible to the reader in a timely manner.