Read last line from website without saving file on disk

Question

I have a website with many large CSV files (up to 100,000 lines each). From each CSV file, I need to read the last line in the file. I know how to solve the problem when I save the file on disk before reading its content:

                var url = "http://data.cocorahs.org/cocorahs/export/exportreports.aspx?ReportType=Daily&Format=csv&Date=1/1/2000&Station=UT-UT-24"
            var client = new System.Net.WebClient();
            var tempFile = System.IO.Path.GetTempFileName();
            client.DownloadFile(url, tempFile);
            var lastLine = System.IO.File.ReadLines(tempFile).Last();

Is there any way to get the last line without saving a temporary file on disk? I tried:

using (var stream = client.OpenRead(seriesUrl))
{
    using (var reader = new StreamReader(stream))
    {
        var lastLine = reader.ReadLines("file.txt").Last();
    }
}

but the StreamReader class does not have a ReadLines method ...

If you want only the last part of some content, you should implement [HTTP Range headers](https://stackoverflow.com/questions/1434647/). — Dour High Arch, Jul 07 '17 at 19:01

Lance U. Matthews · Accepted Answer · 2017-07-07T19:15:51.697

StreamReader does not have a ReadLines method, but it does have a ReadLine method to read the next line from the stream. You can use it to read the last line from the remote resource like this:

using (var stream = client.OpenRead(seriesUrl))
{
    using (var reader = new StreamReader(stream))
    {
        string lastLine;

        while ((lastLine = reader.ReadLine()) != null)
        {
            // Do nothing...
        }

        // lastLine now contains the very last line from reader
    }
}

Reading one line at a time with ReadLine will use less memory compared to StreamReader.ReadToEnd, which will read the entire stream into memory as a string. For CSV files with 100,000 lines this could be a significant amount of memory.

score 1 · Answer 2 · answered Jul 07 '17 at 18:49

This worked for me, though the service did not return data (Headers of CSV only):

public void TestMethod1()
{
    var url = "http://data.cocorahs.org/cocorahs/export/exportreports.aspx?ReportType=Daily&Format=csv&Date=1/1/2000&Station=UT-UT-24";
    var client = new System.Net.WebClient();

    using (var stream = client.OpenRead(url))
    {
        using (var reader = new StreamReader(stream))
        {
            var str = reader.ReadToEnd().Split('\n').Where(x => !string.IsNullOrEmpty(x)).LastOrDefault();

            Debug.WriteLine(str);
            Assert.IsNotEmpty(str);
        }
    }

}

Read last line from website without saving file on disk

2 Answers2