How to Read last lines from a big file with Go every 10 secs

Question

how can I read the last two lines from a big log file without load it into memory completely?

I need read it every 10 secs(On a Win machine)...and I'm stuck trying to read the last lines..

package main

import (
    "fmt"
    "time"
    "os"
)

const MYFILE = "logfile.log"

func main() {
    c := time.Tick(10 * time.Second)
    for now := range c {
        readFile(MYFILE)
    }
}

func readFile(fname string){
    file, err:=os.Open(fname)
    if err!=nil{
        panic(err)
    }
    buf:=make([]byte, 32)
    c, err:=file.ReadAt(32, ????)
    fmt.Printf("%s\n", c)


}

The log file is something like:

07/25/2013 11:55:42.400, 0.559
07/25/2013 11:55:52.200, 0.477
07/25/2013 11:56:02.000, 0.463
07/25/2013 11:56:11.800, 0.454
07/25/2013 11:56:21.600, 0.424
07/25/2013 11:56:31.400, 0.382
07/25/2013 11:56:41.200, 0.353
07/25/2013 11:56:51.000, 0.384
07/25/2013 11:57:00.800, 0.393
07/25/2013 11:57:10.600, 0.456

Thanks!

Answer 1

You can use file.Seek() or file.ReadAt() to almost the end and then Reading forward. You can only estimate where to start seeking unless you can know that 2 lines = x bytes.

You can get the File length by using the os.Stat(name)

Here is an example based on ReadAt, Stat, and your sample log file:

package main

import (
    "fmt"
    "os"
    "time"
)

const MYFILE = "logfile.log"

func main() {
    c := time.Tick(10 * time.Second)
    for _ = range c {
        readFile(MYFILE)
    }
}

func readFile(fname string) {
    file, err := os.Open(fname)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    buf := make([]byte, 62)
    stat, err := os.Stat(fname)
    start := stat.Size() - 62
    _, err = file.ReadAt(buf, start)
    if err == nil {
        fmt.Printf("%s\n", buf)
    }

}

Answer 2

Some people will come to this page looking for efficiently reading the last line of a log file (like the tail command line tool).

Here is my version to read the last line of a big file. It use two previous suggestions (using Seek and file Stat).

It read the file backward, byte by byte (no need to set a buffer size) until finding the beginning of a line or the beginning of the file.

func getLastLineWithSeek(filepath string) string {
    fileHandle, err := os.Open(filepath)

    if err != nil {
        panic("Cannot open file")
        os.Exit(1)
    }
    defer fileHandle.Close()

    line := ""
    var cursor int64 = 0
    stat, _ := fileHandle.Stat()
    filesize := stat.Size()
    for { 
        cursor -= 1
        fileHandle.Seek(cursor, io.SeekEnd)

        char := make([]byte, 1)
        fileHandle.Read(char)

        if cursor != -1 && (char[0] == 10 || char[0] == 13) { // stop if we find a line
            break
        }

        line = fmt.Sprintf("%s%s", string(char), line) // there is more efficient way

        if cursor == -filesize { // stop if we are at the begining
            break
        }
    }

    return line
}

Answer 3

I think a combination of File.Seek(0, 2) and File.Read() should work.

The Seek call gets you to the end of file. You can Seek to a position a bit before the EOF to get last few lines. Then you Read till the EOF and just sleep in your goroutine for 10 seconds; next Read has a chance to get you more data.

You can snatch the idea (and the scan-back logic for initially showing few last lines) from GNU tail's source.

Answer 4

Well, this is only a raw idea and maybe not the best way, you should check and improve it, but seems to work...

I hope that experienced Go users could contribute too..

With Stat you can get the size of the file and from it get the offset for use with ReadAt

func readLastLine(fname string) {
    file, err := os.Open(fname)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    fi, err := file.Stat()
    if err != nil {
        fmt.Println(err)
    }

    buf := make([]byte, 32)
    n, err := file.ReadAt(buf, fi.Size()-int64(len(buf)))
    if err != nil {
        fmt.Println(err)
    }
    buf = buf[:n]
    fmt.Printf("%s", buf)

}

Answer 5

I used tail for a smaller footprint. Not sure how it compares performance wise.

// use "-1" as count for just last line
func printLastLines(count, path string) {
    c := exec.Command("tail", count, path)
    output, _ := c.Output()
    fmt.Println(string(output))
}

For windows you have to do something like this

func printLastWindows(count, path string) {
    ps, _ := exec.LookPath("powershell.exe")
    args := strings.Split(fmt.Sprintf(`Get-Content %s | Select-Object -last %s`, path, count), " ")
    c := exec.Command(ps, args...)
    output, _ := c.Output()
    fmt.Println(string(output))
}

Answer 6

Here's the code I wrote for reading large bytes in reverse line order. It doesn't break on trailing whitespace.

What this code do is loop the bytes in reverse, it count up the number of byte it has encounter. When it detech a newline character, it loop back by that number to write the line and append() it into the resulting []byte and then reset the number. It do this until the maxLine variable is satisfied.

This is overly complicated, if you just want to read bytes from specific line, there might be a better way for that. The variable names has been longed for easier reading.

func ReverseByte(fileByte []byte, maxLine int) []byte {
    // This is a byte "code" for NewLine or "\n"
    nl := byte(10)

    var reverseFileByte []byte
    var lineLen, lineWritten int

    byteIndex := len(fileByte) - 1
    for lineWritten < maxLine {
        if fileByte[byteIndex] == nl {
            currentLine := make([]byte, lineLen)
            byteLineIndex := byteIndex
            var currentLineIndex int
            for currentLineIndex < lineLen {
                currentLine[currentLineIndex] = fileByte[byteLineIndex]
                byteLineIndex++
                currentLineIndex++
            }
            reverseFileByte = append(reverseFileByte, currentLine...)
            lineLen = 0
            lineWritten++
        }
        lineLen++
        byteIndex--
    }
    return reverseFileByte
}

https://go.dev/play/p/qKDFxiJQAfF