How can I make a file reader function more efficiently?

Question

I'm trying this code:

// GetFooter returns a string which is the Footer of an edi file
func GetFooter(file *os.File) (out string, err error) {
    // TODO can scanner read files backwards?  Seek can get us to the end of file 
    var lines []string
    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        lines = append(lines, scanner.Text())
    }
    line1 := lines[len(lines)-2]
    line2 := lines[len(lines)-1]

    return line1 + "\n" + line2, scanner.Err()  
}

I'm wondering if there's a cheaper way to get the last two lines of a file?

Scanner can't read backwards, no. But you don't need to read all the lines into memory if you only need the last two, so you can certainly make it more memory efficient by only retaining the last two lines read. — Adrian, Feb 21 '20 at 17:13
Seek to end of file minus some block size and read that block. Scan block in reverse for line terminators. If two lines not found, seek back another block and scan for line terminators, and so on. Use Adrian's suggestion if file is not "large". — Charlie Tumahai, Feb 21 '20 at 17:17
https://stackoverflow.com/questions/46764241/read-log-file-from-the-end-and-get-the-offset-of-a-particular-string/46767098#46767098 — , Feb 21 '20 at 17:24

sdgluck · Accepted Answer · 2020-02-21T17:28:07.353

3

You can keep only the last two lines in memory as you scan the buffer.

Try it on Go playground.

package main

import (
    "fmt"
    "bufio"
    "bytes"
    "strconv"
)

func main() {
    var buffer bytes.Buffer
    for i := 0; i < 1000; i++ {
        s := strconv.Itoa(i)
        buffer.WriteString(s + "\n")
    }   
    fmt.Println(GetFooter(&buffer))
}

func GetFooter(file *bytes.Buffer) (out string, err error) {
    var line1, line2 string
    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        line1, line2 = line2, scanner.Text()
    }
    return line1 + "\n" + line2, scanner.Err()  
}

edited Feb 21 '20 at 17:28

answered Feb 21 '20 at 17:23

sdgluck

24,894
8
75
90

I'm loving this solution. The only thing better would be if scanner can read a file backwards, line by line. – Jason Michael Feb 21 '20 at 19:26

score 1 · Answer 2 · answered Feb 21 '20 at 17:23

If you know roughly the size of the last two lines, you could set SOME_NUMBER to be that size plus some extra bytes to make sure you always capture the last two, then do something like

file, err := os.Open(fileName)
if err != nil {
    panic(err)
}
defer file.Close()

buf := make([]byte, SOME_NUMBER)
stat, err := os.Stat(fileName)
start := stat.Size() - SOME_NUMBER
_, err = file.ReadAt(buf, start)
if err != nil {
    panic(err)
}
lines := strings.Split(string(start), "\n", -1)
lines = lines[len(lines)-2:]

I don't know their size at all, sadly. – Jason Michael Feb 21 '20 at 19:26 — Jason Michael, Feb 21 '20 at 19:26

How can I make a file reader function more efficiently?

2 Answers2