Golang, is there a better way read a file of integers into an array?

Question

I need to read a file of integers into an array. I have it working with this:

package main

import (
    "fmt"
    "io"
    "os"
)

func readFile(filePath string) (numbers []int) {
    fd, err := os.Open(filePath)
    if err != nil {
        panic(fmt.Sprintf("open %s: %v", filePath, err))
    }
    var line int
    for {

        _, err := fmt.Fscanf(fd, "%d\n", &line)

        if err != nil {
            fmt.Println(err)
            if err == io.EOF {
                return
            }
            panic(fmt.Sprintf("Scan Failed %s: %v", filePath, err))

        }
        numbers = append(numbers, line)
    }
    return
}

func main() {
    numbers := readFile("numbers.txt")
    fmt.Println(len(numbers))
}

The file numbers.txt is just:

1
2
3
...

ReadFile() seems too long (maybe because of the error handing).

Is there a shorter / more Go idiomatic way to load a file?

You're missing `fd.Close()`. Add a `defer fd.Close()` as line two of `readFile`. — Mostafa, Mar 25 '12 at 19:20
Place the 'defer fd.Close()' after the error checking. You will get runtime panics from this line when the file read fails because fd is nil. Check for the error first, then defer your close. You won't need to close anyway if you failed to open. — burfl, Jul 18 '13 at 12:51
To clarify, this is because defers are evaluated immediately and executed later. So when you try to defer fd.Close() on a nil fd (which has no methods) you will get a panic. 'x := 2; defer fmt.Print(x); x = 3' will print '2', not 3. — burfl, Jul 18 '13 at 12:59

score 25 · Answer 1 · edited Aug 03 '16 at 14:52

Using a bufio.Scanner makes things nice. I've also used an io.Reader rather than taking a filename. Often that's a good technique, since it allows the code to be used on any file-like object and not just a file on disk. Here it's "reading" from a string.

package main

import (
    "bufio"
    "fmt"
    "io"
    "strconv"
    "strings"
)

// ReadInts reads whitespace-separated ints from r. If there's an error, it
// returns the ints successfully read so far as well as the error value.
func ReadInts(r io.Reader) ([]int, error) {
    scanner := bufio.NewScanner(r)
    scanner.Split(bufio.ScanWords)
    var result []int
    for scanner.Scan() {
        x, err := strconv.Atoi(scanner.Text())
        if err != nil {
            return result, err
        }
        result = append(result, x)
    }
    return result, scanner.Err()
}

func main() {
    tf := "1\n2\n3\n4\n5\n6"
    ints, err := ReadInts(strings.NewReader(tf))
    fmt.Println(ints, err)
}

score 5 · Answer 2 · edited Dec 23 '21 at 22:52

5

I would do it like this:

package main

import (
"fmt"
    "io/ioutil"
    "strconv"
    "strings"
)

// It would be better for such a function to return error, instead of handling
// it on their own.
func readFile(fname string) (nums []int, err error) {
    b, err := ioutil.ReadFile(fname)
    if err != nil { return nil, err }

    lines := strings.Split(string(b), "\n")
    // Assign cap to avoid resize on every append.
    nums = make([]int, 0, len(lines))

    for _, l := range lines {
        // Empty line occurs at the end of the file when we use Split.
        if len(l) == 0 { continue }
        // Atoi better suits the job when we know exactly what we're dealing
        // with. Scanf is the more general option.
        n, err := strconv.Atoi(l)
        if err != nil { return nil, err }
        nums = append(nums, n)
    }

    return nums, nil
}

func main() {
    nums, err := readFile("numbers.txt")
    if err != nil { panic(err) }
    fmt.Println(len(nums))
}

edited Dec 23 '21 at 22:52

Xavier Egea

4,712
3
25
39

answered Mar 25 '12 at 19:20

Mostafa

26,886
10
57
52

1

In my opinion, the "Assign cap to avoid resize on every append" isn't avoiding the resize because a resize of `[]string` is hidden somewhere in `strings.Split`. – Mar 26 '12 at 07:12
No, `strings.Split` first finds number of occurrences of `sep` and uses that number for allocation. See [`genSplit`](http://code.google.com/p/go/source/browse/src/pkg/strings/strings.go#186). – Mostafa Mar 26 '12 at 07:40
Indeed, but `strings.Split` is doing it at the cost of going through the string twice - I didn't expect that. In any case, it isn't true that `append` resizes on every append. – Mar 26 '12 at 08:06
Right. Do you know `append`'s behavior with reallocation? I also guess that it may allocate bigger than what is needed right now, but don't know how much bigger. Does anybody know where can we find it in the source code? – Mostafa Mar 26 '12 at 08:51
Source code: function `runtime·growslice` in http://weekly.golang.org/src/pkg/runtime/slice.c, argument `n` is the number of elements to append (such as: n=1). – Mar 26 '12 at 09:00
Seems like I found it: the size goes up to the next power of two. [See this](http://play.golang.org/p/eD1kaIhP4e). – Mostafa Mar 26 '12 at 09:01
I tried to test the code and it says "i declared and not used" – Tbalz Feb 14 '16 at 08:37
1

@Tbalz fixed! Just changed "i" by "_". – Xavier Egea Dec 23 '21 at 22:53

score 0 · Answer 3 · edited May 23 '17 at 10:31

0

Your solution with fmt.Fscanf is fine. There are certainly a number of other ways to do though, depending on your situation. Mostafa's technique is one I use a lot (although I might allocate the result all at once with make. oops! scratch that. He did.) but for ultimate control you should learn bufio.ReadLine. See go readline -> string for some example code.

edited May 23 '17 at 10:31

Community

1
1

answered Mar 25 '12 at 19:53

Sonia

27,135
8
52
54

Golang, is there a better way read a file of integers into an array?

3 Answers3

Linked