Reading bytes into Go buffer with a fixed stride size

Question

I'd like to read and process 1024 bytes at a time in my file given by filename. I don't understand how to construct the outer loop correctly, especially to accommodate the final stride in which the buffer will contain fewer than 1024 bytes

What I have tried:

fs, _ := os.Open(filename)
defer fs.Close()
n := 1024 // 1kb
buff := make([]byte, n)
for {
    buff = make([]byte, n) // is this initialized correctly?
    n1, err := fs.Read(buff)
    if err != nil {
        if err == io.EOF {
            break
        }
        fmt.Println(err)
        break
    }
    fmt.Println("read n1 bytes...", n1)
    fmt.Println(buff)
}

I have seen the following resources:

• Reading specific number of bytes from a buffered reader in golang
• Read arbitrary amount of bytes into buffer Golang
• How to read a binary file in Go
• Working with raw bytes from a network in go
• https://gobyexample.com/reading-files
• Writing into fixed size Buffers in Golang with offsets
• https://rwinslow.com/posts/use-flatbuffers-in-golang/
• reading file line by line in go
• How do I read in a large flat file in Golang
• https://golang.org/pkg/io/#ReadFull

Answer 1

Reading bytes into Go buffer with a fixed stride size

read and process 1024 bytes at a time in my file given by filename.

accommodate the final stride in which the buffer will contain fewer than 1024 bytes.

---

For number of bytes read guarantees, use ioutil.ReadFull. For efficient stream reads, use bufio.Reader. For efficiency, allocate the read buffer once and reuse it.

For example,

package main

import (
    "bufio"
    "fmt"
    "io"
    "io/ioutil"
    "os"
)

func main() {
    stride := 1024
    filename := testname(stride)

    f, err := os.Open(filename)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    defer f.Close()

    r := bufio.NewReader(f)
    buf := make([]byte, 0, stride)
    for {
        n, err := io.ReadFull(r, buf[:cap(buf)])
        buf = buf[:n]
        if err != nil {
            if err == io.EOF {
                break
            }
            if err != io.ErrUnexpectedEOF {
                fmt.Fprintln(os.Stderr, err)
                break
            }
        }

        fmt.Println("read n bytes...", n)
        // process buf
    }
}

func testname(stride int) string {
    f, err := ioutil.TempFile("", "test.stride.")
    if err != nil {
        panic(err)
    }
    defer f.Close()
    _, err = f.Write(make([]byte, 2*stride+stride/2))
    if err != nil {
        panic(err)
    }
    return f.Name()
}

Playground: https://play.golang.org/p/iYOY-z7hkoz

Output:

read n bytes... 1024
read n bytes... 1024
read n bytes... 512