How to create a 10MB file filled with "000000..." data in golang?

Question

I am intended to use fdatasync in a system like log or diskqueue. The first thing is to create a 10MB file with "000000..." in file system like ext4. But I don't know how to do it properly.

Answer 1

jnml@fsc-r630:~/src/tmp/SO/16797380$ ls -l
celkem 4
-rw-rw-r-- 1 jnml jnml 186 kvě 29 07:54 main.go
jnml@fsc-r630:~/src/tmp/SO/16797380$ cat main.go 
package main

import (
        "log"
        "os"
)

func main() {
        f, err := os.Create("foo.bar")
        if err != nil {
                log.Fatal(err)
        }

        if err := f.Truncate(1e7); err != nil {
                log.Fatal(err)
        }
}
jnml@fsc-r630:~/src/tmp/SO/16797380$ go run main.go 
jnml@fsc-r630:~/src/tmp/SO/16797380$ ls -l
celkem 4
-rw-rw-r-- 1 jnml jnml 10000000 kvě 29 07:55 foo.bar
-rw-rw-r-- 1 jnml jnml      186 kvě 29 07:54 main.go
jnml@fsc-r630:~/src/tmp/SO/16797380$ 

Answer 2

If you are using unix, then you can create a sparse file very quickly. A sparse file is filled with zero (ascii NUL) and doesn't actually take up the disk space until it is written to, but reads correctly.

package main

import (
    "log"
    "os"
)

func main() {
    size := int64(10 * 1024 * 1024)
    fd, err := os.Create("output")
    if err != nil {
        log.Fatal("Failed to create output")
    }
    _, err = fd.Seek(size-1, 0)
    if err != nil {
        log.Fatal("Failed to seek")
    }
    _, err = fd.Write([]byte{0})
    if err != nil {
        log.Fatal("Write failed")
    }
    err = fd.Close()
    if err != nil {
        log.Fatal("Failed to close file")
    }
}

Which produces a 10MB file called output.

$ ls -l output 
-rw-r--r-- 1 user user 10485760 May 28 18:58 output
$ du -hs output 
4.0K    output

Update much later

I solved exactly this problem in rclone. Namely preallocating files without writing the data, or creating a sparse file. You can't do it directly from the standard library and it isn't cross platform, but using the fallocate syscall with the FALLOC_FL_KEEP_SIZE flag is the way to go in linux. You can also do this in windows.

Here are links to the relevant code in windows and linux and here is the linux code:

var (
    fallocFlags = [...]uint32{
        unix.FALLOC_FL_KEEP_SIZE,                             // Default
        unix.FALLOC_FL_KEEP_SIZE | unix.FALLOC_FL_PUNCH_HOLE, // for ZFS #3066
    }
    fallocFlagsIndex int32
)

// preAllocate the file for performance reasons
func preAllocate(size int64, out *os.File) error {
    if size <= 0 {
        return nil
    }
    index := atomic.LoadInt32(&fallocFlagsIndex)
again:
    if index >= int32(len(fallocFlags)) {
        return nil // Fallocate is disabled
    }
    flags := fallocFlags[index]
    err := unix.Fallocate(int(out.Fd()), flags, 0, size)
    if err == unix.ENOTSUP {
        // Try the next flags combination
        index++
        atomic.StoreInt32(&fallocFlagsIndex, index)
        fs.Debugf(nil, "preAllocate: got error on fallocate, trying combination %d/%d: %v", index, len(fallocFlags), err)
        goto again

    }
    // FIXME could be doing something here
    // if err == unix.ENOSPC {
    //  log.Printf("No space")
    // }
    return err
}

Answer 3

Try this:

package foo

import "io"

func WriteBytes(w io.Writer, c byte, n uint) {
    buf := make([]byte,0x1000)

    for i := 0 ; i < len(buf) ; i++ {
        buf[i] = c
    }

    for i := 0 ; i < n >> 24 ; i++ {
        w.Write(buf)
    }

    w.Write(buf[:n & 0xfff])
}

Answer 4

This will create a 10000000 byte file in the current directory named testfile.

package main

import (
    "fmt"
    "os"
)

func main() {
    data := make([]byte, int(1e7), int(1e7)) // Initialize an empty byte slice
    f, err := os.Create("testfile")
    if err != nil {
        fmt.Printf("Error: %s", err)
    }
    defer f.Close()
    size, err := f.Write(data) // Write it to the file
    if err != nil {
        fmt.Printf("Error: %s", err)
    }
    fmt.Printf("%d bytes written", size)
}

Hope that helps.