Why write bytes to files is slow for Go, compared with C

Question

I just found that write bytes to files is slow for Go.

I want to create a 10mb file. Go takes almost 1 min to do it, but it is less than 5 seconds in C.

This is Go code:

package main

import (
    "fmt"
    "os"
)

func main() {
    f, _ := os.Create("./src/test/test.txt")
    count := int(1.024e7)
    for i := 0; i < count; i++ {
        f.Write([]byte{byte('a' + i%24)})
    }
    f.Close()
    fmt.Println("ok")
}

And C:

#include <stdio.h>

int main()
{
    FILE *fp=fopen("data.txt","w");
    int  size=1.024e7;
    for(int i=0;i<size;i++)
        putc('a'+i%24,fp);
    fclose(fp);
    printf("ok");
    return 0;
}

Go defaults to unbuffered I/O. Hence, unlike the maze of different buffering behaviors you get in C by default, it's up to the programmer to pick a buffering system to use. — torek, Dec 08 '19 at 14:34

peterSO · Accepted Answer · 2019-12-08T14:33:15.490

5

Meaningless microbenchmarks produce meaningless results.

Go:

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    f, _ := os.Create("tmp/sotest/test.txt")
    w := bufio.NewWriter(f)
    count := int(1.024e7)
    for i := 0; i < count; i++ {
        w.Write([]byte{byte('a' + i%24)})
    }
    w.Flush()
    f.Close()
    fmt.Println("ok")
}

Output:

ok
real    0m0.159s
user    0m0.160s
sys     0m0.004s

C:

#include <stdio.h>

int main()
{
    FILE *fp=fopen("data.txt","w");
    int  size=1.024e7;
    for(int i=0; i<size; i++)
        putc('a'+i%24,fp);
    fclose(fp);
    printf("ok");
    return 0;
}

Output:

ok
real    0m0.058s
user    0m0.045s
sys     0m0.004s

edited Dec 08 '19 at 14:33

answered Dec 08 '19 at 14:29

peterSO

158,998
31
281
276

but do you know why ? Is the C implementation uses buffering out of the box ? – Dec 08 '19 at 14:30
4

@mh-cbon: Yes, C stdio streams are normally either line-buffered or fully-buffered depending on whether they are connected to "terminal-like" devices or "file-like" data. However, the `stderr` stream is normally *un*-buffered, regardless of its data sink. Systems may choose to provide additional overrides, such as magic environment variables (though not enough have a nice way to de-buffer or forcibly line-buffer stdout for cases of piped output). – torek Dec 08 '19 at 15:10
@mh-cbon, for instance, see [`stdbuf`](https://manpages.debian.org/1/stdbuf) – kostix Dec 08 '19 at 16:40

Why write bytes to files is slow for Go, compared with C

1 Answers1

Linked