Why is writing files witht syscall.O_DIRECT flag make writing files slower in go?

Question

I've got a small peice of code named test.go. It counts time(ns) when doing two writings that write a same byte slice to 2 files, one with the flag syscall.O_DIRECT and the other not.

The code is below:

package main;

import (
        "os"
        "time"
        "fmt"
        "strconv"
        "bytes"
        "syscall"
        // "os/exec"
)
func main() {
        num, _ := strconv.Atoi(os.Args[1]);
        writeContent:= bytes.Repeat( ([]byte)("1"), num );
        t1:= time.Now().UnixNano();
        fd1, err := syscall.Open("abc.txt", syscall.O_WRONLY | syscall.O_DIRECT | syscall.O_TRUNC, 0);
        syscall.Write(fd1, writeContent);
        if err != nil {panic(err);}
        t2:= time.Now().UnixNano();
        fmt.Println("sysW1:", t2-t1);

        t1= time.Now().UnixNano();
        fd2, err := syscall.Open("abc.txt", syscall.O_WRONLY | syscall.O_TRUNC, 0);
        syscall.Write(fd2, writeContent);
        if err != nil {panic(err);}
        t2= time.Now().UnixNano();
        fmt.Println("sysW2:", t2-t1);
}

The program is runned in linux command line like this:(after being compiled with go build ./test.go)

./test 1024

I had expected writing file with syscall.O_DIRECT flag to be faster, but the result showed that writing files with syscall.O_DIRECT flag was about 30 times slower than writing without it :(

result:

sysW1: 1107377
sysW2: 37155

Why? I tought writing with syscall.O_DIRECT does less copying and would be faster, but it now turns out to be much slower. Please help me explain it :(

PX: I will not provide playground link since the result is always 0 when running the program on the playground in some reason.

Your expectation is wrong. a) it is a hint that may be ignored and b) if it is not ignored, it will bypass the cache so it will wait until the write to the device is complete, so it will take longer than when the write call uses the cache and completes before the data is physically written to the device. See https://stackoverflow.com/questions/41257656/what-does-o-direct-really-mean — Erwin Bolwidt, Jun 08 '22 at 01:22
When you say "would be faster", do you mean the write would hit the storage faster? Or do you mean the system call would return faster? — David Schwartz, Jun 08 '22 at 01:27
@ErwinBolwidt Thanks, do you know any writing method as fast as writing without `syscall.O_DIRECT` flag but releases the system cache after writing? I mean, I don't want the cache to take too much space in RAM — Sulfuric Acid, Jun 08 '22 at 01:59
The cache is considered free available memory in Linux. If a process needs memory that is occupied by the cache, the kernel will flush/release the cache at that point. The cache doesn't "use up" memory so you don't need to release it. — Erwin Bolwidt, Jun 08 '22 at 02:46
@ErwinBolwidt Thank you very much. Would you like to write this as an answer so that I could accept it? :) — Sulfuric Acid, Jun 08 '22 at 03:59
Cache means fast and direct (=no cache) means slow. Modern hardware is _complicated_ and naive guesses (even guesses based on year long experience and fundamental understanding of all involved components) and expectations are wrong in almost all cases. — Volker, Jun 08 '22 at 05:46

score 1 · Accepted Answer · answered Jun 08 '22 at 07:47

O_DIRECT doesn't do what you think. While it does less memory copying (since it doesn't copy to the cache before copying to the device driver), that doesn't give you a performance boost.

The filesystem cache ensures that the system call can return early before the data is written to the device, and buffer data to send data in larger chunks.

With O_DIRECT, the system call waits until the data is completely transferred to the device.

From the man page for the open call:

O_DIRECT (since Linux 2.4.10)

Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user-space buffers. The O_DIRECT flag on its own makes an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and necessary metadata are transferred.

See also: What does O_DIRECT really mean?

You don't need to manually release the cache after using it. The cache is considered free available memory by the Linux kernel. If a process needs memory that is occupied by the cache, the kernel will flush/release the cache at that point. The cache doesn't "use up" memory.

Why is writing files witht syscall.O_DIRECT flag make writing files slower in go?

1 Answers1