Is os.File.Write() thread safe in golang?

Question

Is it thread safe, when two Goroutines writes to file concurrently by os.File.Write()?

According to this question Is os.File's Write() threadsafe?, it isn't thread safe. However, the output file ./test.txt of the following code didn't occur errors.

And according to this question Safe to have multiple processes writing to the same file at the same time? [CentOs 6, ext4], the POSIX "raw" IO syscalls are thread safe. os.File.Write() uses the POSIX IO syscalls, so can we say it is thread safe?

package main

import (
    "fmt"
    "os"
    "sync"
)

func main() {
    filePath := "./test.txt"

    var wg sync.WaitGroup
    wg.Add(2)

    worker := func(name string) {
        // file, _ := os.Create(filePath)
        file, _ := os.OpenFile(filePath, os.O_APPEND|os.O_CREATE, 0666)
        defer file.Close()
        defer wg.Done()
        for i := 0; i < 100000; i++ {
            if _, err := file.Write([]byte(name + ": is os.File.Write() thread safe?\n")); err != nil {
                fmt.Println(err)
            }
        }
    }

    go worker("worker1")
    go worker("worker2")

    wg.Wait()
}

Yes, they are as safe as the underlying syscalls you've pointed out, but if the file is not in append mode, it makes no sense to do so. Also, the docs for `os.Create` specifically state that `If the file already exists, it is truncated`. — JimB, Jun 25 '21 at 14:13
Oh, I use `os.Create()` in a wrong way. Should I replace it with `os.OpenFile(filePath, os.O_RDWR|os.O_CREATE, 0666)`? @JimB — Zihe Liu, Jun 25 '21 at 14:24
The only way it makes sense to write to a file concurrently is if you are [appending](https://pkg.go.dev/os#example-OpenFile-Append), which at least guarantees that writes are not interleaved (on POSIX, I'm not familiar with the details of windows). Think about trying to write to any stream, like a TCP connection; what good is writing concurrently if the data could be randomly mixed between multiple writers? Just because it's safe, doesn't mean it's logical to do so. — JimB, Jun 25 '21 at 14:30
There may be a misunderstanding or lack of precision about what "threadsafe" means here. In its most general definition, "threadsafe" does not guarantee the system understands and organizes the data how you intended it, it _only_ guarantees there is no data race, or informally, it won't make your program crash. — JimB, Jun 25 '21 at 15:01
@JimB The case is that there are two threads appending log to a file. If two sentences are wrote concurrently, does it guarantee the two sentences aren't mixed together? — Zihe Liu, Jun 25 '21 at 15:23
The system does not know what a grammatical sentence is, so it cannot prevent them from being mixed. When using `O_APPEND` on POSIX, individual writes are guaranteed to not be interleaved. Outside of that, your results may vary. If you want higher-level coordination, you have to supply higher-level coordination. — JimB, Jun 25 '21 at 15:37
Please note that all the logging packages for Go I came across provided proper serialization with regard to writing to their "sink" done by multiple concurrent writers. IMO going down to POSIX guarantees with regard to `write(2)`s on a file descriptor is an interesting idea but only if lock contention in your logging subsystem is an issue. So far I did not witness this to be a problem here at my $dayjob even in services which are heavy on logging. — kostix, Jun 25 '21 at 16:43
Also note that the `write(2)` atomicity guarantees we're discussing are only meaningful when you really do "immediate" I/O on a log file—that is, without any buffering. This might be exactly the thing you want but you might rather quickly hit the I/O throughput limits—as each write of what you called "a sentence" _will_ result in a system call, and they are slow (orders of µs). A natural solution to speed things up is to use buffering (via `bufio` or otherwise), and once you do that, your point of contention naturally moves higher up the stack—from a file descriptor to the in-memory buffer. — kostix, Jun 25 '21 at 16:47

score 9 · Answer 1 · answered Jun 25 '21 at 17:40

Documentation does not explicitly say it is thread safe.

Looking at Go 1.16.5 version source code though:

// Write implements io.Writer.
func (fd *FD) Write(buf []byte) (int, error) {
    if err := fd.writeLock(); err != nil {
        return 0, err
    }
    defer fd.writeUnlock()
    ...

It uses internal synchronization. Unless you're coding a mars lander I'd say it's fine to assume writes are thread safe.

Kyle Lemons · Answer 2 · 2021-06-25T18:50:54.427

In general, you should not expect that Write calls to an io.Writer will be written out atomically, i.e. all at once. Synchronization at a higher level is recommended if you don't want interleaved outputs.

Even if you can assume that for *os.File each call to Write will be written out atomically because of either internal locking or because it's a single system call, there is no guarantee that whatever is using the file will do so. For example:

fmt.Fprintf(f, "[%s] %s\n", date, message)

The fmt library does not guarantee that this will make a single call to the io.Writer. It may, for example, flush [, then the date, then ] then the message, and then \n separately, which could result in two log messages being interleaved.

Practically speaking, writes to *os.File will probably be atomic, but it is difficult to arrange for this to be useful without incurring significant allocations, and making this assumption might compromise the portability of your application to different operating systems or architectures, or even different environments. It is possible, for example, that your binary compiled to WASM will not have the same behavior, or that your binary when writing to an NFS-backed file will behave differently.

Fprintf writes to an internal buffer first and then calls w.Write to write the buffer's contents, so it should be fine. I don't think you can end up interleaving writes to a file this way. If you can demonstrate otherwise I would be interested. — Eratosthenes, Aug 09 '23 at 16:41

Is os.File.Write() thread safe in golang?

2 Answers2