c++ - mutex or flock fcntl.h to lock only write operation

Question

I am trying to append (write append) to a file from different threads (similar to logging) and so there is no inter-process locking needed.

I have studied flock in fcntl.h and it says flock can do granular lock along with inter-process which is not necessary in my situation.

char* file = "newfile.txt";
int fd;
struct flock lock;

printf("opening %s\n", file);
fd = open(file, O_APPEND);
if (fd >= 0) {
    memset(&lock, 0, sizeof (lock));
    lock.l_type = F_WRLCK;
    fcntl(fd, F_SETLKW, &lock);
    //do my thing here
    lock.l_type = F_UNLCK;
    fcntl(fd, F_SETLKW, &lock);
    close(fd);
}

Will there be overhead as it can do granular locking and inter-process locking? What happens when program crashes when there is lock?

My current preference is mutex,

static std::mutex fileMutex;
fileMutex.lock();
//do my thing here    
fileMutex.unlock();

Is it okay to go with the mutex approach as the synchronization (or locking) is needed only within the process(only multithreading),

or is it okay to implement the code with flock in fcntl.h?

open in O_APPEND mode! – Jean-Baptiste Yunès Feb 13 '17 at 12:22 — Jean-Baptiste Yunès, Feb 13 '17 at 12:22

Johannes Overmann · Answer 1 · 2017-02-13T16:29:48.660

First you need clarify multiple threads vs. multiple processes:

Multiple threads: I would suggest to use a mutex. You can also make this more efficient by adding log messages to the end of a memory buffer, and you only protect the accesses to that buffer using a mutex. You can then have another thread, or some regular maintenance function flush the buffer to the file without locking other threads out while I/O is going on, and without locking the file. The access to the file would then neither be protected by a mutex, nor by a file lock, because it would only every be accessed by a single thread.

(Since you indicate there is no inter-process communication, I would suggest to go this way.)

Multiple processes: You have to use some lock mechanism which is visible by all processes, e.g. the file locks you propose.

Both: Use both mechanisms. Try to lock the file only for the absolute minimum amount of time.

Side node:

The first rule of multithreaded programming is not 'use a mutex', it is 'try hard not to access data by multiple threads', or even 'try hard not to use multiple threads, unless absolutely necessary for performance reasons' (e.g. you can always do without threads for asynchronous operations).

i have mentioned that there is **no inter-process** communication (no multi processes) — fury.slay, Feb 13 '17 at 13:05
so you are suggesting that in case of multi-thread mutex is much better than flock and that there is no need to go for flock, aren't you? — fury.slay, Feb 13 '17 at 13:10
@fury.slay: Yes, definitely. You could put a mutex around your file operations. That would be functionally sufficient and correct, but slow. So I would suggest to only every access the file from one specific thread (so you do not need a mutex for the file I/O) and then you protect a shared memory buffer (a queue of strings or bytes or whatever you have) using a mutex. — Johannes Overmann, Feb 13 '17 at 16:26

score 1 · Accepted Answer · edited May 23 '17 at 12:16

You may not need any locking.

Make your open() call with the O_APPEND flag set, as @Jean-BaptisteYunès mentions in the comments.

Then write your data using a single write() call. POSIX guarantees that if the file is opened in append mode, individual write() operations will be atomic. Per the POSIX standard:

If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation. [emphasis mine]

Your only issue is how to handle a partial write() - where a single write() operation doesn't write all the requested data. The standard requires each write() operation to be atomic - it doesn't guarantee that a request to write 34 MB will result in writing the entire 34 MB. In my experience, partial write() calls to actual files simply do not happen until write() calls request to move a lot of bytes - I've never observed a partial write() result on any single IO operation on a file under 1 MB - and I've done SAN installs and benchmarking for quite a few large organizations.

So if you limit your write() calls to under PIPE_BUF or less bytes (on Linux), you almost certainly can avoid all locking and let the internal locking inside the kernel solve your problem.

For more details, see the following:

Is file append atomic in UNIX?

Make sure to read the questions linked from there, too.

what if the write exceeds PIPE_BUF, the size can easily exceed. I can now think of only mutex with ofstream write. — fury.slay, Feb 14 '17 at 06:20
@fury.slay Since you're running on Linux, if you control your entire platform - including the file system you'll be writing to - you'll have to test things and find out what the actual limit is. See http://stackoverflow.com/questions/10650861 Were you on Solaris or AIX, you'd almost certainly be safe up to well over 1 MB per write operation (note in the link that on ZFS - from Solaris - every `write()` was atomic...) With later versions of Linux, you *may* be. — Andrew Henle, Feb 14 '17 at 10:54
I can test on my system, but the application is bundled to be run on different types of linux os of the users, so I will not have any guarantee on the safety. Correct me if I am wrong. — fury.slay, Feb 15 '17 at 06:50

c++ - mutex or flock fcntl.h to lock only write operation

2 Answers2