synchronized write operation in C

Question

I am working on a smart camera that runs linux. I capture images from the camera streaming software and writes the images on a SD card (attached with the camera). For writing the individual JPEG images, I used fopen and fwrite C functions. For synchronizing the disk write operation, I use fflulsh(pointer) to flush the buffers and write the data on the SD card. But it seems it has no effect as the write operation uses system memory and the memory gets decreased after every write operation. I also used low-level open and write functions in conjunction with fsync (filedesc), but it also has no effect.

The flushing of buffers take place only when I dismount the SD card and then the memory is freed. How can I disable this cache write instead of SD card write? or how can I force the data to be written on the SD card at the same time instead of using the system memory?

`fsync(2)` should do what you want, unless there's a bug in the SD card driver, or you have write caching enabled on it. — Chris Dodd, Oct 05 '11 at 15:57

score 3 · Answer 1 · edited Oct 05 '11 at 16:34

sync(2) is probably your best bet:

SYNC(2)                    Linux Programmer's Manual                   SYNC(2)

NAME
       sync - commit buffer cache to disk

SYNOPSIS
       #include <unistd.h>

       void sync(void);

DESCRIPTION
       sync() first commits inodes to buffers, and then buffers to disk.

BUGS
       According  to  the  standard specification (e.g., POSIX.1-2001), sync()
       schedules the writes, but may return before the actual writing is done.
       However,  since  version  1.3.20 Linux does actually wait.  (This still
       does not guarantee data integrity: modern disks have large caches.)

score 2 · Answer 2 · edited May 23 '17 at 11:55

2

You can set the O_SYNC if you open the file using open(), or use sync() as suggested above.

With fopen(), you can use fsync(), or use a combination of fileno() and ioctl() to set options on the descriptor.

For more details see this very similar post: How can you flush a write using a file descriptor?

edited May 23 '17 at 11:55

Community

1
1

answered Oct 05 '11 at 16:11

Brian McFarland

9,052
6
38
56

score 1 · Answer 3 · answered Oct 05 '11 at 16:11

1

Check out fsync(2) when working with specific files.

answered Oct 05 '11 at 16:11

jbruni

1,238
8
12

doron · Answer 4 · 2011-10-05T16:40:41.570

There may be nothing that you can really do. Many file systems are heavily cached in memory so a write to a file may not immediately be written to disk. The only way to guarantee a write in this scenario is to actually unmount the drive.

When mounting the disk, you might want to specify the sync option (either using the -oflag in mount or on your fstab line. This will ensure that at least your writes are written synchronously. This is what you should always use for removable media.

score 1 · Answer 5 · answered Oct 06 '11 at 05:33

1

Just because it's still taking up memory doesn't mean it hasn't also been written out to storage - a clean (identical to the copy on physical storage) copy of the data will stay in the page cache until that memory is needed for something else, in case an application later reads that data back.

Note that fflush() doesn't ensure the data has been written to storage - if you are using stdio, you must first use fflush(f), then fsync(fileno(f)).

If you know that you will not need to read that data again in the forseeable future (as seems likely for this case), you can use posix_fadvise() with the POSIX_FADV_DONTNEED flag before closing the file.

answered Oct 06 '11 at 05:33

caf

233,326
40
323
462

I tried both high-level fopen and fwrite functions with fflush(), and low-level open and write function with fsync(). In both cases, the write operation is synchronized with the SD card, but the cache always contains a copy of the written data and thus the available system memory decreases after every write operation. I also found that if the SD card is mounted with -o sync option, it works even without fflush() or fsync(). I, then, tried clearing the cache after every write operation with: echo 1 > /proc/sys/vm/drop_caches. This clears the cache after every write operation. – user846400 Oct 06 '11 at 10:27
But, surprisingly, the image write speed is horribly slow (i.e., around 1 image/sec). Even if I don't clear the cache, let the system memory gets decreased, mount the SD card with -o sync option (or use fflush/fsync), the image saving speed remains same (i.e., 1 image/sec). The images are being captured at a rate of 10 FPS (5MP JPEG). I measure time of the save function and it gives me just 60-70 ms for the save operation. But I don't know what is taking so long to complete the write operation. – user846400 Oct 06 '11 at 10:28
@user846400: It is not clear why you care about memory used by clean page cache pages - such memory is available to be used if it is needed (and even if you do care, you can use `posix_fadvise()` to drop those pages rather than the big stick of `drop_caches`). It is possible that your memory card simply isn't any faster than that. – caf Oct 06 '11 at 11:25
posix_fadvise() with POSIX_FADV_DONTNEED has solved the problem. Thanks for your help. – user846400 Oct 06 '11 at 11:52

synchronized write operation in C

5 Answers5