7

it is very important to me to write to a file with the O_DIRECT flag.

This is how I open the file:

//Open the file
int fd;
if((fd = open(inFilepath, O_WRONLY | O_CREAT |O_SYNC |O_DIRECT,S_IRUSR|S_IWUSR))<0) {
    //Error handling
    return;
}

I know about O_DIRECT's alignment restrictions. This is why I initialize my buffer with calloc:

char *buff = (char *) calloc((size_t) 1,sizeof(char));

if(write(fd,buff,(size_t)1)<1) {
    //Error logging
    free(buff);
    return -1;
}

And I get the write: Invalid argument error. I even tried to use more extreme measures such as memalign and posix_memalign, but had issues with them (memalign got stuck, and posix_memalign is missing for the ARM processor).

When I comment out the O_DIRECT flag, everything works as it should (but I/O is not direct, which is what I need).

Anyone has any insight as to why this is happening? If O_DIRECT was not implemented in Android, then it should've failed at open(), not at write(); so I must be doing something wrong!

Thanks -LD

LoneDuck
  • 1,852
  • 1
  • 21
  • 28
  • 1
    There can also be size restrictions on IO operations using direct IO. One byte may not work. – Andrew Henle Dec 09 '15 at 16:00
  • 1
    with `O_DIRECT`, the I/O size needs to be the same as a disk file block size (often 4096 bytes) and aligned on a page (address divisible by 4096). You might want to look at the `setbuf()` function as a viable alternative. The only thing `calloc()` does special is set the allocated memory to all 0x00. The malloc, calloc and realloc all return a pointer to a a block of memory those address is evenly divisible by the underlying hardware bus size. (32 or 64 bit) – user3629249 Dec 10 '15 at 17:55
  • @user3629249 - Restrictions on direct IO on Linux are a lot more complex than that. For example, the ext4 file system seems to allow unaligned direct IO. See http://lxr.free-electrons.com/source/fs/ext4/file.c#L102 I've also noticed that the requirement for page- or block-size-only IO operations has been relaxed in recent Linux releases, which makes sense as few files are exact multiples of those values. Using direct IO on Linux is somewhat unpredictable despite there being obvious use cases (e.g., streaming large amounts of data that will never need to be in any page cache). – Andrew Henle Dec 11 '15 at 12:11

2 Answers2

11

I solved it (with your guidance)- and wanted to post my solution in case anyone in the future has similar problems.

The trick was that with the O_DIRECT flag you need to align both the memory address and your buffer to the filesystem's block size (or at least, block size worked for me; sector didn't).

struct stat fstat;
stat(filepath, &fstat); 
int blksize = (int)fstat.st_blksize;
int align = blksize-1;

const char *buff = (char *) malloc((int)blksize+align);
buff = (char *)(((uintptr_t)buff+align)&~((uintptr_t)align));

if(write(fd,buff,(size_t)blksize)<1) { 
        //Error handling
        free((char *)buff);
        return -1;
}

I did two main things:

  1. Found the filesystem hosting my file's block size using stat() and accessing the st_blksize attribute.
  2. Allocated align more bytes than I need. I then added those extra align bytes to the pointer address so that masking off the bits to the lower block size alignment wouldn't leave me with less memory allocated than I wanted. Then of course you AND the bits with the mask (created by flipping the bits of align which is blksize-1), and voila- your buffer is blksize-aligned.

Also note that the amount you write also has to be aligned to block size (at least in my case).

-LD

LoneDuck
  • 1,852
  • 1
  • 21
  • 28
  • 1
    How does one write arbitrarily sized files using this method? I always get an invalid argument when trying to write the last fraction of a block and the file is truncated – Arne Apr 26 '18 at 13:12
  • 1
    Fixed it by truncating the file to correct size after writing last block. – Arne Apr 27 '18 at 08:05
2

Calloc will not align the memory good enough in this case. Allocate more memory than you need, and round it up to the next 4k or so page. Also read the notes below in the manpage for open() with O_DIRECT.

ctrl-d
  • 392
  • 1
  • 8
  • You can use `posix_memalign()` or `valloc()` to get page-aligned memory. See http://man7.org/linux/man-pages/man3/posix_memalign.3.html – Andrew Henle Dec 09 '15 at 15:58
  • 1
    I see. Using mmap to allocate the memory would also work. – ctrl-d Dec 09 '15 at 16:05
  • regarding `valloc()`, per the man page: "The obsolete function valloc() allocates size bytes and returns a pointer to the allocated memory. The memory address will be a multiple of the page size. It is equivalent to memalign(sysconf(_SC_PAGE‐ SIZE),size)." – user3629249 Dec 11 '15 at 20:07
  • regarding `posix_memalign()`, per the man page: "The function posix_memalign() allocates size bytes and places the address of the allocated memory in *memptr. The address of the allo‐ cated memory will be a multiple of alignment, which must be a power of two and a multiple of sizeof(void *). If size is 0, then the value placed in *memptr is either NULL, or a unique pointer value that can later be successfully passed to free(3)." but says nothing about memory alignment on a page boundary. – user3629249 Dec 11 '15 at 20:09
  • An issue I encountered when using O_DIRECT was that once you have allocated your aligned memory, you must write the entire chunk. So if your buffer size is 4096 bytes you must write all 4096 bytes. – schuess Feb 21 '17 at 16:29