1

How to use direct io write a small file with out change file size.

Assuming I have a file, file size is 1.1KB, I use direct io to read the file, and change some data, and write file......

see below

//open file with O_DIRECT
handle_ = open("1.html", O_RDWR | O_DIRECT | O_SYNC | O_LARGEFILE,0777))

//file size is 1.1KB (i read 4KB)
pread64(handle_,buffer,4096,0);

memcpy(buffer,"11111",5);

//error (512b Aligned)
pwrite64(handle_,buffer,1.1KB,0);


//OK,but file size has been extended to 4KB 
pwrite64(handle_,buffer,4KB,0);

This is a real world problem because some software may use this file too, like webserver, it will get a wrong file size and response to client a bigger file.

(must be direct io, I have my reasons)

winidis
  • 140
  • 1
  • 7
aaaa
  • 79
  • 5
  • 4
    If this is a c++ program, why are you not using filestreams? – styphNate Aug 01 '19 at 09:34
  • Here's an example of how to read the whole file. https://stackoverflow.com/questions/56135248/how-to-load-a-wav-file-in-an-array-in-c/56136348#56136348 Writing it back is not much different. If the file is changed out of your control while you do this operation, you're out of luck. – Ted Lyngmo Aug 01 '19 at 09:36
  • 1
    Why do you use `O_LARGEFILE`? – KamilCuk Aug 01 '19 at 09:44
  • 3
    "_must be direct io,i have my reasons_" - care to share those? – Ted Lyngmo Aug 01 '19 at 09:46
  • 1
    `pwrite64(handle_,buffer,1.1KB,0);`?!? What is this? Post your actual code. You are checking the return values from `open()`, `pread()`, and `pwrite()`? – Andrew Henle Aug 01 '19 at 10:01
  • According to [open documentation](http://man7.org/linux/man-pages/man2/open.2.html): *'O_LARGEFILE: Allow files whose sizes cannot be represented in an off_t (but can be represented in an off64_t) to be opened.'* So at very first, not needed at all. According to [this answer](https://stackoverflow.com/a/3425945/1312382), it shouldn't ever be used at all (and the proposed alternative is recommended in cited documentation as well). *Just a pure guess (!)*: using the flag might enforce 4K (or 512 at least, which 1.1k doesn't match either) block sizes. Have you tried without? – Aconcagua Aug 01 '19 at 12:11
  • O_LARGEFILE is a flag for support > 2GB file on 32-bit system,did not cause any problem,i am sure . – aaaa Aug 01 '19 at 12:36

2 Answers2

2

how to use direct io write a small file with out extend file size

Sorry, you can't - O_DIRECT requires the I/O you do to be 512 byte aligned. Assuming KB is 1024 bytes, 1.1 * 1024 = 1126.4 (which is a strange number because it won't be a whole byte) so you would have to round that up to 1536 to use O_DIRECT thus causing extension.

Your options are limited to:

  • Do the O_DIRECT write then truncate the file to the correct size (it sounds like this is what you wound up doing).
  • Write only full block multiples of the file using O_DIRECT then write the last little bit using a non-O_DIRECT write.

If both of these sound awful then I can only point you towards @ted-lyngmo's request for more information:

"must be direct io,i have my reasons" - care to share those?

O_DIRECT comes with heavy usage rules...

Anon
  • 6,306
  • 2
  • 38
  • 56
-1

I think you can use mapping file into memory, in unix its mmap syscall. Also check out this answer

nonForgivingJesus
  • 595
  • 4
  • 15
  • 2
    If OP insists on direct I/o, there's a warning in the man page about mixing it with mmap. – Shawn Aug 01 '19 at 10:23
  • thanks ,but mmap with direct io is not a good idea for me,now i'm using a stupid solution like this : after call pwrite64, i use ftruncate64 method to reset file size to 1.1Kb – aaaa Aug 01 '19 at 11:19