2

When writing lots of sequential data to disk I found that having an internal 4MB buffer and when opening the file for writing I specify [FILE_FLAG_NO_BUFFERING][1], so that my internal buffer is used.

But that also creates a requirement to write in full sector blocks (512 bytes on my machine).

How do I write the last N<512 bytes to disk?

Is there some flag to WriteFile to allow this?

Do I pad them with extra NUL characters and then truncate the file size down to the correct value? (With SetFileValidData or similar?)

For those wondering the reason for trying this approach. Our application logs a lot. To handle this a dedicated log-thread exists, which formats and writes logs to disk. Also if we log with fullest detail we might log more per second than the disk-system can handle. (Usually noticed for customers with SAN systems that are not well tweaked.)

So, the goal is log write as much as possible, but also notice when we start to overload the system, and then hold back a bit, like reducing the details of the logs.
Hence the idea to have a fill a big memory-block and give that to the OS, hoping to reduce the overheads.

leiflundgren
  • 2,876
  • 7
  • 35
  • 53
  • No. The odds you should use FILE_FLAG_NO_BUFFERING are extremely low. It is not meant to "use my internal buffer". – Hans Passant Mar 05 '20 at 17:49
  • [We're currently using FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH, but we would like our WriteFile to go even faster](https://devblogs.microsoft.com/oldnewthing/20140306-00/?p=1583). Once you understand what's going on, you'll probably see why your proposed optimization turned negative. – IInspectable Mar 05 '20 at 19:18
  • 1
    To write the last chunk, you still have to write a full sector, and then adjust the size of the file using `SetEndOfFile`. – Jonathan Potter Mar 05 '20 at 20:08
  • Yes, it requires [sector-aligned](https://learn.microsoft.com/en-us/windows/win32/fileio/file-buffering). Do you mind sharing the reason of using `FILE_FLAG_NO_BUFFERING`? – Rita Han Mar 06 '20 at 07:14

1 Answers1

0

As the comments suggest, doing file writing this way is probably not the best solution for real world situations. But if writing with FILE_FLAG_NO_BUFFERING is used,
SetFileInformationByHandle is the way to mark the file shorter than whole blocks.

int data_len = len(str);
int len_last_block = BLOCKSIZE%datalen;
int padding_to_fill_block = (data_last_block == BLOCKSIZE ? 0 : (BLOCKSIZE-len_last_block);
str.append('\0', padding_to_fill_block);

ULONG bytes_written = 0;
::WriteFile(hFile, data, data_len+padding_to_fill_block, &bytes_written, NULL));
m_filesize += bytes_written;;

LARGE_INTEGER end_of_file_pos;
end_of_file_pos.QuadPart = m_filesize - padding_to_fill_block;
if (!::SetFileInformationByHandle(hFile, FileEndOfFileInfo, &end_of_file_pos, sizeof(end_of_file_pos)))
{
    HRESULT hr = ::GetLastErrorMessage();
}
leiflundgren
  • 2,876
  • 7
  • 35
  • 53