Reserve disk space before writing a file for efficiency

Question

I have noticed a huge performance hit in one of my projects when logging is enabled for the first time. But when the log file limit is reached and the program starts writing to the beginning of the file again, the logging speed is much faster (about 50% faster). It's normal to set the log file size to hundreds of MBs.

Most download managers allocate dummy file with the required size before starting to download the file. This makes the writing more effecient because the whole chunk is allocated at once.

What is the best way to reserve disk space efficiently, by some fixed size, when my program starts for the first time?

This is a Windows question, not a C++ question. It's a matter of the Windows API rather than how to call it with a C++ program. I'd suggest removing the C++ tag; there's a whole lot of .NET types around here who know what to look for. — David Thornley, Mar 01 '10 at 15:23
Possible duplicate of [How do you pre-allocate space for a file in C/C++ on Windows?](https://stackoverflow.com/questions/7970333/how-do-you-pre-allocate-space-for-a-file-in-c-c-on-windows) — ivan_pozdeev, Oct 20 '17 at 05:55
Why can't you just create an empty log using any size you want? — wRAR, Mar 01 '10 at 15:18
That's exactly what I need. My questions is how to do it. What is the most efficient way to create that empty log file with some size on disk? — haggag, Mar 01 '10 at 15:21

score 9 · Accepted Answer · answered Mar 01 '10 at 15:29

9

void ReserveSpace(LONG spaceLow, LONG spaceHigh, HANDLE hFile)
{
    DWORD err = ::SetFilePointer(hFile, spaceLow, &spaceHigh, FILE_BEGIN);

    if (err == INVALID_SET_FILE_POINTER) {
        err = GetLastError();
        // handle error
    }
    if (!::SetEndOfFile(hFile)) {
        err = GetLastError();
        // handle error
    }
    err = ::SetFilePointer(hFile, 0, 0, FILE_BEGIN); // reset
}

answered Mar 01 '10 at 15:29

plinth

48,267
11
78
120

Thanks. That's exactly what I wanted. – haggag Mar 01 '10 at 15:36

score 4 · Answer 2 · answered Mar 01 '10 at 15:23

4

wRAR is correct. Open a new file using your favourite library, then seek to the penultimate byte and write a 0 there. That should allocate all the required disk space.

answered Mar 01 '10 at 15:23

Ali Lown

2,323
1
18
22

1

Is is true? I always thought that NTFS supports file holes, but I cannot find a source for that right now. – dmeister Mar 01 '10 at 15:26
1

"Most applications are not aware of sparse files and will not create sparse files. The fact that an application is reading a sparse file is transparent to the application. An application that is aware of sparse-files should determine whether its data set is suitable to be kept in a sparse file. After that determination is made, the application must explicitly declare a file as sparse, using the FSCTL_SET_SPARSE control code." – wRAR Mar 01 '10 at 15:30
1

As wRAR implies, NTFS supports file holes, but you have to create the explicitly with FSCTL_SET_SPARSE or by setting the file to be compressed. – Gabe Mar 01 '10 at 15:39
In case the disk is compressed then all files are essentially "sparse" automatically. Holes will be compressed just as much as sparse file would have gained. So in this case you would need to fill the log file with "dummy" data that is compressible about the same amount you real log data will be. Maybe by cloning first 4096 bytes of actual log data? – Roland Pihlakas Dec 13 '16 at 12:45

score 3 · Answer 3 · answered Nov 12 '19 at 18:39

If you are using C++ 17, you should do it with std::filesystem::resize_file

Link

Changes the size of the regular file named by p as if by POSIX truncate: if the file size was previously larger than new_size, the remainder of the file is discarded. If the file was previously smaller than new_size, the file size is increased and the new area appears as if zero-filled.

#include <iostream>
#include <iomanip>
#include <fstream>
#include <filesystem>

namespace fs = std::filesystem;

int main()
{
    fs::path p = fs::current_path() / "example.bin"; 
    std::ofstream(p).put('a');
    fs::resize_file(p, 1024*1024*1024); // resize to 1 G
}

I like this approach. Only difference I'd propose is to open and close the file without actually writing anything to it, e.g.: fstream fs; fs.open(path, ios::binary | ios::out); fs.close(); — Markus, Apr 14 '20 at 12:20

score 1 · Answer 4 · edited Oct 11 '12 at 06:46

You can use the SetFileValidData function to extend the logical length of a file without having to write out all that data to disk. However, because it can allow to read disk data to which you may not otherwise have been privileged, it requires the SE_MANAGE_VOLUME_NAME privilege to use. Carefully read the Remarks section of the documentation.

Also implementation of SetFileValidData depend on fs driver. NTFS support it and FAT only since Win7.

score 1 · Answer 5 · answered Nov 03 '20 at 22:12

If you're using C#, you can call SetLength on a FileStream to instantly set an initial size to the file.

e.g.

using (var fileStream = File.Open(@"file.txt", FileMode.Create, FileAccess.Write, FileShare.Read))
{
    fileStream.SetLength(1024 * 1024 * 1024); // Reserve 1 GB
}

score 0 · Answer 6 · answered Mar 01 '10 at 15:32

0

Here's a simple function that will work for files of any size:

void SetFileSize(HANDLE hFile, LARGE_INTEGER size)
{
    SetFilePointer(hFile, size, NULL, FILE_BEGIN);
    SetEndOfFile(hFile);
}

answered Mar 01 '10 at 15:32

Gabe

84,912
12
139
238

Reserve disk space before writing a file for efficiency

6 Answers6

Linked