Allocate file on disk without zeroing

Question

I need to allocate huge file without zeroing it's content. I'm producing this steps fopen => ftruncate => fclose => mmap => (...work...) => munmap with huge file sizes (hundreds of gigabytes). App hangs on termination for a few minutes while system is trying zeroing file bytes – IMHO because of ftruncate usage.

ftruncate(ofd, 0);

#ifdef HAVE_FALLOCATE

    int ret = fallocate(ofd, 0, 0, cache_size);
    if (ret == -1) {
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }

#elif defined(HAVE_POSIX_FALLOCATE)

    int ret = posix_fallocate(ofd, 0, cache_size);
    if (ret == -1) {
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }

#elif defined(__APPLE__)

    fstore_t store = {F_ALLOCATECONTIG, F_PEOFPOSMODE, 0, cache_size, 0};
    int ret = fcntl(ofd, F_PREALLOCATE, &store);
    if (ret == -1) {
        store.fst_flags = F_ALLOCATEALL;
        ret = fcntl(ofd, F_PREALLOCATE, &store);
    }
    if (ret == -1) { // read fcntl docs - must test against -1
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }
    struct stat sb;
    ret = fstat(ofd, &sb);
    if (ret != 0) {
        printf("Failed to write to file to establish the size.\n");
        exit(-1);
    }
    //ftruncate(ofd, cache_size); <-- [1]

#endif

It seems it does not work with commented line [1]. But uncommenting this line produces file zeroing which I am trying to avoid. I really don't care dirty file content before writing. I just wanna avoid hang on app termination.

SOLUTION:

According to @torfo's answer, replaced all my Apple-related code with this few lines:

unsigned long long result_size = cache_size;
int ret = fcntl(ofd, F_SETSIZE, &result_size);
if(ret == -1) {
    printf("Failed set size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
    exit(-1);
}

But only works for superuser!

**deleted** (`ftruncate()` should create a sparse file, too, so it just seems your OS / filesystem doesn't support sparse files) — , Jun 02 '17 at 07:30
If you need to make a huge file without zeroing its content, just `ftruncate` without doing the other things you do. That will create a sparse file if your filesystem supports it. `fallocate` and the other calls will do the opposite, they do preallocate and zero the disk blocks. You run the risk of running out of disk space when writing to a sparse file, but you don't waste time zeroing it. — Art, Jun 02 '17 at 07:52
@Art app with `ftruncate` hangs for a few minutes on termination via `Ctrl+C`. — k06a, Jun 02 '17 at 07:55
Note allocating previously used disk space without zeroing it and without "sparsing it" creates a potential security hole. — tofro, Jun 02 '17 at 08:09

tofro · Accepted Answer · 2017-06-02T12:07:03.063

5

This is MacOS X, apparently.

You can try to replace the ftruncate call with

fcntl(ofd, F_SETSIZE, &size);

(note needs root privilege and might create a security hole because it might provide access to old file contents that was previously there, so must be handled with extreme caution. The "dirty file content" you don't care about might actually be the user's bank account passwords he deleted a week ago...)

MacOS X doesn't really support sparse files - It does create and maintain them, but its file system driver is very eager to fill the holes as soon as possible.

edited Jun 02 '17 at 12:07

answered Jun 02 '17 at 08:01

tofro

5,640
14
31

Thanks! `fcntl(ofd, F_SETSIZE, &result_size);` works like a charm! – k06a Jun 02 '17 at 08:17
Yep, I don't care security problems for this use case. It's a plots generation for HDD mining: https://github.com/r-majere/mjminer/pull/3 – k06a Jun 02 '17 at 09:48

Allocate file on disk without zeroing

1 Answers1