HOWTO: Cross-Operating-System Large File IO in C?

Question

In short: Cross-Operating-System, Large-File Support in C is horrendous.

Goal: I am trying to have "one way" (most likely macro based) to allow 32 bit AND 64 bit to have large file support. Ideally with typedef's, #ifdef's, #(n)defined, etc a macro wrapper could allow basic large file support in the form of an #include library or set of defined macros.

Research: POSIX's file operations have been performing great across BSD/Mac/Linux for 32 and 64 bit IO with files greater than the typical 2^31 size, but even with clang or mingw on Windows I cannot leverage these calls due to M$'s silly implementation of POSIX(if that's what we want to call it...). I am leaning towards using CreateFile(), ReadFile(), WriteFile() on Windows, but this is COMPLETELY DIFFERENT than POSIX's open()/read()/write()/close()/etc in terms of methodology and data types used.

Question: After banging my head against my keyboard, and several text books, I've decided to poll all of you to see: how go you guys/gals accomplish Cross OS File I/O that supports large files?

P.S. I have research links:

score 1 · Answer 1 · answered Apr 24 '14 at 04:58

1

It seems, you need a different version of mingw:

http://mingw-w64.sourceforge.net/

The w64 variant has linux compatible large file support even on 32b windows.

answered Apr 24 '14 at 04:58

oakad

6,945
1
22
31

score 1 · Answer 2 · answered Aug 03 '16 at 18:34

As much as we all love to hate on M$ for their crappy standards conformance, this one was actually the ISO C Committee's fault. Originally they used size_t for all file parameters, but size_t is chosen based on the ALU/memory architecture and not based on the OS file-handling capabilities. When everybody switched to 64-bit CPUs, MS stuck with a 32-bit long, which they were perfectly allowed to do and still be conformant, but now their files are bigger than their biggest arithmetic type.

Note that this was eventually solved in C99, but MSVC C99 support is basically non-existent.

Internally, however, they actually do use a 64-bit pointer for tracking your location within the file. The issue is because of the unfortunate cstdlib API you can't use "fseek" or "ftell" for anything bigger than 32-bits.

To demonstrate that Windows does actually use a 64-bit file pointer, this code will actually work as expected when compiled with MSVC++, and will generate a 40GB file (unsigned long is 32-bits) on your hard drive.

#include <stdio.h>

int main(int argc, char **argv) {
    FILE *my_file;
    unsigned long i, j;

    my_file = fopen("bigfile.bin", "wb");

    for(i = 0; i < 10; i++) {
        for(j = 0; j < (1024 * 1024 * 1024); j++) {
            fwrite(&j, sizeof(j), 1, my_file);
        }
    }

    fclose(my_file);

    return 0;
}

So how does this help you? Well, MS provides their own non-standard API that allows 64-bit fseek() and ftell()

https://msdn.microsoft.com/en-us/library/75yw9bf3.aspx

Also though, you can actually move the file pointer with regular fseek() by doing it in increments... basically if you go:

fseek(my_file, 0, SEEK_SET);
for(i = 0; i < 10; i++) {
    fseek(my_file, (1024 * 1024 * 1024), SEEK_CUR);
}

It will effectively move the file pointer to the 10GB mark.

With ftell() though, you're probably fucked without using the MS API for it.

TL;DR - fopen(), fread(), and fwrite() do work on MSVC with large files >2GB, but ftell() and fseek() don't because the API wasn't properly designed.

score 0 · Answer 3 · answered Dec 30 '16 at 01:45

You are not completely without options in Windows:

In Linux, you can use -D_FILE_OFFSET_BITS=64 (a #define _FILE_OFFSET_BITS 64 might work, not sure), and fseeko / ftello. Many systems also have fseeko64 and ftello64), which work regardless of that #define.

HOWTO: Cross-Operating-System Large File IO in C?

3 Answers3