How can I portably turn on large file support?

Question

I am currently writing a C program that reads and writes files that might be over 2 GiB in size. On linux feature_test_macros(7) specifies:

   _LARGEFILE64_SOURCE
          Expose definitions for the alternative API specified by the LFS (Large File Summit) as a  "tran‐
          sitional  extension"  to  the  Single  UNIX  Specification.  (See ⟨http://opengroup.org/platform
          /lfs.html⟩) The alternative API consists of a set of new objects  (i.e.,  functions  and  types)
          whose names are suffixed with "64" (e.g., off64_t versus off_t, lseek64() versus lseek(), etc.).
          New programs should not employ this interface; instead _FILE_OFFSET_BITS=64 should be employed.

   _FILE_OFFSET_BITS
          Defining this macro with the value 64 automatically converts references to 32-bit functions  and
          data  types related to file I/O and file system operations into references to their 64-bit coun‐
          terparts.  This is useful for performing I/O on large files (> 2 Gigabytes) on  32-bit  systems.
          (Defining  this macro permits correctly written programs to use large files with only a recompi‐
          lation being required.)  64-bit systems naturally permit file sizes greater  than  2  Gigabytes,
          and on those systems this macro has no effect.

Is this guaranteed to be portable?

Solaris suggests in lfcompile(5):

 Applications can be compiled in the large  file  compilation
 environment by using the following methods:

     o    Use the getconf(1) utility with one or more of  the
          arguments listed in the table below. This method is
          recommended for portable applications.

          ____________________________________________________________
         |     argument     |                 purpose                |
         |__________________|________________________________________|
         | LFS_CFLAGS       |  obtain compilation flags necessary  to|
         |                  |  enable   the  large  file  compilation|
         |                  |  environment                           |
         | LFS_LDFLAGS      |  obtain link editor options            |
         | LFS_LIBS         |  obtain link library names             |
         | LFS_LINTFLAGS    |  obtain lint options                   |
         |__________________|________________________________________|

     o    Set the compile-time flag _FILE_OFFSET_BITS  to  64
          before including any headers. Applications may com-
          bine objects produced in the large file compilation
          environment  with  objects  produced in the transi-
          tional compilation environment, but must be careful
          with  respect  to  interoperability  between  those
          objects. Applications  should  not  declare  global
          variables of types whose sizes change between  com-
          pilation environments.

Is this portable? On my linux machine, passing these keys to getconf will output nothing. (Which means the keys exist).

Is there any portable approach?

@CarlNorum It's not about machines that can't handle them. Even on current i386 Linuxes, if you open() a file that is larger than 4 GiB without O_LARGEFILE, you get an EOVERFLOW. Dealing with large files requires that you have a 64 bits datatype. Since some utilities relying on a 32 bit offsets into files, you have to turn on large file support during compilation before you can use large files. — fuz, Aug 17 '13 at 21:11
Once, I demoed a simple XOR file encryption program I wrote. I thought why not encrypt that really big movie file I have. Short story, the demo didn't go that well. I would like to know what (if any) portable solutions exist. — recursion.ninja, Aug 17 '13 at 21:20
@FUZxxl: If `getconf` is outputting nothing it probably means that you *don't* need any special flags for large file support - probably because your environment is 64 bit. — caf, Aug 18 '13 at 00:08

score 0 · Accepted Answer · answered Aug 17 '13 at 22:54

0

Use the confstr interface or the corresponding getconf program and try the various _CS_POSIX_V7_ILP32_OFFBIG_CFLAGS, etc. options until you find one that works. This will give you CFLAGS, LDFLAGS, and LIBS needed for an appropriate environment.

answered Aug 17 '13 at 22:54

R.. GitHub STOP HELPING ICE

208,859
35
376
711

1

"Find one that works" does not really sound like a portable way. I don't have every existing Unix system at my disposal and guesswork won't give me certanity. – fuz Aug 18 '13 at 00:23
Did you actually read the links I provided that document these interfaces? That's actually how you use them. On any conforming system, if there is a compilation environment with the desired property that `off_t` is 64 bits, this will find it. If not, you're out of luck. – R.. GitHub STOP HELPING ICE Aug 18 '13 at 00:33
Yes I did. These documents define configuration keys for certain variable width values only. None of these keys mention large file support. Of course I could assume that the configuration key for large file support that exists on Solaris is portable, but this assumption is not backed up without a document describing that this behavior is portable. – fuz Aug 18 '13 at 09:08
I'm sorry for being such a dickhead. I have been enlighted. – fuz Jul 18 '14 at 14:36
Is there a way to do so on windows, and on posix it'd be great if I didn't have to specify the version of POSIX... – MarcusJ Oct 02 '17 at 14:46

How can I portably turn on large file support?

1 Answers1

Linked