2

I've noticed some C projects compiling code which accesses files with _FILE_OFFSET_BITS=64. Now, on my system (which is 64-bit), adding or removing this doesn't seem to do much - but perhaps on other systems it does.

When should I use _FILE_OFFSET_BITS=64 (or any other value for it)? Or, alternatively, what do I need to check to make sure that I don't actually need it?

einpoklum
  • 118,144
  • 57
  • 340
  • 684
  • @AnttiHaapala: Not a duplicate of that, as far as I can see. That question asks what the drawbacks are on systems where it has an effect. This question asks how to detect on which systems it has an effect. –  Jan 06 '18 at 13:12
  • @hvd it should *always* be used - there is no point in not using it, except to compile programs on old systems that cannot work for files larger than 2G. – Antti Haapala -- Слава Україні Jan 06 '18 at 13:13
  • @AnttiHaapala Even on systems where that macro has absolutely zero meaning at all? Nonsense. Regardless, that'd be an *answer*, and one not provided in the linked questions, so still not a duplicate. –  Jan 06 '18 at 13:14
  • The problem is that the standard headers and function definitions are braindead. – Antti Haapala -- Слава Україні Jan 06 '18 at 13:15
  • I don't expect problems in 64-bit; in 32-bit, you should compile everything with large-file-support or compile everything without it: object modules / shared libs compiled with different setting won't work well together. – Lorinczy Zsigmond Jan 06 '18 at 17:45

2 Answers2

3

Yes, do define it.

There is no drawback on using -D_FILE_OFFSET_BITS=64 on a 64-bit platform. On 32-bit platform if you're doing something that could potentially need to seek/tell anything on files where the offsets exceed 2³¹, you need this one or be prepared to use the separate 64-bit functions.

The reason why the behaviour isn't the default is that the C standard and POSIX specifies that long int be used for various functions - on 32-bit Unixen, long int can usually hold a value up to 2³¹ only. Now, the use of -D_FILE_OFFSET_BITS=64 would guarantee that the code that uses lseek, ftello etc, would continue to work in a 32-bit system as it would in a 64-bit system.

Quoting GLibc feature test macros:

Macro: _FILE_OFFSET_BITS

This macro determines which file system interface shall be used, one replacing the other. Whereas _LARGEFILE64_SOURCE makes the 64 bit interface available as an additional interface, _FILE_OFFSET_BITS allows the 64 bit interface to replace the old interface.

[...]

If the macro is defined to the value 64, the large file interface replaces the old interface. I.e., the functions are not made available under different names (as they are with _LARGEFILE64_SOURCE). Instead the old function names now reference the new functions, e.g., a call to fseeko now indeed calls fseeko64.

This macro should only be selected if the system provides mechanisms for handling large files. On 64 bit systems this macro has no effect since the *64 functions are identical to the normal functions.

And fseeko(3) man pages:

On some architectures, both off_t and long are 32-bit types, but defining _FILE_OFFSET_BITS with the value 64 (before including any header files) will turn off_t into a 64-bit type.

Community
  • 1
  • 1
Antti Haapala -- Слава Україні
  • 129,958
  • 22
  • 279
  • 321
0

Autoconf provides a macro for this: AC_SYS_LARGEFILE. It adds the required option by detecting at configure time whether off_t is already a 64-bit type by default, and if not, whether setting _FILE_OFFSET_BITS to 64 turns it into a 64-bit type. It also detects another preprocessor macro, _LARGE_FILES, the same way, used on certain non-GNU systems ("AIX-style hosts" according to a comment).

If you're using autoconf, you can use this macro directly. If you're not using it, you can re-implement this same logic.

These macros are in the reserved name space for any use by the implementation, so it's better not to set them on systems that do not use them the way GNU does. They may have unexpected effects on other systems. While I suspect there are no malicious implementations out there that intentionally break when these macros are defined, some implementations, possibly future ones, could legitimately use macros with these same names, by coincidence, for subtly different purposes.

  • How about CMake? – einpoklum Jan 06 '18 at 13:59
  • @einpoklum I'm not a CMake person myself, but [here's Michihiro NAKAJIMA's approach as used in libarchive](https://github.com/libarchive/libarchive/blob/master/build/cmake/CheckFileOffsetBits.cmake), also available in other projects. It looks like the same approach as Autoconf uses: first tries to compile with default options, then if it fails retries with `-D_FILE_OFFSET_BITS=64`. The C code [CheckFileOffsetBits.c](https://github.com/libarchive/libarchive/blob/master/build/cmake/CheckFileOffsetBits.c) used here is code which is supposed to compile if and only if `off_t` is sufficiently large. –  Jan 06 '18 at 17:35