7

I'm trying to write a portable program that deals with ustar archives. For device files, these archives store the major and minor device numbers. However, the struct stat as laid out in POSIX only contains a single st_rdev member of type dev_t described with “Device ID (if file is character or block special).”

How can I convert between a pair of major and minor device numbers and a single st_rdev member as returned by stat() in a portable manner?

fuz
  • 88,405
  • 25
  • 200
  • 352

3 Answers3

4

While all POSIX programming interfaces use the device number (of type dev_t) as is, FUZxxl pointed out in a comment to this answer that the common UStar file format -- most common tar archive format -- does split the device number into major and minor. (They are typically encoded as seven octal digits each, so for compatibility reasons one should limit to 21-bit unsigned major and 21-bit unsigned minor. This also means that mapping the device number to just the major or just the minor is not a reliable approach.)

The following include file expanding on Jonathon Reinhart's answer, after digging on the web for the various systems man pages and documentation (for makedev(), major(), and minor()), plus comments to this question.

#if defined(custom_makedev) && defined(custom_major) && defined(custom_minor)
/* Already defined */
#else

#undef custom_makedev
#undef custom_major
#undef custom_minor

#if defined(__linux__) || defined(__GLIBC__)
/* Linux, Android, and other systems using GNU C library */
#ifndef _BSD_SOURCE
#define _BSD_SOURCE 1
#endif
#include <sys/types.h>
#define custom_makedev(dmajor, dminor) makedev(dmajor, dminor)
#define custom_major(devnum)           major(devnum)
#define custom_minor(devnum)           minor(devnum)

#elif defined(_WIN32)
/* 32- and 64-bit Windows. VERIFY: These are just a guess! */
#define custom_makedev(dmajor, dminor) ((((unsigned int)dmajor << 8) & 0xFF00U) | ((unsigned int)dminor & 0xFFFF00FFU))
#define custom_major(devnum)           (((unsigned int)devnum & 0xFF00U) >> 8)
#define custom_minor(devnum)           ((unsigned int)devnum & 0xFFFF00FFU)

#elif defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__NetBSD__) || defined(__DragonFly__)
/* FreeBSD, OpenBSD, NetBSD, and DragonFlyBSD */
#include <sys/types.h>
#define custom_makedev(dmajor, dminor) makedev(dmajor, dminor)
#define custom_major(devnum)           major(devnum)
#define custom_minor(devnum)           minor(devnum)

#elif defined(__APPLE__) && defined(__MACH__)
/* Mac OS X */
#include <sys/types.h>
#define custom_makedev(dmajor, dminor) makedev(dmajor, dminor)
#define custom_major(devnum)           major(devnum)
#define custom_minor(devnum)           minor(devnum)

#elif defined(_AIX) || defined (__osf__)
/* AIX, OSF/1, Tru64 Unix */
#include <sys/types.h>
#define custom_makedev(dmajor, dminor) makedev(dmajor, dminor)
#define custom_major(devnum)           major(devnum)
#define custom_minor(devnum)           minor(devnum)

#elif defined(hpux)
/* HP-UX */
#include <sys/sysmacros.h>
#define custom_makedev(dmajor, dminor) makedev(dmajor, dminor)
#define custom_major(devnum)           major(devnum)
#define custom_minor(devnum)           minor(devnum)

#elif defined(sun)
/* Solaris */
#include <sys/types.h>
#include <sys/mkdev.h>
#define custom_makedev(dmajor, dminor) makedev(dmajor, dminor)
#define custom_major(devnum)           major(devnum)
#define custom_minor(devnum)           minor(devnum)

#else
/* Unknown OS. Try a the BSD approach. */
#ifndef _BSD_SOURCE
#define _BSD_SOURCE 1
#endif
#include <sys/types.h>
#if defined(makedev) && defined(major) && defined(minor)
#define custom_makedev(dmajor, dminor) makedev(dmajor, dminor)
#define custom_major(devnum)           major(devnum)
#define custom_minor(devnum)           minor(devnum)
#endif
#endif

#if !defined(custom_makedev) || !defined(custom_major) || !defined(custom_minor)
#error Unknown OS: please add definitions for custom_makedev(), custom_major(), and custom_minor(), for device number major/minor handling.
#endif

#endif

One could glean additional definitions from existing UStar-format -capable archivers. Compatibility with existing implementations on each OS/architecture is, in my opinion, the most important thing here.

The above should cover all systems using GNU C library, Linux (including Android), FreeBSD, OpenBSD, NetBSD, DragonFlyBSD, Mac OS X, AIX, Tru64, HP-UX, and Solaris, plus any that define the macros when <sys/types.h> is included. Of the Windows part, I'm not sure.

As I understand it, Windows uses device 0 for all normal files, and a HANDLE (a void pointer type) for devices. I am not at all sure whether the above logic is sane on Windows, but many older systems put the 8 least significant bits of the device number into minor, and the next 8 bits into major, and the convention seems to be that any leftover bits would be put (without shifting) into minor, too. Examining existing UStar-format tar archives with references to devices would be useful, but I personally do not use Windows at all.

If a system is not detected, and the system does not use the BSD-style inclusion for defining the macros, the above will error out stopping the compilation. (I would personally add compile-time machinery that could help finding the correct header definitions, using e.g. find, xargs, and grep, in case that happens, with a suggestion to send the addition upstream, too. touch empty.h ; cpp -dM empty.h ; rm -f empty.h should show all predefined macros, to help with identifying the OS and/or C library.)

Originally, POSIX stated that dev_t must be an arithmetic type (thus, theoretically, it might have been some variant of float or double on some systems), but IEEE Std 1003.1, 2013 Edition says it must be an integer type. I would wager that means no known POSIX-y system ever used a floating-point dev_t type. It would seem that Windows uses a void pointer, or HANDLE type, but Windows is not POSIX-compliant anyway.

Community
  • 1
  • 1
Nominal Animal
  • 38,216
  • 5
  • 59
  • 86
  • You could simplify the code a little by using `#undef HAVE_MAKEDEV` once at the top and then simply setting it when you find a supported platform — unless you're intending to let users set `-DHAVE_MAKEDEV` on the command line and somehow magically include the right header independently of this code for an unsupported platform. – Jonathan Leffler Feb 21 '16 at 08:14
  • @JonathanLeffler: I'm worried that the included headers might define `HAVE_MAKEDEV`. I don't see why, really (they should use `_HAVE_MAKEDEV` if they need a detection macro), but that's the reason why I keep `#undef HAVE_MAKEDEV` after every `#include`. I tread carefully in old Unixen... – Nominal Animal Feb 21 '16 at 08:42
  • POSIX has one interface that uses major and minor device numbers: the ustar file format saves device numbers splitted into major and minor numbers. That's what I need this for. – fuz Feb 21 '16 at 09:51
  • @FUZxxl: Darn, you're [right](https://en.wikipedia.org/wiki/Ustar#UStar_format). It's a common file format, too. Another wrinkle is that both major and minor may only have room for a 21-bit unsigned integer, so you do need correct device-major/minor mapping on each architecture. I shall edit my answer accordingly. – Nominal Animal Feb 21 '16 at 19:38
  • @JonathanLeffler: Per the comment by FUZxxl, I decided to pretty much rewrite the answer. Since UStar file format is very widely used, I can see the need to not make guesses about the splitting. – Nominal Animal Feb 21 '16 at 20:40
  • I don't really understand the point of this elaborate header file. Every case but Windows is identical to my very simple answer. – Jonathon Reinhart Oct 12 '18 at 11:43
  • @JonathonReinhart: Have you ever maintained a large (multi-developer), complex/long-lived project ported to multiple systems? (This is not intended in any snarky way, but as a plain honest question.) Although the code ends up being the same, written like this in an expanded way lets other/future programmers know which systems are already supported, and users (using the library) if their system does not seem to be supported (at compile time). In some cases different include files are required, but the main point is to make things clearer to us human programmers. – Nominal Animal Oct 12 '18 at 13:03
  • @NominalAnimal I have, but not this many systems. As I look over your cases again, I see that there are subtle differences that I missed. I concede to your desire to be very explicit about the supported platforms. But, given that these are almost all identical, I would have forgone `custom_*` and stuck with `makedev`,`minor`,`major`, defined them for Windows, and used `#ifdefs` just around the header file inclusion. – Jonathon Reinhart Oct 12 '18 at 16:28
  • @JonathonReinhart: I do have two reasons for preferring `custom_makedev()` over `makedev()` here: First, a nagging suspicion that there might be a platform where `makedev()` produces incorrect results, and needs to be massaged somehow (just cannot recall details!); and second, that developers might not think of looking at project-internal header files when they see a problem with such a common macro. Both of these are very, very weak arguments; in my humble opinion, just enough to warrant using in an example, but no more. Definitely not enough to warrant any changes in an existing codebase! – Nominal Animal Oct 12 '18 at 16:38
3

Use the major() and minor() macros after defining BSD_SOURCE.

The makedev(), major(), and minor() functions are not specified in POSIX.1, but are present on many other systems.

http://man7.org/linux/man-pages/man3/major.3.html

Jonathon Reinhart
  • 132,704
  • 33
  • 254
  • 328
  • Do you know any common platforms that do not define these functions? – fuz Feb 14 '16 at 13:34
  • I've used them on Linux and BSD. Are there any other UNIX-like platforms that qualify as "common"? – Jonathon Reinhart Feb 14 '16 at 13:36
  • OS X, Solaris, Microsoft Windows, more flavours of BSD, just to name a few. – fuz Feb 14 '16 at 13:36
  • Well I was considering OSX a BSD, but I haven't used this there. Solaris might have their own macros, I cant remember. Windows isnt a Unix, and doesn't have the concept of major/minor devices. – Jonathon Reinhart Feb 14 '16 at 13:39
  • Windows is still a POSIX platform and I want my software to run there, too. But what you say is likely sufficiently portable. – fuz Feb 14 '16 at 13:40
  • Ok, I wasn't sure how portable you were hoping to be. I would define my own macro `device_major`, and then stuff the `#ifdef`ery in a compatibility header file. – Jonathon Reinhart Feb 14 '16 at 13:40
  • What exactly are you trying to do? Anything to do with device major/minor numbers on Windows is certainly going to be emulated or just plain unuseful. – Jonathon Reinhart Feb 14 '16 at 13:42
  • That's the generic solution for any portability problem. I was asking this question in the hope that someone knew the specifics (I can't imagine that this is such a rare thing to do). – fuz Feb 14 '16 at 13:42
  • 1
    @FUZxxl, Windows isn't even close to being a POSIX platform. It does support *some* of the POSIX API, but not nearly enough for most POSIX C programs to work without modifications. – Nominal Animal Feb 21 '16 at 07:05
  • @NominalAnimal Windows actually has a [POSIX certified subsystem](https://en.wikipedia.org/wiki/Windows_Services_for_UNIX). – fuz Feb 21 '16 at 09:49
  • @FUZxxl: Have you actually used it? I've worked with over a dozen different OSes on different hardware architectures, and SFU/SUA is one of the most blatant cases of getting a certification in order to avoid being excluded from government contracts while still being completely impractical (and, to my knowledge, irrelevant) for real life applications. I am not saying Windows is bad, I'm just saying one should not expect to be able to compile or run applications that compile fine on most POSIXy systems, on Windows. – Nominal Animal Feb 21 '16 at 20:48
  • @NominalAnimal I know that. Interix is amazing in how carefully crafted to be completely unusable it is. I just wanted to point out that yes, Windows has a certified POSIX subsystem. – fuz Feb 21 '16 at 20:52
  • The point here is that the device numbers returned are going to be effectively useless. I mean, what is your goal? – Jonathon Reinhart Feb 21 '16 at 22:58
2

I have a program based on an antique version of ls for Minix, but much mangled modified by me since then. It has the following code to detect the major and minor macros — and some comments about (now) antique systems where it has worked in the past. It assumes a sufficiently recent version of GCC is available to support #pragma GCC diagnostic ignored etc. You have to be trying pretty hard (e.g. clang -Weverything) to get the -Wunused-macros option in effect unless you include it explicitly.

#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-macros"
/* Defines to ensure major and minor macros are available */
#define _DARWIN_C_SOURCE    /* In <sys/types.h> on MacOS X */
#define _BSD_SOURCE         /* In <sys/sysmacros.h> via <sys/types.h> on Linux (Ubuntu 12.0.4) */
#define __EXTENSIONS__      /* Maybe beneficial on Solaris */
#pragma GCC diagnostic pop

/* From Solaris 2.6 sys/sysmacros.h
**
** WARNING: The device number macros defined here should not be used by
** device drivers or user software. [...]  Application software should make
** use of the library routines available in makedev(3). [...]  Macro
** routines bmajor(), major(), minor(), emajor(), eminor(), and makedev()
** will be removed or their definitions changed at the next major release
** following SVR4.
**
** #define  O_BITSMAJOR 7       -- # of SVR3 major device bits
** #define  O_BITSMINOR 8       -- # of SVR3 minor device bits
** #define  O_MAXMAJ    0x7f    -- SVR3 max major value
** #define  O_MAXMIN    0xff    -- SVR3 max major value
**
** #define  L_BITSMAJOR 14      -- # of SVR4 major device bits
** #define  L_BITSMINOR 18      -- # of SVR4 minor device bits
** #define  L_MAXMAJ    0x3fff  -- SVR4 max major value
** #define  L_MAXMIN    0x3ffff -- MAX minor for 3b2 software drivers.
** -- For 3b2 hardware devices the minor is restricted to 256 (0-255)
*/

/* AC_HEADER_MAJOR:
** - defines MAJOR_IN_MKDEV if found in sys/mkdev.h
** - defines MAJOR_IN_SYSMACROS if found in sys/macros.h
** - otherwise, hope they are in sys/types.h
*/

#if defined MAJOR_IN_MKDEV
#include <sys/mkdev.h>
#elif defined MAJOR_IN_SYSMACROS
#include <sys/sysmacros.h>
#elif defined(MAJOR_MINOR_MACROS_IN_SYS_TYPES_H)
/* MacOS X 10.2 - for example */
/* MacOS X 10.5 requires -D_DARWIN_C_SOURCE or -U_POSIX_C_SOURCE - see above */
#elif defined(USE_CLASSIC_MAJOR_MINOR_MACROS)
#define major(x)    ((x>>8) & 0x7F)
#define minor(x)    (x & 0xFF)
#else
/* Hope the macros are in <sys/types.h> or otherwise magically visible */
#endif

#define MAJOR(x)    ((long)major(x))
#define MINOR(x)    ((long)minor(x))

You will justifiably not be all that keen on the 'hope the macros are … magically visible' part of the code.

The reference to AC_HEADER_MAJOR is to the macro in the autoconf that deduces this information. It would be relevant if you have a config.h file generated by autoconf.

POSIX

Note that the POSIX pax command defines the ustar format and specifies that it includes devmajor and devminor in the information, but adds:

… Represent character special files and block special files respectively. In this case the devmajor and devminor fields shall contain information defining the device, the format of which is unspecified by this volume of POSIX.1-2008. Implementations may map the device specifications to their own local specification or may ignore the entry.

This means that there isn't a fully portable way to represent the numbers. This is not wholly unreasonable (but it is a nuisance); the meanings of the major and minor device numbers varies across platforms and is unspecified too. Any attempt to create block or character devices via ustar format will only work reasonably reliably if the source and target machines are running the same (version of the same) operating system — though usually they're portable across versions of the same operating system.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278