is there a way to make dlopen fail intentionally

Question

I would like to be able to control from inside my library if it is allowed to be loaded or not without using exceptions, meaning for some cases i would like dlopen("mylib.so") to return NULL and only if all conditions are right it will succeed.

Many have asked about the motivation, inside my library i use dlopen several times and i want to make sure all needed components have been loaded before my library can be loaded.

Take in mind i have to use standard solutions, meaning i can't use external plugins or do things like rewriting dlopen.

Answer 1

This is probably some XY problem. We cannot guess your motivation and goals (and these are what really matters). What you want to do is not possible on Linux.

But read carefully and several times the dlopen(3) man page. You'll notice that since "mylib.so" has no / it is handled specifically by using the LD_LIBRARY_PATH environment variable. That is why I generally use an absolute file path for dlopen. See e.g. realpath(3), glob(3), wordexp(3). Notice that there is no documented way to make the dlopen fail outside of the documented failure cases, that is:

On success, dlopen() and dlmopen() return a non-NULL handle for the loaded library. On error (file could not be found, was not readable, had the wrong format, or caused errors during loading), these functions return NULL.

Then you can use dlerror(3) to understand the error cause.

So you could play wild tricks with that LD_LIBRARY_PATH but you should not. You might be weird (actually crazy) enough to e.g. use putenv(3) on that, or add into that path a directory inside some FUSE filesystem managed by your program, or do some LD_PRELOAD trick. But you really should not do such insane tricks.

So be reasonable: solve your actual need in some other ways. Expect dlopen to behave as documented (so to usually succeed), and don't call it if you don't want to. When coding, it is important to use your common sense.

Be aware of rpath (it could be explicitly set at link time), and read carefully Program Library HowTo and Drepper's How To Write Shared Libraries. Read also the C++ dlopen minihowto and be aware of name mangling.

Notice that in practice dlopen is part of your C standard library on Linux, and that libc and ld-linux.so(8) is generally some free software (e.g. GNU glibc or musl-libc). So if you are not happy with the system's dlopen, you could in principle change it (but I don't recommend doing that, since libc is the cornerstone of every Linux system).

You could consider (probably not a good idea) to use some ELF parsing library (like libelf, libbfd, ...) or some ELF analyzing program (like readelf(1) or objdump(1) ...) on that shared object before your dlopen (but a malicious process or user might still alter the shared library after the analysis but before dlopen). You could study the elf(5) format yourself and do such a parsing by hand (probably even more bad idea).

If you are writing that mylib.so library (on Linux, and perhaps some other similar OSes; but this behaviour is non-standard since non-specified in POSIX dlopen), you could be interested by function attributes like __attribute__((constructor)) & __attribute__((visibility)) (see also this and that). If you want to "reject" being dlopen-ed (when some conditions are met) from your mylib.so you could consider having some constructor function testing these conditions and calling exit when they fail. If your mylib.so is a plugin loaded from some other program that you could improve, you might simply seek some initialization function with dlsym, call it after the dlopen, and fail the main program if that initialization function failed. BTW, throwing some C++ exception from such a constructor function (or some obsolete _init one) is unwise, because the dlopen machinery might consume internal resources that won't be released in that case.

At last, in theory, you could re-implement dlopen yourself (above open(2), mmap(2) etc... and care about the relocations in ELF explicitly). That could take a few years (and is processor specific). Study the appropriate x86 ABI.

You probably can achieve your (unstated) goal by just using the usual dlopen, and do some test before it, and perhaps some test (using dlsym) after it. Most programs using plugins are doing that.

Perhaps you might have every exported function of mylib.so do appropriate checks when running. Maybe you could have some static boolean flag set by some function with __attribute__((constructor)) (so it would be called once at dlopen time) and have other public functions check that flag.

In a recent edit you explain at last:

inside my library i use dlopen several times and i want to make sure all needed components have been loaded before my library can be loaded.

There is no need to play with dlopen or its constructors (and you probably don't need to use dlopen inside your library; and if you do that, you need to explain why, how, and where). You just link mylib.so with all the required shared libraries it uses (see this). If they are not loadable or accessible at dlopen time the entire dlopen of mylib.so fails (intuitively, on Linux, dynamic loading is somehow "recursive").

BTW, if you indeed call dlopen inside your mylib.so, that dlopen happens after mylib.so has been dlopen-ed (unless you call dlopen from some constructor function of mylib.so, which is weird but should be ok and makes a different question).

Answer 2

As far as I remember, no, you can't, at least in a normal way. If the library exists at given path, it will be loaded. The dlopen it is not designed to do any "business checks" for you at your discretion. At most, it will abide any filesystem permissions/etc and return error if process has no access to the file, and that's it.

If you have full control on the code that will load your library, then wrap it like Atterson suggested and use dlopen2 and it's done.

If you don't have full control, then dlopen2 would still not prevent anyone from using the original dlopen and bypass the checks. You could try to make it more smart, for example, make your dlopen2 do something detectable so then the library can deny working if it was opened by dlopen instead of dlopen2, but then.. someone could fake up that "something detectable", then use dlopen, done. Then it boils down to making that "something detectable" hard to reproduce by attackers.

Simply, it was not intended to do these things. It's meant to load the library if the OS allows (~filesystem permissions, etc).

For any other "access checks" like "do you have license? no? then go away" you have to implement it inside the library. Let them load it via dlopen, then make the library check the permissions for example, at each call to its functions. Do use exceptions, or just do-nothing and return NULLs. Or even better, you probably could use initialization function (see https://stackoverflow.com/a/1602459/717732 + http://tldp.org/HOWTO/Program-Library-HOWTO/miscellaneous.html#INIT-AND-CLEANUP) and do the check once when the lib gets loaded. Note that this functions take and return void so there's still no way to make the dlopen fail, but at least the library can have its moment to disable its functions.

Answer 3

You could always wrap it into a function

void *dlopen2(const char *filename, int flags){
    if(/*your conditions*/)
        return void *dlopen(filename, flags);
    return (void *) NULL;
}

Answer 4

The correct way to deal with this kind of scenarios, is to make use of seccomp and limit what the dlopen call is allowed to do.

http://man7.org/linux/man-pages/man3/seccomp_rule_add.3.html

Naturally such configuration requires root access.