Can you dynamically compile and link/load C code into a C program?

Question

I read about eval in C, and it makes sense that if you write a C string parser/evaluator, you can just map it to specific functions in your main C program. But it doesn't actually place it into executable memory from what I understand, like a JIT compiler seems to do. I don't fully understand JIT compilers (I never made one) but I get the gist.

So what I'm wondering is if you can create sort of a JIT compiler in C without doing too much work of parsing C strings and converting to ASTs and whatnot. Basically, can you do like in JavaScript and dynamically create a function (in C), such that that function is exactly the same as any other C function (i.e. it is compiled directly into executable machine code in the executable part of the program sort of thing).

If it's not possible to do that, a second approach would be to dynamically load C imports/files/modules. So you spin off a process that tells the clang compiler to compile some library function(s), and after it's done, without stopping the current program, it links/attaches that new program library to itself, and so can execute the code that way.

If that's not possible, maybe an option is to simply recompile the program in the background, then swap the current program with the new program which boots up from scratch. That would just be very primitive though.

Trying to figure out if you have some structs for your own custom function datatype in C, how you can then execute that function in C in the most optimized way.

The [current top answer](https://stackoverflow.com/a/39105000/298225) in the question you link to appears to show a method to compile C code and link the result into the current program. That is exactly what you request in your third paragraph. It is of course possible, in typical general-purpose systems, to compile the code entirely in main memory instead of disk or other storage and to execute that code, but that is of course considerably more work than invoking an already existing compiler and linker and requires detailed knowledge of the ABI on your target system.… — Eric Postpischil, Jul 01 '19 at 02:26
… In practice, invoking the compiler and linker is essentially never worthwhile except as an academic exercise, and doing it all in memory even less so. Certainly “if you have some structs for your own custom function datatype in C,” then executing a function this way will not be “the most optimized way.” — Eric Postpischil, Jul 01 '19 at 02:28
Would `dlopen` (Unix) or `LoadLibrary` (Windows) suit your needs? They do load the compiled binary into executable memory. — Andrew Sun, Jul 01 '19 at 02:30
I once wrote a full-fledged C interpreter that did all of this. It was a fun challenge -- difficult, but doable. It's both much harder and somewhat easier today, because things have gotten much more complicated, although there are now libraries to do some of the things I had to do by hand. — Steve Summit, Jul 01 '19 at 02:59
(continued) (1) If you want to load and dynamically link to code in a `.so` file, that's what `dlopen` is for. (2) If you want to jump to arbitrary code you've loaded (passing arbitrary arguments), that's what [libffi](https://sourceware.org/libffi/) is for. (3) If you want to load and dynamically link to individual `.o` files (or old-style static `.a` libraries), I don't know of a good way to do that, and I'd be tempted to dust off my old home-brew dynamic linker and teach it about modern object file formats. — Steve Summit, Jul 01 '19 at 03:00

score 8 · Accepted Answer · answered Jul 01 '19 at 02:27

On POSIX systems (Linix, Mac, UNIX) you have the dlopen and dlsym functions you can work with. These functions can be used to load a shared library at run time and execute a function from it.

As far as creating a library, the simplest thing to do would be to write the relevant source code to a file, run gcc in a separate process to compile it, then use dlopen/dlsym to run the functions it contains.

For example:

#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

const char *libsrc =
"#include <stdio.h>\n"
"\n"
"void f1()\n"
"{\n"
"  printf(\"in f1\\n\");\n"
"}\n"
"\n"
"int add(int a, int b)\n"
"{\n"
"  return a+b;\n"
"}\n";

int main()
{
    FILE *libfile = fopen("mylib.c", "w");
    fputs(libsrc, libfile);
    fclose(libfile);
    system("gcc -fPIC -shared -g -Wall -Wextra -o libmylib.so mylib.c");

    void *lib = dlopen("libmylib.so", RTLD_NOW);
    if (!lib) {
        printf("dlopen failed: %s\n", dlerror());
        return 1;
    }

    void (*f)() = dlsym(lib, "f1");
    if (f) {
        f();
    } else {
        printf("dlsym for f1 failed: %s\n", dlerror());
    }

    int (*a)(int, int) = dlsym(lib, "add");
    if (a) {
        int x = a(2,3);
        printf("x=%d\n", x);
    } else {
        printf("dlsym for add failed: %s\n", dlerror());
    }

    dlclose(lib);
    return 0;
}

Antti Haapala -- Слава Україні · Answer 2 · 2019-07-01T07:02:17.760

4

There is also the Tiny C compiler that can be used as a library, with which you can compile a program on the fly and call the functions within the newly compiled code from the existing code without having to resort to dynamic library loading.

The code will not be the most optimized C possible, but it is not too bad either.

An example in this answer.

edited Jul 01 '19 at 07:02

answered Jul 01 '19 at 04:12

Antti Haapala -- Слава Україні

129,958
22
279
321

score 0 · Answer 3 · answered Jul 01 '19 at 08:51

In addition to the dlload route, some expression evaluators and SIMD math kernel tools generate dynamically code into a memory block that is made executable with mprotect(2) and PROT_EXEC.

Though that usually isn't a HLL like C, but just simple math expressions. If your requirements are fairly simple that could be a route. Typically it is used for simple functions that are speed sensitive due to their usage in e.g. 2D/3D plots or image transformations

Can you dynamically compile and link/load C code into a C program?

3 Answers3

Linked