0

I would like to generate some machine code in my program and then run it. One way to do it would be to write out a .so file and then load it in the program but that seems too expensive.

IS there a way in linux for me to write out the code in my data pages and then set my function ointer there and just call it? I've seen something similar on windows where you can allocate a page with the NX protection turned off for that page, but I can't find a similar OS call for linux.

Amit Prakash
  • 1,171
  • 10
  • 19
  • 2
    See: http://stackoverflow.com/questions/3125756/allocate-executable-ram-in-c-on-linux , specially the linked article from Ulrich Drepper. – ninjalj Oct 03 '12 at 18:55

1 Answers1

2

The mmap(2) (with munmap(2)) and mprotect(2) syscalls are the elementary operations to do that. Recall that syscalls are elementary operations from the point of view of an application. You want PROT_EXEC

You could just strace any dynamically linked executable to get a clue about how you might call them, since the dynamic linker ld.so is using them.

Generating a shared object might be less expensive than you imagine. Actually, generating C code, running the compiler, then dlopen-ing the resulting shared object has some sense, even when you work interactively. My MELT domain specific language (to extend GCC) is doing this. Recall that you can do a big lot of dlopen-s without issues.

If you want to generate machine code in memory, you could use GNU lightning (quick generation of slow machine code), libjit from dotgnu (generate less bad machine code), LuaJit, asmjit (x86 or amd64 specific), LLVM (slowly generate optimized machine code). BTW, the SBCL Common Lisp implementation is dynamically compiling to memory and produces good machine code at runtime (and there is also all the JIT for JVMs doing that).

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
  • Thanks a lot for a great answer. On a modern hardware (2GHz Core i7, 7200RPM disk, DDR4) how long do you think it would take to write out say 5MB of code and the load it through dlopen? I'd like it to be under 100ms if possible so that it does not add too much overhead to my processing. – Amit Prakash Oct 04 '12 at 01:28
  • 1
    I don't know, it really depends of the code; you could measure it. Most of the time would be spent in constructing the machine code. BTW, 5MB of code is quite a large amount. On my Debian/Sid/AMD64, I have 3920 files in /usr/bin but only 30 are bigger than 5MB. If you need to emit so much code is so few time, you may be limited to GNU lightning (but the code generated is really slow). Are you sure your 5MB binary code threshold in 100 ms is realistic? It seems huge to me. – Basile Starynkevitch Oct 04 '12 at 04:59
  • Also, 5MB/100ms means a growth of 6Gb of your process address space in two minutes. (and of 180Gb in an hour) I believe that might be not reasonable. Your 5MB wish in 0.1 s seems unreasonable (and it is hard to generate good enough machine code -think of register allocation & instruction scheduling- of that size in so little time). – Basile Starynkevitch Oct 04 '12 at 05:10
  • And 5MB/100ms means 50MBytes/sec which is a significant fraction of a disk transfer rate.... A common 7200RPM hard disk achieve only 150MB/sec on average random IO e.g. http://www.storagereview.com/hitachi_deskstar_7k3000_3tb_review_hds723030ala640 – Basile Starynkevitch Oct 04 '12 at 05:21
  • Thanks a lot for your thoughtful responses, Basile. I Was quoting 5MB from some other app I read about on the web. Now I have more info bout my project and it seems my work can be done in a couple of hundred K of machine code! My queries can work in 5-10 seconds without this optimization and I'm trying to see if I can get it down to a couple of seconds with precompilation. Also, I can dump/reuse the same space for the next query so growth is not important to me as much. – Amit Prakash Oct 05 '12 at 04:37
  • Be careful about reusing generated code space, especially if you have closures, pointer to functions, etc... – Basile Starynkevitch Oct 05 '12 at 04:40