9

In my on-again-off-again compiler project, I've implemented closures as allocated memory with an executable prefix. So a closure is allocated like this:

c = make_closure(code_ptr, env_size, env_data);

c is a pointer to a block of allocated memory, which looks like this:

movl $closure_call, %eax
call *%eax
.align 4
; size of environment
; environment data
; pointer to closure code

closure_call is a helper function that looks at the address most recently placed on the stack and uses it to find the closure data and code pointer. Boehm GC is used for general memory management, and when the closure is no longer referenced it can be deallocated by the GC.

Anyway this allocated memory needs to be marked as executable; in fact the entire pages it spans get marked. As closures are created and deallocated, more and more heap memory in the process will be executable.

For defensive programming reasons I'd prefer to minimise the amount of executable heap. My plan is to try to keep all closures together on the same page(s), and to allocate and deallocate executable pages as needed; i.e. to implement a custom allocator for closures. (This is easier if all closures are the same size; so the first step is moving the environment data into a separate non-executable allocation that can be managed normally. It also makes defensive programming sense.)

But the remaining issue is GC. Boehm already does this! What I want is to somehow tell Boehm about my custom allocations, and get Boehm to tell me when they're able to be GC'd, but to leave it up to me to deallocate them.

So my question is, are there hooks in Boehm that provide for custom allocations like this?

Edmund
  • 10,533
  • 3
  • 39
  • 57

1 Answers1

3

You may be able to do what you want with a finalizer - Boehm GC would still deallocate it, but you would have an opportunity beforehand to memset the closure with breakpoint ops (0xCC on x86) and mark its page non-executable if possible.

However, finalizers have a performance cost, so should not be used lightly. Boehm GC is based on a mark-sweep algorithm, which first identifies all chunks that should not be freed (mark.c), then frees everything else all at once (reclaim.c). In your case, it makes sense to modify the reclamation process to also fill all free space in your executable region with breakpoint ops, and mark pages non-executable as they become completely empty. This avoids finalizers, at the expense of forking the library (I couldn't find any extensibility mechanism for this).

Finally, note that execution prevention is a defense-in-depth measure, and should not be your only security protection. Return-oriented programming can be used to execute arbitrary code using non-modifiable executable regions.

Chiara Coetzee
  • 4,201
  • 1
  • 24
  • 20
  • So, with the finalizer approach, the effect is that executable allocations will be made through the heap, but when there are freed they will return to non-executable if possible? That is an improvement, though we'll need a method to keep track of other executable allocations on that page so we know when to re-mark it. The use of breakpoint operations seems slightly orthoganal to the allocations themselves -- just good practice to have free space filled with that on an executable page, I guess. – Edmund Feb 06 '12 at 22:27
  • Right. One benefit of forking the GC is that during the mark phase you can keep track of which pages have reachable objects on them pretty easily (just use a bitmap with a bit for each executable page). Then afterwards you could mark the ones that were never reached as non-executable. I'm not aware of a way to do anything similar using finalizers, other than doing your own walk of the reference graph. – Chiara Coetzee Feb 06 '12 at 22:57
  • I could maintain a count of allocated executable objects on each page. Increment when an object is created, decrement in the object's finalizer. Things get a little trickier if an object spans a page boundary though (just need to increment/decrement both pages). – Edmund Feb 06 '12 at 23:22
  • That sounds like a great plan, yes. Keep in mind you can't necessarily unmap empty pages, because allocation metadata for chunks on the next page may fall on the current page. You should be able to mark them non-executable however. – Chiara Coetzee Feb 09 '12 at 10:45