Self-modifying code on macOS v10.15 (Catalina) / x64

Question

As part as porting a Forth compiler, I'm trying to create a binary that allows for self-modifying code. The gory details are at https://github.com/klapauciusisgreat/jonesforth-MacOS-x64

Ideally, I create a bunch of pages for user definitions and call mprotect like so:

#define __NR_exit 0x2000001
#define __NR_open 0x2000005
#define __NR_close 0x2000006
#define __NR_read 0x2000003
#define __NR_write 0x2000004
#define __NR_mprotect 0x200004a

#define PROT_READ 0x01
#define PROT_WRITE 0x02
#define PROT_EXEC 0x04
#define PROT_ALL (PROT_READ | PROT_WRITE | PROT_EXEC)
#define PAGE_SIZE 4096


// https://opensource.apple.com/source/xnu/xnu-201/bsd/sys/errno.h
#define EACCES    13              /* Permission denied */
#define EINVAL    22              /* Invalid argument */
#define ENOTSUP   45              /* Operation not supported */


/* Assembler entry point. */
        .text
        .globl start
start:
        // Use mprotect to allow read/write/execute of the .bss section
        mov $__NR_mprotect, %rax                // mprotect
        lea user_defs_start(%rip), %rdi         // Start address
        and $-PAGE_SIZE,%rdi                    // Align at page boundary
        mov $USER_DEFS_SIZE, %rsi               // Length
        mov $PROT_ALL,%rdx
        syscall
        cmp $EINVAL, %rax
        je 1f
        cmp $EACCES,%rax
        je 2f
        test %rax,%rax
        je 4f                                   // All good, proceed:

        // must be ENOTSUP
        mov $2,%rdi                     // First parameter: stderr
        lea errENOTSUP(%rip),%rsi       // Second parameter: error message
        mov $8,%rdx                     // Third parameter: length of string
        mov $__NR_write,%rax            // Write syscall
        syscall
        jmp 3f

1:
        mov $2,%rdi                     // First parameter: stderr
        lea errEINVAL(%rip),%rsi        // Second parameter: error message
        mov $7,%rdx                     // Third parameter: length of string
        mov $__NR_write,%rax            // Write syscall
        syscall
        jmp 3f

2:
        mov $2,%rdi                     // First parameter: stderr
        lea errEACCES(%rip),%rsi        // Second parameter: error message
        mov $7,%rdx                     // Third parameter: length of string
        mov $__NR_write,%rax            // Write syscall
        syscall

3:
        // did't work -- then exit
        xor %rdi,%rdi
        mov $__NR_exit,%rax     // syscall: exit
        syscall

4:
// All good, let's get started for real:

.
.
.

        .set RETURN_STACK_SIZE,8192
        .set BUFFER_SIZE,4096
        .set USER_DEFS_SIZE,65536*2 // 128 kiB ought to be enough for everybody

        .bss
        .balign 8
user_defs_start:
        .space USER_DEFS_SIZE

However, I get an EACCES return value. I suspect that this is because of some security policy apple has set up, but I do not find good documentation.

Where is the source code for mprotect, and/or what are the methods to mark the data area executable and writable at the same time?

I found that compiling with

gcc -segprot __DATA rwx rwx

does mark the entire data segment rwx, so it must somehow be possible to do the right thing. But I would prefer to only make the area hosting Forth words executable, not the entire data segment.

I found a similar discussion here, but without any solution.

Answer 1

The segment that I want to 'unprotect' exec permission in has really two values describing its permissions:

1. the initial protection settings, which for __DATA I want rw-

2. the maximum protection (loosest) settings, which I want to be rwx.

So first I need to set the maxprot field to rwx. According to the ld manpage, this should be achieved by invoking gcc or ld with the flags -segprot __DATA rwx rw. However, a recent change made by Apple to the linker essentially ignores the maxprot value, and sets maxprot=initprot.

Thanks to Darfink, you can use this script to tweak the maxprot bits after the fact. I thought additional code signing with special entitlements was required, but it's not, at least for the __DATA segment. The __TEXT segment may need code signing with the com.apple.security.cs.disable-executable-page-protection entitlement.

Also see here for a few more details.

Looking at the larger picture, I should also point out that rather to unprotect pieces of an otherwise protected __DATA segment, it may be better to create a complete new data/code segment just for the self-modifying code with rwx permissions from the start. This allows still protecting the rest of data by the operating system, and requires no non-standard tools.