What are the possible ways of intercepting system calls on unix environments?

Question

What are the possible ways of intercepting system calls on unix environments? I'm looking to do in AIX.

Thanks

Related: https://stackoverflow.com/questions/69859/how-could-i-intercept-linux-sys-calls — Matheus Santana, Apr 03 '18 at 11:56

score 3 · Answer 1 · answered May 21 '09 at 10:32

Not familiar with AIX, but the following works on Linux and Solaris. You can use the LD_PRELOAD environment variable, which tells ld.so to load a shared library before libc and then write your own version of the system call, and optionally call the original. man ld.so for more information. Something along the lines of

#include <dlfcn.h>

typedef int (*ioctl_fn)(int, int, void*);

static 
int
my_ioctl(int      fildes,
         int      request,
         void*    argp,
         ioctl_fn fn_ptr)

{
    int result = 0;

    /* call original or do my stuff */
    if (request == INTERESTED)
    {
        result = 0;
    }
    else
    {
        result = (*fn_ptr)(fildes, request, argp);
    }

    return result;
}

/*
 * override ioctl() - on first call get a pointer to the "real" one
 * and then pass it onto our version of the function
 */
int
ioctl(int fildes,
      int request,
      void* argp)
{
    static ioctl_fn S_fn_ptr = 0;

    if (S_fn_ptr == 0)
    {
        S_fn_ptr = (ioctl_fn)dlsym(RTLD_NEXT, "ioctl");
    }

    return my_ioctl(fildes, request, argp, S_fn_ptr);
}

Carved this out of some code I had lying around, apologies if I have made it incorrect.

Nitpick: You're intercepting (system) library calls, not system calls. Probably what the questioner meant to ask, but worth noting that your answering a slightly different question. Still a +1 answer. — John M, May 21 '09 at 21:51
@John M - I guess the question is a bit vague - to what purpose does @debugger want to intercept the system calls? If it is to modify behaviour, then the above will work, if it is for tracing purposes, then use truss/strace/dtrace or the AIX equivalent, ... Could get all existential and debate "what is a system call". Thanks for the feedback! — Beano, May 22 '09 at 08:30

score 1 · Answer 2 · answered May 22 '09 at 14:49

1

Well, there's always systrace.

answered May 22 '09 at 14:49

Nikolai Fetissov

82,306
11
110
171

score 0 · Answer 3 · edited Jun 30 '17 at 04:52

I'm not sure about AIX, but I've done it on Linux. On Linux, the system call table is contained in the sys_call_table array. We need to first find out the address of this table. Now, this is a tricky thing and there are multiple ways to do it.

We can find its address by looking at sysmap file:

punb200m2labs08vm1:/ # cat /boot/System.map-4.4.21-69-default | grep sys_call_table
ffffffff81600180 R sys_call_table

Hence, ffffffff81600180 is the address of sys_call_table on my machine. In your kernel module, you can just change the default function corresponding to certain system call number (that you're changing) and assign it to your own function.

e.g. Suppose you want to intercept 'open' system call whose number is __NR_open on Linux. After you get the sys_call_table address from above, just assign your function to index __NR_open of sys_call_table:

sys_call_table[__NR_open] = your_function;

where your_function is implemented by you to intercept 'open' system call. From now on, every open system call will go through this function.

The details would differ on AIX, but the overall idea would be similar, I guess. You just need to find out AIX specific procedure to achieve this.

What are the possible ways of intercepting system calls on unix environments?

3 Answers3