0

I am trying to develop a framework that will compile and execute (mostly random) C++ and Java packages.

However, given their random nature, I want to check the source (or the executable -- pre-execution) for any linux system calls before execution. If there is such a system call, I don't want to execute the program.

It is safe to assume that these packages wouldn't need to make any system calls to fulfill their functional purpose (they're not complex packages).

Edit: A bash command/script would be simplest, but any answer is fine.

jab
  • 5,673
  • 9
  • 53
  • 84
  • 4
    I believe [`grep`](http://en.wikipedia.org/wiki/Grep) is what you're looking for. – 101010 Aug 17 '14 at 18:55
  • 3
    If someone wants to intentionally hide a system call, I doubt you can do anything to prevent that. You should look into (at the very least) running the programs in a chroot jail, with limited (or no) libraries available. Also, check out [this question](http://stackoverflow.com/questions/69859/how-could-i-intercept-linux-sys-calls). – hyde Aug 17 '14 at 18:58
  • @40two Yeah I was thinking it would be a simple string search. – jab Aug 17 '14 at 19:00
  • 1
    Do you suppose that the author of the C code is not malicious? – Basile Starynkevitch Aug 17 '14 at 19:00
  • What about the case where a define has been made to hide the system call, what about the cases where `system` is not used, but any other `exec` method is used, what about the cases where execution string, is calculated at runtime? – Tommy Andersen Aug 17 '14 at 19:08
  • 1
    If you are serious about this, you could learn how to use a custom libc and thus prevent system calls at link-time. See http://stackoverflow.com/questions/10763394/how-to-build-a-c-program-using-a-custom-version-of-glibc. This is going to be very hard, though. – Christian Hackl Aug 17 '14 at 19:28
  • I wonder how well would intentional obfuscation work if he compiled the code and looked for the relevant library calls or syscalls in the binary dump – Leeor Aug 17 '14 at 19:34

2 Answers2

4

In short, you cannot detect reliably all malicious syscalls (by static analysis of source code); read about the halting problem and Rice theorem... BTW MELT would be slighty better than grep since it works on GCC gimple representation.

Think of (on Linux)

  • dlopen(3)-ing the libc (or the main executable) then dlsym-ing "system" to get a pointer to the system function
  • knowing the libc layout and version,, then computing system's address by adding some known offset to address of malloc
  • using some JIT libary, e.g. the header only GNU lightning
  • coding the eqivalent of system with fork and execve ....
  • etc....

Of course, you might be trusting your user (I won't do that for a web application). If you trust all your users and just want to detect mistakes you might be able to filter some of them.

You need some container, e.g. docker

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
2

Look in resource limits (setrlimit if you are on POSIX system) as opposed to trying to find the malicious code.

You can limit number of processes, memory, open files, cputime and others. I would suggest you to limit basically everything. And run in chroot jail (even an empty one if you link statically).

virco
  • 257
  • 2
  • 4