30

For my dissertation at University, I'm working on a coding leaderboard system where users can compile / run untrusted code through temporary docker containers. The system seems to be working well so far, but one problem I'm facing is that when code for an infinite loop is submitted, E.g:

while True:
   print "infinite loop"

the system goes haywire. The problem is that when I'm creating a new docker container, the Python interpreter prevents docker from killing the child container as data is still being printed to STDOUT (forever). This leads to the huge vulnerability of docker eating up all available system resources until the machine using the system completely freezes (shown below):

enter image description here

So my question is, is there a better way of setting a timeout on a docker container than my current method that will actually kill the docker container and make my system secure (code originally taken from here)? 

#!/bin/bash
set -e

to=$1
shift

cont=$(docker run --rm "$@")
code=$(timeout "$to" docker wait "$cont" || true)
docker kill $cont &> /dev/null
echo -n 'status: '
if [ -z "$code" ]; then
    echo timeout
else
    echo exited: $code
fi

echo output:
# pipe to sed simply for pretty nice indentation
docker logs $cont | sed 's/^/\t/'

docker rm $cont &> /dev/null

Edit: The default timeout in my application (passed to the $to variable) is "10s" / 10 seconds.

I've tried looking into adding a timer and sys.exit() to the python source directly, but this isn't really a viable option as it seems rather insecure because the user could submit code to prevent it from executing, meaning the problem would still persist. Oh the joys of being stuck on a dissertation... :(

Abdullah Jibaly
  • 53,220
  • 42
  • 124
  • 197
Joel Murphy
  • 2,472
  • 3
  • 29
  • 47
  • 2
    I might be missing something, but why can't you do something like: `ID=$(docker run -d unsecure) && docker run --rm cleanup $ID`, where cleanup is a container that kills another container after X seconds? – Abdullah Jibaly Mar 09 '15 at 05:26
  • "forever" -- let's just try decreasing the stdout buffer size. What state is your containerized-`python` process in after you send the signal? – Brian Cain Mar 10 '15 at 02:44
  • You could set a limit on the cpu time using python with the resources lib, a python script can be checked and changed at runtime using ast – Padraic Cunningham Sep 07 '15 at 16:57
  • Can you provide example commands: docker run --rm=true -ti -v "$PWD":/usr/src/myapp -w /usr/src/myapp python:2 python your-daemon-or-script.py . I don't understand when you take decision to kill it? – Valeriy Solovyov Sep 12 '15 at 04:47

6 Answers6

3

You could set up your container with a ulimit on the max CPU time, which will kill the looping process. A malicious user can get around this, though, if they're root inside the container.

There's another S.O. question, "Setting absolute limits on CPU for Docker containers" that describes how to limit the CPU consumption of containers. This would allow you to reduce the effect of malicious users.

I agree with Abdullah, though, that you ought to be able to docker kill the runaway from your supervisor.

Community
  • 1
  • 1
Nathaniel Waisbrot
  • 23,261
  • 7
  • 71
  • 99
2

If you want to run the containers without providing any protection inside them, you can use runtime constraints on resources.

In your case, -m 100M --cpu-quota 50000 might be reasonable.

That way it won't eat up the parent's system resources until you get around to killing it.

ctolsen
  • 121
  • 1
1

I have achieved a solution for this problem.

First you must kill docker container when time limit is achieved:

#!/bin/bash
set -e
did=$(docker run -it -d -v "/my_real_path/$1":/usercode virtual_machine ./usercode/compilerun.sh 2>> $1/error.txt)
sleep 10 && docker kill $did &> /dev/null && echo -n "timeout" >> $1/error.txt &
docker wait "$did" &> /dev/null
docker rm -f $ &> /dev/null

The container runs in detached mode (-d option), so it runs in the background. Then you run sleep also in the background. Then wait for the container to stop. If it doesnt stop in 10 seconds (sleep timer), the container will be killed.

As you can see, the docker run process calls a script named compilerun.sh:

#!/bin/bash
gcc -o /usercode/file /usercode/file.c 2> /usercode/error.txt && ./usercode/file < /usercode/input.txt | head -c 1M > /usercode/output.txt
maxsize=1048576
actualsize=$(wc -c <"/usercode/output.txt")
if [ $actualsize -ge $maxsize ]; then
    echo -e "1MB file size limit exceeded\n\n$(cat /usercode/output.txt)" > /usercode/output.txt
fi

It starts by compiling and running a C program (its my use case, I am sure the same can be done for python compiller).

This part:

command | head -c 1M > /usercode/output.txt

Is responsible for the max output size thing. It allows output to be 1MB maximum.

After that, I just check if file is 1MB. If true, write a message inside (at the beginning of) the output file.

Pablo Werlang
  • 399
  • 3
  • 10
1

The --stop-timeout option is not killing the container if the timeout is exceeded.

Instead, use --ulimit --cpu=timeout to kill the container if the timeout is exceeded. This is based on the CPU time for the process inside the container.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
Ramki T R
  • 9
  • 1
0

I guess, you can use signals in python like unix to set timeout. you can use alarm of specific time say 50 seconds and catch it. Following link might help you. signals in python

Nitin Tripathi
  • 1,224
  • 7
  • 17
-1

Use --stop-timeout option while running your docker container. this will execute SIGKILL once the timeout occured

Ramki T R
  • 9
  • 1