3

I want to make sure the process gets killed after 10 seconds of CPU time. Docker run command accepts the flag --ulimit cpu=10 that is supposed to do that.

However when I run java command using this, the ulimit setting is ignored. The java process with infinite loop continues even after 10s (actually for minutes until I kill it) Here is the command I used to test.

docker run --rm -i -v /usr/local/src:/classes --ulimit cpu=10 java:8 \
java -cp /classes/ InfiniteLoop

Instead of invoking java directly, if I start bash and then run java c, it works as expected.

docker run --rm -i -v /usr/local/src:/classes --ulimit cpu=10 java:8 \
bash -c 'date; java -cp /classes/ InfiniteLoop'

Why does invoking java program directly does not respect ulimit option?

Edit 1:

$ docker --version
Docker version 1.9.1, build a34a1d5

The java program is, InfiniteLoop.java

import java.util.*;

class InfiniteLoop {
  public static void main(String[] args) throws Exception {
    for (long i = 0; i < 1000_000_000_000L; i++) {
      if (i % 1_000_000_000 == 0) {
        System.out.println(new Date() + ", i = " + i);
      }
    }
  }
}

Edit 2: The following doesn't work either. That is, with only java executed in the bash.

docker run --rm -i -v /usr/local/src:/classes --ulimit cpu=10 java:8 \
bash -c 'java -cp /classes/ InfiniteLoop'

But, adding any noop or ':' command works. Or even an arbitrary word that prints "command not found" also works.

docker run --rm -i -v /usr/local/src:/classes --ulimit cpu=10 java:8 \
bash -c ':; java -cp /classes/ InfiniteLoop'

and this works too.

docker run --rm -i -v /usr/local/src:/classes --ulimit cpu=10 java:8 \
bash -c 'ArbirtraryCommandNotFound; java -cp /classes/ InfiniteLoop'

Edit 3: Similar to using the no-op (:), invoking the process with time also makes the process to be killed exactly after the CPU time is exceeded. 

docker run --rm -i -v /usr/local/src:/classes --ulimit cpu=10 java:8 \
bash -c 'time java -cp /classes/ InfiniteLoop'
JackDaniels
  • 985
  • 9
  • 26
  • This looks suspiciously similar to what has has been reported in a GitHub issue earlier this year: https://github.com/docker/docker/issues/1905 – Michael Hausenblas Nov 27 '16 at 08:58
  • 1
    Because, it is the same, and I still haven't got the answer. It would be great, if you knew the answer, and can help me understand. – JackDaniels Nov 27 '16 at 09:21
  • I'm having hard times to reproduce it since I don't know which Docker version you're using nor do I have access to your Java source files. – Michael Hausenblas Nov 27 '16 at 09:37
  • 1
    Added the java source code, docker version, etc. Another thing I found, in bash command if I put any other commands before java, then ulimit works, otherwise it doesn't. – JackDaniels Nov 27 '16 at 10:02
  • if you run `top`, it should list how much cpu time your java process has taken. An infinite loop printing the date, I would guess, doesn't peg the CPU and would likely require far more clock time than 10 seconds to get to 10 seconds of cpu time. – programmerq Nov 27 '16 at 15:50
  • @programmerq I checked with top The program consistently took 90+% CPU and had TIME column showed more than 50seconds, until I killed the process. It also doesn't explain why running a no-op before invoking java works. – JackDaniels Nov 28 '16 at 07:02
  • So I can reproduce it now on Docker 1.11.2 but still dunno why. Will keep digging until I find the reason. – Michael Hausenblas Nov 28 '16 at 07:05
  • @MichaelHausenblas Thanks. If it helps, I see the same behavior even when running C++ programs. So, it was nothing specific to java – JackDaniels Nov 28 '16 at 07:49
  • @JackDaniels aha! that is a useful information, thanks! I'm investigating with https://github.com/mhausenblas/cinf now to figure if there's something funny going on – Michael Hausenblas Nov 28 '16 at 08:19

1 Answers1

1

After some experimenting I re-read the original question and also took into account the fact that it is independent of the type of program being launched, that is, Java, C++, etc.: the reason why it works in the one case (when invoked with bash -c) and not when you directly invoke it is that ulimit is a bash built in command and the docs for docker run are not entirely transparent about it.

Michael Hausenblas
  • 13,162
  • 4
  • 52
  • 66
  • Thanks. Does that mean ulimit will never work when starting any programs other than with bash? It also doesn't explain why 'bash -c' also works only when there are already two commands (even if the first one is an invalid command)? – JackDaniels Dec 02 '16 at 11:46
  • Yes, ulimit only works with bash but you can use [cgroups](http://containerz.info/) as a generic solution. – Michael Hausenblas Dec 02 '16 at 13:59
  • with cgroups, I can only specify cpu_quota per cpu_period, I want to provide an upper limit on total cpu usage. That aside, I can understand why ulimit will work only in bash, but even in bash it doesn't work all the time. It works only if there are atleast 2 commands, even if the extra command is invalid or when invoked with time. – JackDaniels Dec 03 '16 at 07:42
  • I'm not following @JackDaniels you can totally do that with [cpu.cfs_quota_us](https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt). The other observation is correct, I think, and just means that the Docker implementation is buggy ;) – Michael Hausenblas Dec 03 '16 at 08:03
  • The documentation says, cpu.cfs_quota_us: the total available run-time within a period (in microseconds) cpu.cfs_period_us: the length of a period (in microseconds) When quota exceeds, the process gets throttled, not killed. (Whereas when setting cpu, it gets killed.) – JackDaniels Dec 03 '16 at 09:44
  • Right you are. OK, I also checked the other [cpu](https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt) cgroups docs but seems the only way to emulate ulimit exactly is using the [setrlimit](http://www.informit.com/articles/article.aspx?p=23618&seqNum=6) syscall directly. – Michael Hausenblas Dec 04 '16 at 07:53