How to prevent a Java application from executing processes on GNU/Linux?

Question

In other words, are modern GNU/Linux or JVM (ideally Java 11+) able to prevent a Java process from executing other processes?

Issues such as Log4Shell and Spring4Shell seem possible because the JVM allows a Java application to execute other programs.

At the OS level, I imagine it may be possible with some process isolation, jails etc. Are there complete examples about that?

How about the JVM itself? With the Security Manager gone in recent versions, what options exist to harden a Java Virtual Machine?

Have a look at https://docs.oracle.com/javase/tutorial/essential/environment/security.html — g00se, Apr 09 '22 at 20:38
@g00se Thank you. It is deprecated and disabled by default in recent versions. Are there good replacements yet? — pyb, Apr 09 '22 at 20:43
@g00se JEP 411, adding the link to my question https://openjdk.java.net/jeps/411 — pyb, Apr 09 '22 at 20:51
Ugh, you would be hoping that they'd replace that with a more dynamic framework. OTOH, I can understand the problematic integration. Maybe some kind of injection framework could work. — Maarten Bodewes, Apr 10 '22 at 11:13

score 1 · Answer 1 · answered Apr 11 '22 at 06:58

I'm not sure there's a good way to do that. As discussed in Prevent forking in a child process, you can set RLIMIT_NPROC limit.

However, don't do that for your java app - you will break it. Threads are just a special type of a process on Linux and if you restrict nproc you won't be able to create new threads.

Observe this behavior on a running java application

prlimit
...
NPROC      max number of processes                15107     15107 processes

prlimit --pid $(pgrep java) --nproc=10

Then on the stdout of the java app

[9816123.415s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.

I tried this for a production app running in a docker container.

Note that nproc limit isn't enforced if you the process is running as root.

Morever, it's quite easy to reproduce it with a simple Clojure REPL (or JShell variant of it):

# start a clojure/java process
user=> (require '[clojure.java.shell :as sh])

user=> (dotimes [i 100] (future (sh/sh "sleep" "2")))
nil


# then get change the nproc limit for the process - make sure to pass proper pid
prlimit --pid 2769227 --nproc=99

# then again in the Clojure REPL: 
user=> (dotimes [i 100] (future (sh/sh "sleep" "2")))
[200.656s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[200.656s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[200.657s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[200.657s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[200.657s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Execution error (OutOfMemoryError) at java.lang.Thread/start0 (Thread.java:-2).
unable to create native thread: possibly out of memory or process/resource limits reached

Resources:

Prevent forking in a child process: Prevent forking in a child process
Does nproc in limits.conf refers to number of processes or number of threads? https://unix.stackexchange.com/questions/322056/does-nproc-in-limits-conf-refers-to-number-of-processes-or-number-of-threads
Multiple Java webapps total thread limit on linux Multiple Java webapps total thread limit on linux
prlimit: https://manpages.ubuntu.com/manpages/xenial/man1/prlimit.1.html

Thank you! I wonder if this could work by creating a thread pool that would "consume" RLIMIT_NPROC on startup. — pyb, Apr 11 '22 at 13:03
Perhaps it's possible to allow Java to fork itself while preventing it from running other executables? — pyb, Apr 11 '22 at 13:04
I don't think you can work-around the nproc limit to allow creating threads but preventing processes. Moreover, there are also internal threads managed by JVM and not necessarily just your own program code. For anything but a trivial application this might become a nightmare. — Juraj Martinka, Apr 12 '22 at 08:23

How to prevent a Java application from executing processes on GNU/Linux?

1 Answers1