prevent java program from opening threads

Question

I'm trying to run a java application , more specifically a jar compiled one, using execve() in c
something like that:

char *cmd[] = {"a.jar"};
execve("a.jar",cmd,NULL);

that is working OK but when I try to limit the number of threads that this program can open using something like that:

struct rlimit rlp;
rlp.rlim_cur = rlp.rlim_max = limit_nproc; 
setrlimit(RLIMIT_NPROC,&rlp);

I have a problem with the JVM which would open threads and I'm preventing that so I have this error:

java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.

how can I prevent the threads opened in the java application but not those opened by the JVM ? !

please notice , the question is how to prevent user threads but not system threads , I need a restriction to the running environment like what i did in my second code "RLIMIT_NPROC"

and Thanks!

Answer 1

This can be achieved with JVMTI agent.

The idea is to intercept native Thread.start0() method and throw an exception whenever it is called.

Here is a sample agent written in C++:

#include <jvmti.h>

// Original native implementation of Thread.start0(), if you wish to call it
extern "C" void JNICALL JVM_StartThread(JNIEnv* env, jthread thread);

void JNICALL StartThreadHook(JNIEnv* env, jthread thread) {
    env->ThrowNew(env->FindClass("java/lang/Error"), "Threads forbidden");
}

void JNICALL VMInit(jvmtiEnv* jvmti, JNIEnv* env, jthread thread) {
    // After VM is initialized, intercept Thread.start0() with our hook function
    jclass thread_class = env->FindClass("java/lang/Thread");
    JNINativeMethod start0 = {(char*)"start0", (char*)"()V", (void*)StartThreadHook};
    env->RegisterNatives(thread_class, &start0, 1);
}

JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM* vm, char* options, void* reserved) {
    jvmtiEnv* jvmti;
    vm->GetEnv((void**)&jvmti, JVMTI_VERSION_1_0);

    jvmtiEventCallbacks callbacks = {0};
    callbacks.VMInit = VMInit;
    jvmti->SetEventCallbacks(&callbacks, sizeof(callbacks));
    jvmti->SetEventNotificationMode(JVMTI_ENABLE, JVMTI_EVENT_VM_INIT, NULL);

    return 0;
}

Compile the agent:

g++ -fPIC -shared -olibnothreads.so -Wl,-soname,libnothreads.so nothreads.cpp

Run the application with the agent:

java -agentpath:/path/to/libnothreads.so -jar app.jar

Note that you may also use JVMTI to implement custom logic when to allow and when to deny starting new threads. For example, ThreadStart and ThreadEnd events will help to count created threads. GetStackTrace function will help to find which classes are trying to create a thread.

Answer 2

how can I prevent the threads opened in the java application but not those opened by the JVM ?

I'm not sure you can. Preventing the JVM from creating threads is like saying that you want to limit the number of Strings that it creates. If the code creates a thread then there is nothing you can do about it.

About the only thing that might help is the security policy, but as I read it, thread creation is not controlled. See Java's docs on the permissions under control.

java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.

As you probably know, in addition to the "main" thread, Java starts a number of other JVM specific threads that work in the background. For example, a simple main(String[] args) program starts "main" and an additional 5 threads for me which doesn't included the GC thread(s) I believe. The JVM threads are needed for memory management and other important tasks. If you are limiting threads up to a point where the GC thread(s) can't start then the JVM is not going to be able to run at all.

<HACK> One thing that you could do is to limit the number of threads to the precise number to include the system threads and "main". Inside of main(), before the user code has an opportunity to start more threads, you could count the number of running threads using Thread.getAllStackTraces().size() or and then set your OS limits to that number. If this still fails then try adding 1 or 2 to the size() to account for other background threads not accounted for in the stack trace map. </HACK>

All this said, the question I have is what are you trying to accomplish? If you are worried about a Java process taking over your server then I wonder if there are OS settings that would be better in controlling how many system resources are given to the process. How about limiting the concurrency instead of the number of threads. Maybe look for thread affinity settings? This would be very OS dependent however.