7

A bug in a third party library is causing an infinite loop in a worker thread on a JBoss instance of mine. Do you know of a way to kill this "stuck" thread without restarting the server? We'd like to be able to recover from this until a fix is deployed, preferably without having to restart.

I've seen a few people mention using Thread.interrupt() - if I were to code my own MBean, how would I get a handle to the thread in question in order to interrupt it?

Update: Wasn't able to solve using any of these methods. I did come across another thread about the same issue that had a link to why Thread.stop() is deprecated. Someone else has asked a similar question with similar results. It seems like more sophisticated containers should provide this kind of health mechanism, but I guess their hands are tied w/r/t the JVM.

Community
  • 1
  • 1
cwash
  • 4,185
  • 5
  • 43
  • 53

4 Answers4

2

I had a similar bug (infinite loop) in a 3rd party lib. I ended up applying the fix myself (while waiting for the people from the 3rd party lib to fix their mess) and then I placed the modified .class in my .war, making sure it is loaded before the bogus .class (the bogus one being inside the bogus 3rd party .jar).

It is not nice but it works, see my question here:

Order of class loading from a .war file

What I mean is this: if you have to wait for the people responsible for the 3rd party bugged lib to fix their stuff, you can potentially be waiting a very long time. We couldn't afford that. We needed a fix ASAP. So we ended up applying a patch/hack to their code.

You could for example add a boolean check inside the infinite loop and then forcing the loop to exit when you want the bogus thread to "die".

Note that I haven't used the deprecated Thread stop() since ten years and I really didn't want to use it in the above case.

Community
  • 1
  • 1
SyntaxT3rr0r
  • 27,745
  • 21
  • 87
  • 120
  • Agreed re: not waiting on the third party developers to fix it, I was only talking about waiting in the very short term (next few days) to get our patch vetted through testing. Thanks for they options for a programmatic fix, the sound like good approaches - I'm really looking for a built in recovery option. I can't believe there's no way to kill a single Thread you know is malignant! – cwash Jan 21 '11 at 20:47
2

I suppose the most difficult part is to identify the hanging thread. You provide no info about it, but perhaps you can build some rules around the thread's name or its current stack trace.

If you can identify the thread by its name, I would get all threads in the VM by getting my own thread group with Thread.currentThread().getThreadGroup(), then walk up the thread group hierarchy by calling getParent() on the thread group until it returns null. You now have the top level thread group. You can now fill a preallocated array with all threads using the enumerate(Thread[] list) method on the top level thread group.

If you need the stack traces anyway to identify the thread, you can also use the static utility method Map<Thread,StackTraceElement[]> Thread.getAllStackTraces() to get all threads. Computing the stack traces is however quite expensive, so this might not be the best solution if you don't actually need them.

After identifying the thread you must call the stop() method on it. Interrupting it won't help, unless the implementation of the running code actually evaluates the thread's interrupted flag and behaves as you expect it to. Not that the stop() method is deprecated and that using it may have many funny side effects. You can find more details in the API documentation.

jarnbjo
  • 33,923
  • 7
  • 70
  • 94
  • I've identified it through a ThreadDump in VisualVM. I know exactly what it is - getting it in the code is another matter, though. The method you propose to get the threadGroup and then walk up sounds feasible, but I need an actual instance handle to the thread to call stop() or interrupt() on it. – cwash Jan 21 '11 at 20:42
  • @cwash - the enumerate method returns actual thread instances. the passed in array is populated with the with all the Thread instances found. if you know the thread id, you can then walk this array looking for the Thread with the matching id. Note that the enumerate() method has weird semantics (it does not tell you if you overflow), so getting it all to work is tricky. note, your can use ThreadGroup.getActiveCount() for a first guess at the array size. – jtahlborn Jan 21 '11 at 21:29
  • @cwash - see some examples here: http://nadeausoftware.com/articles/2008/04/java_tip_how_list_and_find_threads_and_thread_groups – jtahlborn Jan 21 '11 at 21:31
1

You could use the discouraged myThread.stop() method. But then it is very likely the Thread is still referenced there, so you should use some reflection magic to remove all references to this thread from the components holding it.

How to find the Thread? Use Thread.getThreadGroup() and ThreadGroup.getThreadGroup() to go up to the root ThreadGroup(), and then use the iterate() functions to go through all threads.

Daniel
  • 27,718
  • 20
  • 89
  • 133
  • Sounds feasible, but I guess I'd have to try it. Don't know of anything that already looks a Thread up by name? Have you ever used the iterate() to sort through threads? – cwash Jan 21 '11 at 20:44
0

Try my jkillthread which tries to do something like this.

Jesse Glick
  • 24,539
  • 10
  • 90
  • 112