3

I want to redefine the bytecode of the StackOverflowError constructor so I have a "hook" for when a stack overflow occurs. All I want to do is insert a single method call to a static method of my choosing at the start of the constructor. Is it possible to do this?

0xbe5077ed
  • 4,565
  • 6
  • 35
  • 77
  • 1
    It's possible using the [-Xbootclasspath](http://docs.oracle.com/javase/7/docs/technotes/tools/windows/java.html) flag, but the license says you cannot distribute an app that uses that to override a JRE class. So as long as you don't distribute the app... – vanza Sep 16 '14 at 01:38
  • @vanza, thanks. I definitely need to distribute it though so that won't work. – 0xbe5077ed Sep 16 '14 at 01:41

2 Answers2

2

You should be able to do it using one of two ways (unless something changed in the last 1-2 years, in which case I'd love some links to changelogs/docs):

  1. Mentioned in a comment, not very feasible I guess, modify the classes you are interested in, put them in a jar and then use the -bootclasspath option to load them instead of the default ones. As was mentioned before this can have some legal issues (and is a pain to do in general).

  2. You should be able to (or at least you used to be able to) instrument almost all core classes (iirc Class was the only exception I've seen). One of many problems you might have is the fact that many of core classes are being initialized before the agents you provide (or well their premain methods to be exact) are consulted. To overcome this you will have to add Can-Retransform-Classes property to your agent jar and then re-transform the classes you are interested in. Be aware that re-transformation is a bit less powerful and doesn't give you all the options you'd have normally with instrumentation, you can read more about it in the doc.

I am assuming you know how to do instrumentation?

Mateusz Dymczyk
  • 14,969
  • 10
  • 59
  • 94
  • Thanks @Mateusz Dymczyk. I've never actually instrumentation. I've done bytecode generation with ASM so I assume it's just a matter of: get the class bytes, "echo" them through an ASM ClassReader/ClassVisitor and when you hit the method(s) you want to change, insert the extra method call... I assume it's possible to modify package-protected and private methods in this way? – 0xbe5077ed Sep 16 '14 at 06:31
  • @0xbe5077ed: yes, you can do it this way. – Holger Sep 16 '14 at 09:41
1

There are several things to consider.

  • It is possible to redefine java.lang.StackOverflowError. I tried it successfully on 1.7.0_40. isModifiableClass(java.lang.StackOverflowError.class) return true and I successfully redefined it inserting a method invocation into all of its constructors
  • You should be aware that when you insert a method call into a class via Instrumentation you still have to obey the visibility imposed by the ClassLoader relationships. Since StackOverflowError is loaded by the bootstrap loader it can only invoke methods of classes loaded by the bootstrap loader. You would have to add the target method’s class(es) to the bootstrap loader
  • This works if the application’s code throws a StackOverflowError manually. However, when a real stackoverflow occurs, the last thing the JVM will do is to invoke additional methods (keep in mind what the error says, the stack is full). Consequently it creates an instance of StackOverflowError without calling its constructor (a JVM can do that). So your instrumentation is pointless in this situation.
  • As already pointed out by others, a “Pure Java Application” must not rely on modified JRE classes. It is only valid to use Instrumentation as add-on, i.e. development or JVM management tool. You should keep in mind that the fact that Oracle’s JVM 1.7.0_40 supports the redefinition of StackOverflowError does not imply that other versions or other JVMs do as well.
Holger
  • 285,553
  • 42
  • 434
  • 765