7

I finally wrote a Java agent with Byte Buddy which uses the Advice API to print a message upon entering and leaving a method. With my current configuration, this agent appears to apply on classes loaded by the Application ClassLoader only.

However, I would like it to apply also to classes loaded by any classloader. I have come across multiple techniques (see enableBootstrapInjection() or ignore()) which does not seem to work. Indeed, enableBootstrapInjection() has disappeared from ByteBuddy, and the ignore() method makes my JVM panic, as I believe I have circular issues like trying to instrument the java.lang.instrument class (but this does not seem to be the only issue, and I cannot find a way to list those errors).

Here is a simplified version of my agent:

AgentBuilder mybuilder = new AgentBuilder.Default()
        .ignore(nameStartsWith("net.bytebuddy."))
        .disableClassFormatChanges()
        .with(RedefinitionStrategy.RETRANSFORMATION)
        .with(InitializationStrategy.NoOp.INSTANCE)
        .with(TypeStrategy.Default.REDEFINE);
mybuilder.type(nameMatches(".*").and(not(nameMatches("^src.Agent")))) // to prevent instrumenting itself
        .transform((builder, type, classLoader, module) -> {
            try {
                return builder
                .visit(Advice.to(TraceAdvice.class).on(isMethod()));
                } catch (SecurityException e) {
                    e.printStackTrace();
                    return null;
                }
            }
        ).installOn(inst);
System.out.println("Done");

and a simplified version of my Advice class, if necessary :

public class TraceAdvice {
    @Advice.OnMethodEnter
    static void onEnter(
        @Origin Method method,
        @AllArguments(typing = DYNAMIC) Object[] args
    ) {
        System.out.println("[+]");
    }

    @Advice.OnMethodExit
    static void onExit() {
        System.out.println("[-]");
    }
}

I am conscious about the circular dependency of instrumenting java.io.PrintStream.println for instance, I could just ignore such methods (with .and(not(nameMatches("^java.io.PrintStream"))) on line 7 for example).

AntoineG
  • 93
  • 7

2 Answers2

6

Here is how you can activate logging and get helpful log output. I am also showing you how to manually retransform an already loaded bootstrap class. Bootstrap classes which are loaded after installing the transformer, will automatically be transformed, as you can also see in the log below.

import net.bytebuddy.agent.ByteBuddyAgent;
import net.bytebuddy.agent.builder.AgentBuilder;
import net.bytebuddy.asm.Advice;

import java.lang.instrument.Instrumentation;
import java.lang.instrument.UnmodifiableClassException;
import java.lang.reflect.Method;
import java.util.Properties;

import static net.bytebuddy.agent.builder.AgentBuilder.RedefinitionStrategy.RETRANSFORMATION;
import static net.bytebuddy.matcher.ElementMatchers.*;

class ByteBuddyInstrumentBootstrapClasses {
  public static void main(String[] args) throws UnmodifiableClassException {
    Instrumentation instrumentation = ByteBuddyAgent.install();
    installTransformer(instrumentation);

    // Use already loaded bootstrap class 'Properties'
    System.out.println("Java version: " + System.getProperties().getProperty("java.version"));
    // Retransform already loaded bootstrap class 'Properties'
    instrumentation.retransformClasses(Properties.class);
    // Use retransformed bootstrap class 'Properties' (should yield advice output)
    System.out.println("Java version: " + System.getProperties().getProperty("java.version"));
  }

  private static void installTransformer(Instrumentation instrumentation) {
    new AgentBuilder.Default()
      .disableClassFormatChanges()
      .with(RETRANSFORMATION)
      // Make sure we see helpful logs
      .with(AgentBuilder.RedefinitionStrategy.Listener.StreamWriting.toSystemError())
      .with(AgentBuilder.Listener.StreamWriting.toSystemError().withTransformationsOnly())
      .with(AgentBuilder.InstallationListener.StreamWriting.toSystemError())
      .ignore(none())
      // Ignore Byte Buddy and JDK classes we are not interested in
      .ignore(
        nameStartsWith("net.bytebuddy.")
          .or(nameStartsWith("jdk.internal.reflect."))
          .or(nameStartsWith("java.lang.invoke."))
          .or(nameStartsWith("com.sun.proxy."))
      )
      .disableClassFormatChanges()
      .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION)
      .with(AgentBuilder.InitializationStrategy.NoOp.INSTANCE)
      .with(AgentBuilder.TypeStrategy.Default.REDEFINE)
      .type(any())
      .transform((builder, type, classLoader, module) -> builder
        .visit(Advice.to(TraceAdvice.class).on(isMethod()))
      ).installOn(instrumentation);
  }

  public static class TraceAdvice {
    @Advice.OnMethodEnter
    static void onEnter(@Advice.Origin Method method) {
      // Avoid '+' string concatenation because of https://github.com/raphw/byte-buddy/issues/740
      System.out.println("[+] ".concat(method.toString()));
    }

    @Advice.OnMethodExit
    static void onExit(@Advice.Origin Method method) {
      // Avoid '+' string concatenation because of https://github.com/raphw/byte-buddy/issues/740
      System.out.println("[-] ".concat(method.toString()));
    }
  }
}

Console log:

[Byte Buddy] BEFORE_INSTALL net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport@2a54a73f on sun.instrument.InstrumentationImpl@16a0ee18
[Byte Buddy] TRANSFORM com.sun.tools.attach.VirtualMachine [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, module jdk.attach, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM ByteBuddyInstrumentBootstrapClasses [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, unnamed module @4e07b95f, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM com.intellij.rt.execution.application.AppMainV2$1 [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, unnamed module @4e07b95f, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM com.intellij.rt.execution.application.AppMainV2 [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, unnamed module @4e07b95f, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM com.intellij.rt.execution.application.AppMainV2$Agent [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, unnamed module @4e07b95f, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM sun.text.resources.cldr.ext.FormatData_de [jdk.internal.loader.ClassLoaders$PlatformClassLoader@7203c7ff, module jdk.localedata, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM sun.util.resources.provider.LocaleDataProvider [jdk.internal.loader.ClassLoaders$PlatformClassLoader@7203c7ff, module jdk.localedata, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM sun.util.resources.provider.NonBaseLocaleDataMetaInfo [jdk.internal.loader.ClassLoaders$PlatformClassLoader@7203c7ff, module jdk.localedata, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM sun.util.resources.cldr.provider.CLDRLocaleDataMetaInfo [jdk.internal.loader.ClassLoaders$PlatformClassLoader@7203c7ff, module jdk.localedata, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formattable [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$Conversion [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$Flags [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$FormatSpecifier [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$FixedString [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$FormatString [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.regex.ASCII [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.regex.IntHashSet [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.regex.Matcher [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.regex.MatchResult [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.lang.CharacterData00 [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.lang.StringUTF16$CharsSpliterator [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.stream.IntPipeline$9$1 [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.stream.Sink$ChainedInt [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] INSTALL net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport@2a54a73f on sun.instrument.InstrumentationImpl@16a0ee18
Java version: 14.0.2
[Byte Buddy] TRANSFORM java.util.Properties [null, module java.base, Thread[main,5,main], loaded=true]
[+] public java.lang.String java.util.Properties.getProperty(java.lang.String)
[-] public java.lang.String java.util.Properties.getProperty(java.lang.String)
Java version: 14.0.2
[Byte Buddy] TRANSFORM java.util.IdentityHashMap$IdentityHashMapIterator [null, module java.base, Thread[main,5,main], loaded=false]
[Byte Buddy] TRANSFORM java.util.IdentityHashMap$KeyIterator [null, module java.base, Thread[main,5,main], loaded=false]
[+] public boolean java.util.IdentityHashMap$IdentityHashMapIterator.hasNext()
[-] public boolean java.util.IdentityHashMap$IdentityHashMapIterator.hasNext()
[+] public java.lang.Object java.util.IdentityHashMap$KeyIterator.next()
[+] protected int java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex()
[-] protected int java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex()
[-] public java.lang.Object java.util.IdentityHashMap$KeyIterator.next()
[+] public boolean java.util.IdentityHashMap$IdentityHashMapIterator.hasNext()
[-] public boolean java.util.IdentityHashMap$IdentityHashMapIterator.hasNext()
[Byte Buddy] TRANSFORM java.lang.Shutdown [null, module java.base, Thread[DestroyJavaVM,5,main], loaded=false]
[Byte Buddy] TRANSFORM java.lang.Shutdown$Lock [null, module java.base, Thread[DestroyJavaVM,5,main], loaded=false]
[+] static void java.lang.Shutdown.shutdown()
[+] private static void java.lang.Shutdown.runHooks()
[-] private static void java.lang.Shutdown.runHooks()
[-] static void java.lang.Shutdown.shutdown()

Please note how the first call of System.getProperties().getProperty("java.version") does not yield advice logging, but the second call after retransformation does.

Update after taking a look at your GitHub repository:

Am I understanding correctly? Module launcher tries to dynamically attach module agent to another, already running JVM. This looks complicated. Did you try with starting the other JVM with a -javaagent:/path/to/agent.jar parameter? You can still try the other strategy later. But either way, please note that your agent classes Agent and CompleteSTE will not be on the boot classpath like this.

Given the fact that advice code will be inlined into target classes (also bootstrap classes), it means that the bootstrap classes need to be able to find all classes referenced by the advice code on the boot classpath. There are two ways to achieve that:

  1. Add -Xbootclasspath/a:/path/to/agent.jar to the target JVM command line in addition to -javaagent:/path/to/agent.jar. This of course only works, if you have influence on the target JVM's command line. Dynamic attachment to any running JVM does not work that way, because you are too late to specify a boot classpath option.

  2. Partition the actual agent into a "springboard agent" and another JAR containing classes referenced by your advice code. The additional JAR can be packaged inside the agent JAR as a resource or reside somewhere on the filesystem, depending on how universal your solution ought to be. The springboard agent would

  • optionally unpack the additional JAR to a temporary location (if nested inside the springboard agent),
  • dynamically add the additional JAR to the boot classpath by calling method Instrumentation::appendToBootstrapClassLoaderSearch(JarFile),
  • make sure not to reference any of the classes from the additional JAR directly, but if at all, only via reflection after the JAR is on the boot classpath already. Think Class.forName(..).getMethod(..).invoke(..).

BTW, if the classes referenced by the Byte Buddy (BB) advice use the BB API themselves, you also need to put BB itself on the boot classpath. All of this is far from trivial, so you want to try and avoid that. I went through all of this when trying to figure out how to best implement my special-purpose mocking tool Sarek.

Update 2: I mavenised and massively restructured the OP's original repository in this GitHub fork.

kriegaex
  • 63,017
  • 15
  • 111
  • 202
  • Sadly, this makes the JVM saturate for me... It transforms 863 classes (counted the number of "TRANSFORM" in the output) and then the following error : https://pastebin.com/CQ0hjt27 The error occurs in the "installTransformer" method of your program – AntoineG Aug 10 '21 at 08:01
  • I thought this error was due to my TraceAdvice class being more complex than what I showed in the original post, so I simplified it and got the same error :/ The only way I can get it to work is if I had 2 ignore statements : .or(nameStartsWith("java.")).or(nameStartsWith("sun.")) But then this defeats the whole point of the question. I will try to narrow down the ignore statements to pinpoint the problematic classes. – AntoineG Aug 10 '21 at 08:29
  • I cannot help you any further if I don't see your real code. Why would you need to trace the whole JDK? In order to make it extra slow? You didn't explain your purpose. Of course you need to exclude some problematic cases. A stack overflow error probably comes from infinite recursion. You need to exclude code called directly or indirectly by your advice, if that triggers an endless loop of logging and method calls. I need to see your project in order to be able to say more. The problem sits in front of the keyboard in this case. – kriegaex Aug 10 '21 at 08:58
  • One more thing: The last class it tries to transform is probably the one (or one of several) you need to exclude. So the full log would be more helpful then just the last error message. But do post a demo project on GitHub, please, if you want more help. Otherwise, we are done here, sorry. Your information hiding policy, even admitting that your real advice looks different from you example, is counter-productive. The advice code is exactly what makes the difference here. – kriegaex Aug 10 '21 at 09:02
  • Really sorry about that, but the comment sections feels inapropriate for large error messages... Here is the link to the code : https://github.com/Antoine-Gicquel/JavaAgent – AntoineG Aug 10 '21 at 09:26
  • 2
    The comment section **is** inappropriate for large error messages, of course. I was referring to the error message you posted on PasteBin. Thanks for the link, I am going to take a look as soon as I am no longer busy. **Edit:** Oh, a makefile! That is unusual for a Java project. No Maven or Gradle? Anyway, I am going to either paste to makefile contents to the console or quickly create a Maven POM. – kriegaex Aug 10 '21 at 09:43
  • I couldn't thank you enough for the help you provide me, ByteBuddy is not easy to apprehend... About the Makefile : yeah, I'm more used to it, never worked with Maven nor Gradle, and I'm not working on a proper Java IDE – AntoineG Aug 10 '21 at 09:56
  • I updated my answer, because the information is too much for a comment. – kriegaex Aug 10 '21 at 11:01
  • 1
    **Update 2:** I mavenised and massively restructured the OP's original repository in [this GitHub fork](https://github.com/kriegaex/JavaAgent). – kriegaex Aug 11 '21 at 15:38
2

The enableBootstrapInjection method was replaced with a generic API that allows several injection strategies. This previous strategy is still available via an InjectionStrategy that uses instrumentation. This is mainly a reaction to the current flavors of Unsafe since the JVM is shutting down on internal APIs.

As you said, you need to refine your ignore matcher to allow some classes from the boot loader. The more classes you ignore by name, the better. Advice is the right approach for such classes as you can only add code but not change the shape of any class.

As it was mentioned, it is not necessary to put Byte Buddy on the boot path. As a matter of fact, advice methods are just templates, their code will be copy-pasted into the targeted methods. As a consequence, you do not have access to any fields or other methods within these advice classes.

Rafael Winterhalter
  • 42,759
  • 13
  • 108
  • 192