12

So first off I know that you can relocate all references your compiled jar with various shadow plugins on different build systems. I know how that works and am already using it. However I ran into a problem where I can't do that at compile time.

I'll simplify my situation so it's easier to understand (but I'll explaining the full picture at the bottom, in case you are curious).
I'm writing a plugin for two different (but similar) systems (one jar forall). Those platforms are in charge of starting the underlying software and loading/starting all plugins (so I don't have control over the application, including start parameters).
Platform A offers me a library (let's call it com.example.lib). And so does platform B. But it decided to relocate it to org.b.shadow.com.example.lib.
Now in the core code (the code used on both platforms) of my plugin I use the library. Now while I can detect on which platform I am on, I currently do not know how I can rewrite all references in my code to the library at runtime so it works on platform B.

From what I've found it seems like I need to use a custom ClassLoader to achieve that. The issue here being that I don't know I could make the runtime use my custom ClassLoader. Or where to start really.
One important thing is that those relocations may only affect references in classes from my packages (me.brainstone.project for example).
Another dependency I use (and have shaded in) uses ASM and ASM Commons, so if it is possible doing it with those, that would be amazing!

So in summary. I would like to optionally relocate references (to other classes) in only my classes at runtime.

Edit:

While throught my entire (orginal) post I only ever talked about one library, I would like to point out that I will be doing this for serveral libaries. And there for doing things that require me to put a significant effort (writing wrappers for every library (class or section) would be consider as a significant effort) into allowing to me use a libary is not what I am looking for. Instead I want a solution that requries minimal additons for adding new libraries into the mix.


Now here is a a bit more detailed explanation of my setup.
Fist I'd like to preface that I am aware that I can just create two different jars for the different platforms. And I am already doing that. But since surprisingly many people can't seem to figure that out and I'm getting tired of explaining it over and over again (those are the people that wouldn't read docs to save their lives) I'd like to just offer a single jar for both, even if it means I need to spend a significant time on getting it to work (I much prefer this over constantly explaining it).
Now my actual setup looks like this: On platform A the library is provided but on platform B it isn't. I know that other plugins often use the library by shading it in (many not relocating causing all kinds of issues). So to prevent any conflicts I download the library, relocate the classes inside that jar with jar-relocator and then inject it into the classpath using reflections. Downside in this case I currently cannot use the library if it's relocated. That's why I'd like to change the references in my code at runtime. And it also explains why I don't want to change the references of other classes, because I don't want to accidentally break those other plugins. I also think that if I can somehow use my own ClassLoader that I don't need to inject jars into the main ClassLoader because then I can just tell that ClassLoader to use the additional jars without having to resort to reflections.
But as I said, from what I understand the problem is the same as in the simplified version.

BrainStone
  • 3,028
  • 6
  • 32
  • 59
  • 3
    Just don't. Create interface and 2 separate implementations and choose matching one at runtime. Sounds like another over-complicated minecraft plugin for no reason. – GotoFinal Aug 29 '19 at 07:17
  • @GotoFinal Lots and lots of essentially duplicated code and wrapping everything I need to use from external libraries is just what I want. Let alone building that... – BrainStone Aug 29 '19 at 20:03
  • if you will do anything like that you will lose ability to debug such code, even worse thing... and yes, it is better to duplicate some code than create something like that. And with good abstraction you will not need to duplicate anything or just very small things. And building will be simple & straightforward. But I can later write an answer here anyways... when I will be back from work – GotoFinal Aug 30 '19 at 05:59
  • Due to the nature of the project I'm already limited to very basic debugging. I've learned to work with that, so I really don't mind not being able to do something that I couldn't do before. Another thing is that I have more than one library to use. Guessing around 10 to 15. And in those I'm using more than a single class/interface. So having to do that for every single interface I'll be using sounds like an awful chore. I honestly much rather have a solution that is a lot of work once, but then it scales very well. – BrainStone Aug 30 '19 at 07:42
  • And lastly I am using gradle. So while showing how this can be built with maven (if you were to do that) is fine. This place is for everyone. But it won't help me. – BrainStone Aug 30 '19 at 07:43
  • Another thing you should try then are template libraries, they will generate different version of code at compile time, look at projects like https://github.com/vigna/fastutil they generate like 90% of code. And then you can get all the benefits. – GotoFinal Aug 30 '19 at 09:49
  • What's the problem with duplicated code? You can write a script to create that (copypaste) interface and put it into a separate Java file (with some "warning: machine generated code" on top of it). – Lev Leontev Sep 01 '19 at 14:57
  • idk why you insist on creating this in such weird way, but I posted my dirty answer as promised. I still hope you will later decide to just generate that code, but I like weird questions like that so... here you go. – GotoFinal Sep 01 '19 at 19:32
  • @GotoFinal I am a fan of general solutions to my problems. I like having solutions I can reuse when necessary (or maybe even turn into a library for others to use). Also I prefer keeping small patches of somewhat dirty code around instead of littering the project with (duplicated) wrappers. And also I feel like these over the top solutions help building great things. Take package managers for example. They were absoltely over the top when they were created. And nowadays nobody wants to imagine living without them. While I am totally aware that this is no where near that, – BrainStone Sep 02 '19 at 21:38
  • I still feel like this follows the same principle. In the sense that creating an overly complicated system can go a long way. – BrainStone Sep 02 '19 at 21:38
  • It seams like an attempt to re-invent an OSGI platform like Apache Felix. – Victor Gubin Sep 05 '19 at 11:20
  • @VictorGubin he is probably creating a plugin for popular Minecraft engines and want to have single code that works on multiple versions and engines. And it often requires you to use some tricks because not everything is available in API so often to implement a feature you need to deal with original obfuscated code. Nasty but works... – GotoFinal Sep 05 '19 at 12:42
  • Essentially. One jar for all. And yes the hacks are real. Though it really helps with getting the plugin out to people. – BrainStone Sep 05 '19 at 13:03

1 Answers1

12

First you should think about different solution, as every other solution is better than this one, so possible ones:

  1. Just create separate modules.
  2. Use some code generation at compile time to generate that modules so you don't need to duplicate your code, look at https://github.com/vigna/fastutil for example.

But if you really want to do it in very dirty way:
Use java agents. This require use of jdk jvm or/and additional startup arguments. You should probably use byte-buddy-agent library if you want to do this at runtime without startup arguments, and there is also dirty trick on java 8 to run agents in runtime even without proper files from jdk - by just injecting them manually, probably also possible on java 9+ but so far I didn't have time and need to find a way to do this. You can see my instructions here https://github.com/raphw/byte-buddy/issues/374#issuecomment-343786107
But if possible the best way is to just use command line argument to attach agent .jar as separate thing.
First thing to do is to write a class file transformer that will do all the logic you need:

public class DynamicLibraryReferenceTransformer implements ClassFileTransformer {
    private final String packageToProcess;
    private final String originalPackage;
    private final String resolvedPackage;

    DynamicLibraryReferenceTransformer(String packageToProcess, String originalPackage, String resolvedPackage) {
        this.packageToProcess = packageToProcess;
        this.originalPackage = originalPackage;
        this.resolvedPackage = resolvedPackage;
    }

    @Override
    public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
                            byte[] classfileBuffer) {
        if (! className.startsWith(this.packageToProcess)) {
            return null; // return null if you don't want to perform any changes
        }
        Remapper remapper = new Remapper() {
            @Override
            public String map(String typeName) {
                return typeName.replace(originalPackage, resolvedPackage);
            }
        };
        ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES);
        ClassRemapper classRemapper = new ClassRemapper(cw, remapper);
        ClassReader classReader = new ClassReader(classfileBuffer);
        classReader.accept(classRemapper, 0);
        return cw.toByteArray();
    }
}

And then you just need to apply it as java agent, either in runtime:

static { 
    Instrumentation instrumentation= ByteBuddyAgent.install();
    // note that this uses internal names, with / instead of dots, as I'm using simple .replace it's good idea to keep that last / to prevent conflicts between libraries using similar packages. (like com/assist vs com/assistance)
    instrumentation.addTransformer(new DynamicLibraryReferenceTransformer("my/pckg/", "original/pckg/", "relocated/lib/"), true);
    // you can retransform existing classes if needed but I don't suggest doing it. Only needed if some classes you might need to transform are already loaded
    // (classes are loaded on first use, with some smaller exceptions, like return types of methods of loaded class are also loaded if I remember correctly, where fields are not)
    // you can also just retransform only known classes
    instrumentation.retransformClasses(install.getAllLoadedClasses());
}

This code should be run as quickly as possible, like in static block of code inside your main class.

Better option is to include agent to JVM at startup using command line:

First you will need to create new project as this will be separate .jar, and create manifest with Premain-Class: mypckg.AgentMainClass that you will include in meta-inf of agent .jar.
Use same transformer as above, and then you just need to write very simple agent like this:

public class AgentMainClass {
    public static void premain(String agentArgs, Instrumentation instrumentation) {
        instrumentation.addTransformer(new DynamicLibraryReferenceTransformer("my/pckg/", "original/pckg/", "relocated/lib/"), true);
    }
}

And now just include it in your java command to run the app (or server probably) -javaagent:MyAgent.jar.
Note that you can include code of agent and manifest inside your main (plugin?) .jar, just be sure to not mix up dependencies, classes for agent will be loaded using different class loader, so don't make calls between app and agent, that will be 2 separate things inside single .jar.

This uses org.ow2.asm.asm-all library, and net.bytebuddy.byte-buddy-agent (only for runtime version) library.

GotoFinal
  • 3,585
  • 2
  • 18
  • 33
  • 1
    Are we talking about Java 9 modules (in your suggestion on how to solve it in a better way)? I can't use that. I am using Java 8. – BrainStone Sep 02 '19 at 21:42
  • @BrainStone It's for java 8 or 9+ too, it will actually work better in java 8, less issues and things to worry about. – GotoFinal Sep 02 '19 at 21:46
  • I have to two additonal questions: 1) `instrumentation.retransformClasses(install.getAllLoadedClasses());` should just be used to retransform already loaded classes, right? 2) Do I keep the `Instrumentation` instance or do I recreate it for every transformer? (I have different times when the l know which libraries to use and relocate) – BrainStone Sep 02 '19 at 21:48
  • byte-buddy-agent will keep instance of instrumentation safe, so you don't need to worry about that. and yes retransformClasses should only be used to apply changes to loaded classes in best scenario you should not be using it at all, but it might be useful if you will not be able to ensure class loading order for some reason. – GotoFinal Sep 02 '19 at 21:50
  • Ah there is one important limitation I forgot to include here: REtransformed classes (so already loaded ones) can't change field or method signatures, so you really need to ensure as much as you can that these classes will not be loaded yet. (as before class is loaded everything can be changed) – GotoFinal Sep 02 '19 at 21:54
  • Alright. Will give this a try soon. One last question. Can I check from where the request to load that class came? And lastly thanks a lot for that answer! It's much apprechiated. – BrainStone Sep 02 '19 at 21:54
  • @BrainStone Simplest way would be to just put breakpoint inside class loader, with intellij you can create breakpoint that will just print something to console and will not stop the application/thread. So you can print a stacktrace. But no way to find this for already loaded classes, you can only get the class loader. – GotoFinal Sep 02 '19 at 21:57
  • I'm running into the JRE/JDK issue. I've looked at your link but am really not sure what I can do when I know little to nothing about the traget system. Any idea on that? – BrainStone Sep 03 '19 at 22:01
  • You just need to combine code from all systems, similar thing was done by EA: https://github.com/electronicarts/ea-agent-loader but I don't remember if it includes native library too, but they have classes for each platform. So you only need attach.dll/so fo each target platform. – GotoFinal Sep 04 '19 at 07:43
  • You can try asking separate question about injecting agent on specific jvm version, as it's kinda separate issue from this one. As I will not be able to provide more detailed instructions in comments here... – GotoFinal Sep 04 '19 at 08:11
  • While I did not (yet) manage to get this working for my use case I think it is a a great answer and deserves my bounty. If I manage to get it working using this technique I will accept it later. Thanks for putting your time into this. Much apprechiated! – BrainStone Sep 06 '19 at 19:59
  • if you still have some issues just describe them, I think there is some option to move conversation to chat as it might be easier them to share some snippets/errors. – GotoFinal Sep 06 '19 at 20:03
  • Sure. I'm in the Java channel – BrainStone Sep 06 '19 at 20:13
  • Just a note that ClassWriter.COMPUTE_FRAMES shouldn't be needed in this case, as relocating classes shouldn't change them in any way, so as long as reader preserves them (it does), it should work. – barteks2x Mar 31 '20 at 05:28