6

Summary: Loading a jar from a running Java program causes a NoClassDefFoundError caused by a ClassNotFoundException caused by inter-class dependencies (e.g. import statements). How can I get around it?

The problem in more detail:

I am attempting to programmatically load a jar file -- let's call it "Server" -- into the Java Virtual Machine through my own Java program -- let's call it "ServerAPI" -- and use extension and some other tricks to modify the behavior of and interact with Server. ServerAPI depends on Server, but if Server is not present, ServerAPI still has to be able to run and download Server from a website.

To avoid errors caused by ServerAPI loading without satisfying its dependencies from Server, I have made a launcher -- let's call it "Launcher" -- that is intended to download Server and set up ServerAPI as necessary, then load Server and ServerAPI, then run ServerAPI.

However, when I attempt to load jars from Launcher, I get errors caused because the ClassLoaders are unable to resolve the other classes in the file that the class it's loading depends on. In short, if I try to load Class A, it will throw an error if A imports B because I haven't loaded B yet. However, if B also imports A, I'm stuck because I can't figure out how to load two classes at once or how to load a class without the JVM running its validation.

Why all the restrictions have led me to this problem:

I am attempting to modify and add to the behavior of Server, but for complicated legal reasons, I cannot modify the program directly, so I have created ServerAPI that depends on and can tweak the behavior of Server from the outside.

However, for more complicated legal reasons, Server and ServerAPI cannot simply be downloaded together. Launcher (see above) has to be downloaded with ServerAPI, then Launcher needs to download Server. Finally, ServerAPI can be run using Server as a dependency. That's why this problem is so complex.

This problem will also apply to a later part of the project, which will involve a plugin-based API interface that needs to be able to load and unload plugins from jar files while running.

Research I have already done on this problem:

I have read through and failed to be helped by:

  • this question, which only addresses the issue of a single method and does not address inter-class dependency errors;
  • this question, which will not work because I cannot shut down and restart the program every time a jar is loaded or unloaded (mainly for the plugin part I briefly mentioned);
  • this question, which only works for situations where the dependencies are present when the program starts;
  • this question, which has the same problem as #2;
  • this question, which has the same problem as #3;
  • this article, from which I learned about the hidden loadClass(String, boolean) method, but trying with true and false values did not help;
  • this question, which has the same problem as #1;

and more. Nothing has worked.

//EDIT: Attempts I have made so far:

I have tried using URLClassLoaders to load the jar using the JarEntries from the JarFile similar to this question. I tried this both by using and calling a URLClassLoader's loadClass(String) method and by making a class that extends URLClassLoader so that I could utilize loadClass(String, boolean resolve) to try to force the ClassLoader to resolve all the classes it loads. Both ways, I got this same error:

I couldn't find the class in the JarEntry!
entry name="org/apache/logging/log4j/core/appender/db/jpa/converter/ContextMapAttributeConverter.class"
class name="org.apache.logging.log4j.core.appender.db.jpa.converter.ContextMapAttributeConverter"
java.lang.NoClassDefFoundError: javax/persistence/AttributeConverter
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at Corundum.launcher.CorundumClassLoader.load(CorundumClassLoader.java:52)
    at Corundum.launcher.CorundumLauncher.main(CorundumLauncher.java:47)
Caused by: java.lang.ClassNotFoundException: javax.persistence.AttributeConverter
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 12 more

//END EDIT

//EDIT 2:

Here is a sample of the code that I used to load a class while trying to resolve it. This was inside a class that I made that extends URLClassLoader. On the line beginning with Class<?> clazz = loadClass(, I have tried using true and false as the boolean argument; both attempts resulted in the same error above.

public boolean load(ClassLoadAction class_action, FinishLoadAction end_action) {
    // establish the jar associated with this ClassLoader as a JarFile
    JarFile jar;
    try {
        jar = new JarFile(jar_path);
    } catch (IOException exception) {
        System.out.println("There was a problem loading the " + jar_path + "!");
        exception.printStackTrace();
        return false;
    }

    // load each class in the JarFile through its JarEntries
    Enumeration<JarEntry> entries = jar.entries();

    if (entries.hasMoreElements())
        for (JarEntry entry = entries.nextElement(); entries.hasMoreElements(); entry = entries.nextElement())
            if (!entry.isDirectory() && entry.getName().endsWith(".class"))
                try {
                    /* this "true" in the line below is the whole reason this class is necessary; it makes the URLClassLoader this class extends "resolve" the class,
                     * meaning it also loads all the classes this class refers to */
                    Class<?> clazz = loadClass(entry.getName().substring(0, entry.getName().length() - 6).replaceAll("/", "."), true);
                    class_action.onClassLoad(this, jar, clazz, end_action);
                } catch (ClassNotFoundException | NoClassDefFoundError exception) {
                    try {
                        close();
                    } catch (IOException exception2) {
                        System.out.println("There was a problem closing the URLClassLoader after the following " + exception2.getClass().getSimpleName() + "!");
                        exception.printStackTrace();
                    }
                    try {
                        jar.close();
                    } catch (IOException exception2) {
                        System.out.println("There was a problem closing the JarFile after the following ClassNotFoundException!");
                        exception.printStackTrace();
                    }
                    System.out.println("I couldn't find the class in the JarEntry!\nentry name=\"" + entry.getName() + "\"\nclass name=\""
                            + entry.getName().substring(0, entry.getName().length() - 6).replaceAll("/", ".") + "\"");
                    exception.printStackTrace();
                    return false;
                }

    // once all the classes are loaded, close the ClassLoader and run the plugin's main class(es) load() method(s)
    try {
        jar.close();
    } catch (IOException exception) {
        System.out.println("I couldn't close the URLClassLoader used to load this jar file!\njar file=\"" + jar.getName() + "\"");
        exception.printStackTrace();
        return false;
    }

    end_action.onFinishLoad(this, null, class_action);
    System.out.println("loaded " + jar_path);
    // TODO TEST
    try {
        close();
    } catch (IOException exception) {
        System.out.println("I couldn't close the URLClassLoader used to load this jar file!\njar file=\"" + jar_path + "\"");
        exception.printStackTrace();
        return false;
    }
    return true;
}

//END EDIT 2

I realize that there must be a simple solution to this, but for the life of me I cannot seem to find it. Any help would make me eternally grateful. Thank you.

Community
  • 1
  • 1
Variadicism
  • 624
  • 1
  • 7
  • 17
  • Have you confirmed from the stack trace that it is in fact the import itself that is causing the issue, and not (say) an inadvertent access of the other class from a constructor or static variable? (And if it's really the import, what happens if you remove the import and fully-qualify all of the object references to the class instead?) Lastly, can you post how you are loading the jar? – Scott Dudley Oct 02 '14 at 03:15
  • In cases like this, it can be helpful to mock up a very simple example with two do-nothing classes in separate jars and figure out if they have the same problem when you load them. If not, continue increasing the complexity of the model until it mirrors the patterns used in your real-life app. For example, I've noticed (in an OSGi environment) that the classloader likes to resolve all of the classes referenced in one way or another from a class that I load directly, but it doesn't try to resolve classes referenced in the one-step-removed classes until they are invoked. – Scott Dudley Oct 02 '14 at 03:23
  • (In that case, adding a shim class between the source and target classes was enough to make the classloader happy, so long as the shim class never gets invoked until the target class is loaded.) – Scott Dudley Oct 02 '14 at 03:25
  • @ScottDudley I have confirmed from the stack trace that the error occurs at an import. I cannot try fully qualifying instead of using an import because I cannot modify Server, which is also why I can't try your "shim class" idea. I have tried loading the jar with URLClassLoaders both by using them by themselves and by creating a child class to utilize `loadClass(String, boolean)`; neither have worked for the same reason. I will add more info on my attempts at loading to the question, including the stack trace. – Variadicism Oct 02 '14 at 17:32
  • @REALDrummer, you can't modify Server, but you do have control over ServerAPI, right? Could you not load ServerAPI first (which uses a shimmed class to talk to Server), and then load Server next? (And did you try making both JARs accessible at the same time by passing two URLs to your URLClassLoader constructor?) – Scott Dudley Oct 02 '14 at 17:43
  • @ScottDudley A lot of the classes in ServerAPI have to extend classes from Server. Will loading a class that extends an unresolved class throw an error if I fully qualify the extended class? If not, then ServerAPI cannot be loaded before Server because it will throw `NoClassDefError`s on load. Oh, there's also another problem: I have to run the code through a re-obfuscator because Server is obfuscated; that re-obfuscator may discard fully qualified statements, but it may not. I will try. – Variadicism Oct 02 '14 at 18:01
  • In hindsight, I think the fully-qualified classes idea (vs using imports) is unlikely to help you at all. Also, if you're trying to extend a class, you definitely need the superclass accessible before you can load the subclass. I think it's time to get that code posted! – Scott Dudley Oct 02 '14 at 18:05
  • @ScottDudley I added the loading method I used to try to load jars. – Variadicism Oct 02 '14 at 18:50
  • @REALDrummer What IDE are you using? Did you try to clean your project? – Leistungsabfall Oct 02 '14 at 18:53
  • @REALDrummer, I am unclear as to why you're parsing the JAR file directly and trying to load each class individually. Do you have some functional need to do that? How about something like this: https://bitbucket.org/sdudley/jartest/src – Scott Dudley Oct 02 '14 at 19:28
  • @Leistungsabfall I am using Eclipse Luna with Java SE 1.8. with Oracle's Java 8 JDK on Linux Mint 17 Qiana. Cleaning the projet did not seem to make any changes and the test game the same results. – Variadicism Oct 02 '14 at 19:42
  • @ScottDudley The only difference I see between your general approach and mine is that 1) you are loading Server and ServerAPI with the same ClassLoader and 2) you are hardcoding the class names instead of getting the information from the `JarFile`. I don't see how #1 will change anything and #2 is not possible because I may not know the exact contents of the jar being loaded beforehand; in the case of Server, an update will break it immediately, and since I want to use this same setup for a plugin-based API, it will not work for that. – Variadicism Oct 02 '14 at 19:45
  • @REALDrummer 1) Is there any reason you don't want to do that? It seems to be a requirement of what you are trying to accomplish! 2) Loader only references the classes that it uses directly. Notice that Server3 is not referenced anywhere in Loader, but it's still resolved correctly from Server1. – Scott Dudley Oct 02 '14 at 19:49
  • @ScottDudley 1) I have no reason not to do that; I just don't see how it will help. This error pops up just when loading Server, which does not depend on ServerAPI in any way. ServerAPI depends on Server; Server does not depend on ServerAPI. 2) I understand that Server3 SHOULD be resolved through Server1 if Server1 references Server3, but the problem is that that's not what's happening. Instead, the loader sees Server3 and instead of trying to load and resolve Server3, it just throws an error. That's the problem. – Variadicism Oct 02 '14 at 19:56
  • But with the exception you added above, it now doesn't even look like the missing class is in either of your JARs to begin with (`javax.persistence.AttributeConverter`), right? So your problem is just that you can't load a JAR to begin with? (Did you pass in a parent classloader when constructing the URLClassLoader that knows how to resolve the missing classes?) – Scott Dudley Oct 02 '14 at 20:09
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/62367/discussion-between-scott-dudley-and-realdrummer). – Scott Dudley Oct 02 '14 at 20:10
  • Well, I'm not getting the same error, but it doesn't appear until I try to call a method inside one of the classes loaded form another jar. I'm going to post it as a separate question since it's a separate issue. – Variadicism Oct 04 '14 at 00:59

1 Answers1

2

Embarassingly, I found that the answer was that the error message was telling the truth. javax.persistence.AttributeConverter, the class that the loader was claiming was not present, was not in the jar.

I fixed the issue by loading only the main class and the ClassLoader all references classes, essentially loading all the classes in the jar that are used in the program, which is all I need.

Now, I could have sworn that I checked for this before and found that class; I figure I must have actually checked the Apache open source repository for the class rather than the actual Server when I checked that. I can't remember.

In any case, AttributeConverter is missing. I don't know how or why they managed to compile a jar with missing dependencies, but I guess their main processes never use that part of the code, so it never threw errors.

I'm sorry to have wasted everyone's time...including my own. I have been stuck on this problem for a while now.

Moral of this story:

If you're trying to load an executable jar, don't bother loading all the classes in a jar unless you actually have to. Just load the main class; that will load everything the program needs to run.

//EDIT:

I have now started having the same error, but it does not appear until I attempt to call a method from a loaded class. The question is apparently still open. Please downvote and disregard this answer.

Variadicism
  • 624
  • 1
  • 7
  • 17