4

The investigation and details for the accepted answer are in the comments to the question.

I have a small Java project that reads a schedule and puts Joda-Time Intervals into a sorted map (TreeMap in this case) for further processing. This is done via a ScheduledExecutorService:

executor.scheduleWithFixedDelay(new Runnable() {
        @Override
        public void run() {
            try {
                doWork();
            } catch (Exception e) {
                e.printStackTrace();
                throw new RuntimeException(e);
            }
        }
    }, 1, 1, TimeUnit.SECONDS);

doWork() then reads a file, creates some intervals and then populates the map using this comparator (specified in the map constructor):

@Override
public int compare(Interval o1, Interval o2) {
  return o1.getStart().compareTo(o2.getStart());
}

The code then breaks in the comparator when inserting the very first interval. Usually I would think there is something wrong with the interval itself, however I have checked several possibilities and noticed multiple odd things that I got lost in:

  • The interval is fine, o1 and o2 are valid DateTimes with the same long timestamp.
  • No exception is caught. The thread just ceases work.
  • When I launch the app from Eclipse, everything works fine. It only breaks when launching a deployed version. By deployed I mean it was packed into a .jar and copied over to a shared directory, no containers here.
  • Changing

    try {
      doWork();
    } catch (Exception e) {
      e.printStackTrace();
      throw new RuntimeException(e);
    }
    

    to

    try {
      doWork();
    } catch (Throwable e) {
      e.printStackTrace();
      throw new RuntimeException(e);
    }
    

    fixes it. (I.e. map gets populated fine, including the original first interval).

The last part makes me think it is a bug with JIT or JVM, rather than with the code. I have also explored the possibility of this being a build issue, however it does not seem to be the case:

  • Both Eclipse and the build server use Java 7 (Eclipse 7.0.51, Build server: 7.0.25, Launched the deployed version with a 7.0.51 JRE)
  • Joda time library version is the same both in Eclipse and deployed lib folder (2.1)
  • This is NOT a new feature and the very same code works in a different branch, and has been for a couple of weeks now
  • I have tried stopping Eclipse from using its own cached Ivy libraries, and instead use the libraries in the deployed directory. Same stuff - works from Eclipse, doesn't work when launching jar with Java.

After a bit of remote-debugging I have reproduced something of the like: Method "compareTo" with signature "(Lorg/joda/time/ReadableInstant;)I" is not applicable on this object with the target object being class org.joda.time.DateTime when breakpointing in the comparator code.

Any help in how to debug this further would be appreciated.

EDIT:

private void doWork() {
        SortedMap<Interval, String> map = new TreeMap<>(new Comparator<Interval>() {
            @Override
            public int compare(Interval o1, Interval o2) {
                return o1.getStart().compareTo(o2.getStart());
            }
        });

        Collection<String> col1 = new HashSet<>();
        Collection<String> col2 = new HashSet<>();
        String string = "";
        long ts = 0;

        try (FileInputStream input = new FileInputStream(fileName);
                InputStreamReader isr = new InputStreamReader(input);
                BufferedReader reader = new BufferedReader(isr)) {
            String line = reader.readLine();
            map.put(new Interval(new DateTime(), new DateTime()), "");
        }
}

While this does not look like an SSCCE due to a lot of extra code, if I remove the Collection declarations or the line read, or put anything in the map before the try block (and then do the rest as is) - everything works fine. Makes me think of race conditions, however all variables involved here are local (except fileName, which is guaranteed to have been set).

Also, while trying out stuff I found that switching to Joda-time 2.3 from 2.1 apparently fixes the problem. However I do not see anything in their changelog of bugfixes that looks even remotely relevant.

Ordous
  • 3,844
  • 15
  • 25
  • It may be helpful if we can see the definition of `doWork()`. Also, when you say "deployed", what do you mean exactly? Are you running this in a container? – Duncan Jones Apr 02 '14 at 12:05
  • Thanks for the formatting edit. By deploy I mean it was packed into a jar and copied to a remote drive, no containers or web servers (I realize we use this term quite differently here). – Ordous Apr 02 '14 at 16:40
  • Also, it seems whether the bug reproduces or not depends on some obscure factors, like presence of extra imports or volatility or variables not used in the thread (but declared in the same class) – Ordous Apr 02 '14 at 16:46
  • Just speculating: Does the remote side also have a JodaTime version? Maybe an older one before version 2.0 where the `Comparable`-handling changed a little bit (see generified interface `ReadableInstant`)? Please have a look. – Meno Hochschild Apr 03 '14 at 16:07
  • In my previous comment I wanted to say: Maybe there are TWO different Joda versions in parallel on the remote side. See also the [release notes](http://www.joda.org/joda-time/upgradeto200.html) of version 2.0. – Meno Hochschild Apr 03 '14 at 16:16
  • @Meno I have looked into the compiled program - the folder with external dependencies contained only a single Joda-Time jar that had the correct version in its manifest file. That said, I have noticed that another project that shares a dependency with this one does indeed have Joda-Time 1.6, instead of a newer version. They are both compiled by the same build server via Jenkins, so it might be an issue with the server not cleaning up properly. I would expect the library to crop up somewhere in this case though, yet can find no traces of it anywhere in this project. – Ordous Apr 03 '14 at 17:45
  • @Ordous Interesting to hear about this Joda-Time 1.6. Sounds suspicious. If you can extract the exact signature of `compareTo()` via reflection at runtime and the argument type of this method is `Object` then you have the 1.6-version of joda-`DateTime`. – Meno Hochschild Apr 03 '14 at 22:54
  • @Meno More results and more questions. First of all I have looked into the .class files in the libraries (one local Eclipse, the other deployed) to see they have 2 compareTo methods, as expected - one with Object, other with ReadableInstant. At runtime however, when I launch with Eclipse (the working version), I see 141 methods with Reflection and 2 compareTo. When remote-debugging the deployed version I see 132 methods with only 1 compareTo that takes an Object. So it seems that although the .jar is correct, it picks up a different version somewhere. – Ordous Apr 04 '14 at 10:44
  • @Meno OK, using http://stackoverflow.com/questions/1983839/determine-which-jar-file-a-class-is-from in runtime I get... /lib/jruby.jar. Which is definitely not the Joda-Time library) – Ordous Apr 04 '14 at 10:49
  • @Ordous This is confusing. Has JRuby the JodaTime-jar built in its distribution??? Anyway, the whole thing sounds much like "jar-hell" and class-loading-issue. – Meno Hochschild Apr 04 '14 at 11:45
  • Almost got to the bottom of this. The project depends on Spring 3.2.6, which in turn depends on Joda 2.1 and jRuby 1.6.5.1. jRuby depends on Joda 1.6. Some question remain: 1. Why was Joda 1.6 shipped in the Ruby jar? Is it because of backwards incompatibility in Joda 2.x? 2. Why was the app sporadically picking up DateTime from jRuby instead of Joda jar? Is this possible behaviour with 2 libraries? If so, why conflicting dependencies in Spring? Upgrading to Spring 4xx that depends on newer jRuby that uses Joda 2.x, and removing Joda as an explicit dependency seems to resolve problems as well. – Ordous Apr 04 '14 at 11:50
  • @Meno Indeed, this seems to be the case. Pre 1.7 jRuby packaged a modified version of Joda in their jar. They seem to have stopped doing that in 1.7, which is the default used by Spring 4.x. I'm not sure how or why the app suddenly picked it up in this particular place, but it seems others had similar problems (albeit they were using jRuby directly and hence had a somewhat easier time debugging). Could you please post an answer so I can accept it? (You were absolutely right about a deprecated version of Joda hiding somewhere in the libraries) – Ordous Apr 04 '14 at 12:36

1 Answers1

1

Here a summarizing answer on base of my comments:

Does the remote side have a second JodaTime version? Maybe an older one before version 2.0 where the Comparable-handling changed a little bit (see generified interface ReadableInstant)? Pre 2.0 the method in question had the signature compareTo(Object) while in 2.0 and later the new signature compareTo(ReadableInstant), see also the release notes. This situation of two joda-jars and a connected classloading-issue would explain following exception message:

Method "compareTo" with signature "(Lorg/joda/time/ReadableInstant;)I" is not applicable on this object with the target object being class org.joda.time.DateTime.

(Your question was tricky and has deserved even more upvotes. Glad to hear that you found the reason in JRuby-library containing an older JodaTime-version.)

Meno Hochschild
  • 42,708
  • 7
  • 104
  • 126