NPEs trying to get MethodDefinitions out of ClassDefinitions; how to diagnose?

Question

Environment:

the caché server is a x86_64 SuSE Linux Enterprise server; installation is, as far as I can tell, some 2014.x version;
I have SSH access to the machine, but nothing web (VPN, ya know);
the client side is a Debian Jessie machine;
the client code is written in Java, using cachedb.jar version 2015.2.

What I'm doing is writing a SonarQube plugin for ObjectScript, and it works pretty well... Except when it does not.

What I do is separate all ClientMethods with langauge JavaScript into their own files so that the SonarQube JS plugin can analyze them, and for this I use this method to collect the relevant methods:

@SuppressWarnings("unchecked")
private List<MethodDefinition> readJsMethods(final boolean ignoreFailures)
    throws CacheException
{
    final Map<String, MethodDefinition> map;
    try {
        map = classDef.getMethods(); // classDef is a ClassDefiniton
    } catch (CacheException e) {
        if (ignoreFailures) {
            LOGGER.error("Failed to obtain methods for class {}, ignored",
                className);
            return Collections.emptyList();
        }
        throw e;
    }

    final List<MethodDefinition> ret = new ArrayList<>();

    MethodDefinition definition;
    boolean failure = false;

    for (final Map.Entry<String, MethodDefinition> entry: map.entrySet()) {
        definition = entry.getValue();
        if (definition == null) { // <-- PROBLEM HERE
            LOGGER.error("Null method definition! Class {}, key {}",
                className, entry.getKey());
            if (ignoreFailures) {
                LOGGER.warn("Failures are ignored; returning no methods");
                return Collections.emptyList();
            }
            failure = true;
            continue;
        }
        if ("javascript".equalsIgnoreCase(definition.getLanguage()))
            ret.add(definition);
    }

    if (failure)
        throw new SqcaException("one or more method definitions are null"
            + " for class " + className);

    Collections.sort(ret, Comparator.comparing(BY_SEQUENCE));
    return ret;
}

At the line marked // <-- PROBLEM HERE it can happen that the definitions is indeed null. What is more, the exact ClassDefinition and key of the Map are never the same from one run to the other.

I have tried both to restart the Caché server and the whole VM (I do have root access) but the result doesn't change.

I suspect this is a problem on the server side but I have no idea how to diagnose it and, more importantly, how to fix it. And this code has worked flawlessly for several weeks before failing all of a sudden... And when it starts failing it will always fail, always at some random method in some random class...

How do I diagnose this, knowing that I don't have any access to the Caché web management portal at all?

Sample stack trace:

Jul 27, 2015 7:03:57 AM com.roche.sqca.read.Main main
INFO: Loading properties from sqca.properties
Jul 27, 2015 7:03:58 AM com.roche.sqca.read.CacheDbReader getAllClasses
INFO: Reading list of classes from database

Jul 27, 2015 7:04:27 AM com.roche.sqca.read.CacheDbReader getAllClasses
INFO: Result: 12296 total classes, 6920 included, 5376 excluded
Jul 27, 2015 7:04:27 AM com.roche.sqca.read.CosSourceWriter processClasses
INFO: Writing COS class files (directory: /home/sonar/.jenkins/jobs/Infinity/workspace)

Jul 27, 2015 7:04:57 AM com.roche.sqca.read.JsSourceWriter processClasses
INFO: Writing JavaScript client methods, if any (directory: /home/sonar/.jenkins/jobs/Infinity/workspace)

Jul 27, 2015 7:06:23 AM com.roche.sqca.read.CosClass readJsMethods
SEVERE: Null method definition! Class LISPages.WorkArea.Monitoring, key 19
Exception in thread "main" com.roche.sqca.read.SqcaException: one or more method definitions are null for class LISPages.WorkArea.Monitoring
    at com.roche.sqca.read.CosClass.readJsMethods(CosClass.java:199)
    at com.roche.sqca.read.CosClass.getJsMethods(CosClass.java:131)
    at com.roche.sqca.read.JsSourceWriter.processClasses(JsSourceWriter.java:96)
    at com.roche.sqca.read.Main.main(Main.java:142)
Build step 'Execute shell' marked build as failure
Finished: FAILURE

And here is a trace when I actually ignore failures (ignoreFailures in the code above is set to true):

Jul 29, 2015 8:38:10 AM com.roche.sqca.read.Main main
INFO: Loading properties from sqca.properties
Jul 29, 2015 8:38:11 AM com.roche.sqca.read.CacheDbReader getAllClasses
INFO: Reading list of classes from database
Jul 29, 2015 8:38:33 AM com.roche.sqca.read.CacheDbReader getAllClasses
INFO: Result: 12296 total classes, 6920 included, 5376 excluded
Jul 29, 2015 8:38:33 AM com.roche.sqca.read.CosSourceWriter processClasses
INFO: Writing COS class files (directory: /home/sonar/.jenkins/jobs/Infinity/workspace)
Jul 29, 2015 8:39:03 AM com.roche.sqca.read.JsSourceWriter processClasses
INFO: Writing JavaScript client methods, if any (directory: /home/sonar/.jenkins/jobs/Infinity/workspace)
Jul 29, 2015 8:40:17 AM com.roche.sqca.read.CosClass readJsMethods
SEVERE: Null method definition! Class LISPages.WorkArea.Monitoring, key 109
Jul 29, 2015 8:40:17 AM com.roche.sqca.read.CosClass readJsMethods
WARNING: Failures are ignored; returning no methods
# 4 other similar messages for other classes.
# And them, BOOM:
Exception in thread "main" java.sql.SQLException: Invalid Message Count (452); expected: 95590 got: 1163022164. Connection closed
    at com.intersys.jdbc.InStream.invalidMessageReceived(InStream.java:308)
    at com.intersys.jdbc.InStream.checkHeader(InStream.java:108)
    at com.intersys.jdbc.InStream.readHeader(InStream.java:147)
    at com.intersys.jdbc.CacheConnection.sendCustomMessage(CacheConnection.java:1404)
    at com.intersys.cache.jbind.CustomMsgDBAdapter.sendCustomMessage(CustomMsgDBAdapter.java:922)
    at com.intersys.cache.jbind.CustomMsgDBAdapter.getObjectArrayElements(CustomMsgDBAdapter.java:655)
    at com.intersys.cache.jbind.JBindDatabase.getObjectArrayBuffer(JBindDatabase.java:632)
    at com.intersys.classes.ObjectArrayBuffer.getArrayBuffer(ArrayOfObjects.java:296)
    at com.intersys.classes.ArrayBuffer.load(ArrayBuffer.java:407)
    at com.intersys.classes.ArrayBuffer.getNext(ArrayBuffer.java:199)
    at com.intersys.classes.ArrayBuffer._getNext(ArrayBuffer.java:568)
    at com.intersys.classes.AbstractCacheArray._getNext(AbstractCacheArray.java:465)
    at com.intersys.classes.AbstractCacheArray._getFirst(AbstractCacheArray.java:455)
    at com.intersys.classes.AbstractCacheArray$EntryIterator.<init>(AbstractCacheArray.java:153)
    at com.intersys.classes.AbstractCacheArray$EntrySet.iterator(AbstractCacheArray.java:239)
    at com.roche.sqca.read.CosClass.readJsMethods(CosClass.java:182)
    at com.roche.sqca.read.CosClass.getJsMethods(CosClass.java:131)
    at com.roche.sqca.read.JsSourceWriter.processClasses(JsSourceWriter.java:96)
    at com.roche.sqca.read.Main.main(Main.java:142)
com.intersys.objects.CacheServerException: Failed to get elements of array with oref 57323 caused by: java.sql.SQLException: Invalid Message Count (452); expected: 95590 got: 1163022164. Connection closedUnderlying exception: java.sql.SQLException: Invalid Message Count (452); expected: 95590 got: 1163022164. Connection closed

    at com.intersys.cache.jbind.CustomMsgDBAdapter.getObjectArrayElements(CustomMsgDBAdapter.java:660)
    at com.intersys.cache.jbind.JBindDatabase.getObjectArrayBuffer(JBindDatabase.java:632)
    at com.intersys.classes.ObjectArrayBuffer.getArrayBuffer(ArrayOfObjects.java:296)
    at com.intersys.classes.ArrayBuffer.load(ArrayBuffer.java:407)
    at com.intersys.classes.ArrayBuffer.getNext(ArrayBuffer.java:199)
    at com.intersys.classes.ArrayBuffer._getNext(ArrayBuffer.java:568)
    at com.intersys.classes.AbstractCacheArray._getNext(AbstractCacheArray.java:465)
    at com.intersys.classes.AbstractCacheArray._getFirst(AbstractCacheArray.java:455)
    at com.intersys.classes.AbstractCacheArray$EntryIterator.<init>(AbstractCacheArray.java:153)
    at com.intersys.classes.AbstractCacheArray$EntrySet.iterator(AbstractCacheArray.java:239)
    at com.roche.sqca.read.CosClass.readJsMethods(CosClass.java:182)
    at com.roche.sqca.read.CosClass.getJsMethods(CosClass.java:131)
    at com.roche.sqca.read.JsSourceWriter.processClasses(JsSourceWriter.java:96)
    at com.roche.sqca.read.Main.main(Main.java:142)

Is there an exception thrown? Can you add the complete exception including stacktrace? — Konrad, Jul 29 '15 at 06:34
@Konrad added an example; I'll do another run with failures ignored, this can give some strange exceptions as well... — fge, Jul 29 '15 at 06:36
From the code this line `map = classDef.getMethods();` produces a `map` with a NULL as value for key 19. You should check the `getMethods()` method. — Konrad, Jul 29 '15 at 06:44
@Konrad that's cachedb.jar code... See edit, I have added another output where I _try_ and ignore failures. To no avail :( — fge, Jul 29 '15 at 06:53
It is low-level error in jdbc connection, you can add logfile, and see dump with error, so format of jdbc string: `jdbc:Cache://://:::`[documentation](http://docs.intersystems.com/cache20152/csp/docbook/DocBook.UI.Page.cls?KEY=BGJD_connecting#BGJD_C125815) — DAiMor, Aug 12 '15 at 09:26

NPEs trying to get MethodDefinitions out of ClassDefinitions; how to diagnose?

0 Answers0