Hazelcast on deployed servers throws java.lang.ClassNotFoundException and locally not

Question

I wanted to upgrade to Java 11 and tomcat 9 my spring boot application that uses Hazelcast 3.12.9 as cashing mechanism. When I deployed locally everything looks to work fine and the caching successfully works. But when the application runs on the cluster, I receive from all 3 nodes that are available the following error:

com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: com.some.service.some.server.domain.ClassA
    at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:88)
    at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:77)
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)
    at com.hazelcast.map.impl.proxy.MapProxySupport.toObject(MapProxySupport.java:1237)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:120)
    at com.hazelcast.spring.cache.HazelcastCache.lookup(HazelcastCache.java:162)
    at com.hazelcast.spring.cache.HazelcastCache.get(HazelcastCache.java:67)
    at com.some.service.some.server.domain.ClassACache.get(GlassACache.java:28)
    at com.some.service.some.server.domain.ClassAFacade.getClassA(ClassAFacade.java:203)
    at com.some.service.some.server.domain.ClassAFacade.getGlassA(ClassAFacade.java:185)
    at com.some.service.some.server.domain.ClassALogic.lambda$getClassAInParallel$1(ClassALogic.java:196)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:952)
    at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:926)
    at java.base/java.util.stream.AbstractTask.compute(AbstractTask.java:327)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.ClassNotFoundException: com.some.service.some.server.domain.ClassA
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:288)

Hazelcast customizer :

@Configuration
public class ClassAHazelcastConfig {
    private static final MaxSizePolicy HAZELCAST_DEFAULT_MAX_SIZE_POLICY = MaxSizePolicy.PER_NODE;
    private static final EvictionPolicy HAZELCAST_DEFAULT_EVICTION_POLICY = EvictionPolicy.LRU;

    @Bean
    HazelcastConfigurationCustomizer customizer(CachePropertiesHolder cacheProperties) {
        return config -> {
            config.addMapConfig(new MapConfig()
                    .setName(CLASS_A_CACHE)
                    .setMaxSizeConfig(new MaxSizeConfig(cacheProperties.getMaxsize(), HAZELCAST_DEFAULT_MAX_SIZE_POLICY))
                    .setEvictionPolicy(HAZELCAST_DEFAULT_EVICTION_POLICY)
                    .setTimeToLiveSeconds(cacheProperties.getTtl()));

            config.getSerializationConfig().addSerializerConfig(
                    new SerializerConfig()
                            .setImplementation(new OptionalStreamSerializer())
                            .setTypeClass(Optional.class)
            );
        };
    }
}

@Configuration
@EnableConfigurationProperties(CachePropertiesHolder.class)
public class CacheConfig implements CachingConfigurer, EnvironmentAware, ApplicationContextAware {

    public static final String CLASS_A_CACHE = "CACHE_A";

    private Environment environment;
    private ApplicationContext applicationContext;

    @Override
    @Bean(name="cacheManager")
    public CacheManager cacheManager() {
        boolean cachingEnabled = Boolean.parseBoolean(environment.getProperty("cache.enabled"));
        if (cachingEnabled) {
            HazelcastInstance instance = (HazelcastInstance) applicationContext.getBean("hazelcastInstance");
            return new HazelcastCacheManager(instance);
        }
        return new NoOpCacheManager();
    }

    @Override
    public CacheResolver cacheResolver() {
        return new SimpleCacheResolver(Objects.requireNonNull(cacheManager()));
    }

    @Bean
    @Override
    public KeyGenerator keyGenerator() {
        return new SimpleKeyGenerator();
    }

    @Bean
    @Override
    public CacheErrorHandler errorHandler() {
        return new SimpleCacheErrorHandler();
    }

    @Override
    public void setEnvironment(@NotNull Environment environment) {
        this.environment = environment;
    }

    @Override
    public void setApplicationContext(@NotNull ApplicationContext applicationContext) throws BeansException {
        this.applicationContext = applicationContext;
    }
}

Everything works properly fine with Java 8 and tomcat 8.

Update:

After some days of investigation, I see that the only place that these exceptions are thrown into a parallel stream that is used.

return forkJoinPool.submit(() ->
                    items.parallelStream()
                            .map(item -> {
                                try {
                                    return biFunction.apply(item);
                                } catch (Exception e) {
                                    LOG.error("Error", e);
                                    return Optional.<Item>empty();
                                }
                            })

The weird thing is that with Java 8 and tomcat 8 I did not have that issue.

Answer 1

Eventually, it is completely unrelated to Hazelcast. Mostly it is Java 11 difference from Java 8.

In the part that the exception was thrown a ForkJoinPool was used which from Java 11, it is not guaranteed when exactly will be created and it seems to not have the same classloader as the spring application. (Classes with default access result in NoClassDefFound error at runtime in spring boot project for java 11)

I made a wrong assumption because the exception was coming from Hazelcast and I also saw that there were other issues related.

Answer 2

Hazelcast offers two deployment models: embedded mode and client-server mode. Please check the relevant documentation section for more information.

In the former case, you have a single JVM and all of your classes are on your classpath.

In the latter case, you have at least 2 JVMs, one for each client and one for the member. It seems you forgot to set up the classpath of the member to reference ClassA.

How to set the classpath depends on how you launch your Hazelcast members. The method that works in most cases is to use the CLASSPATH environment variable.