1

Sometimes raise this error and go to down the task manager. I used org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode

WARN  org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer  - Falling back to default Kryo serializer because Chill serializer couldn't be found.
    java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.getKryoInstance(KryoSerializer.java:436)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.checkKryoInitialized(KryoSerializer.java:454)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:337)
        at org.apache.flink.api.common.typeutils.CompositeSerializer.deserialize(CompositeSerializer.java:151)
        at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextElementLastAccessTimestamp(RocksDbTtlCompactFiltersManager.java:202)
        at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextUnexpiredOffset(RocksDbTtlCompactFiltersManager.java:189)
    Caused by: java.lang.ClassNotFoundException: scala/Function0
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.twitter.chill.KryoBase.$anonfun$functions$1(KryoBase.scala:41)
        at com.twitter.chill.KryoBase.$anonfun$functions$1$adapted(KryoBase.scala:41)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
        at scala.collection.immutable.Range.foreach(Range.scala:155)
        at scala.collection.TraversableLike.map(TraversableLike.scala:233)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at com.twitter.chill.KryoBase.<init>(KryoBase.scala:41)
        at org.apache.flink.runtime.types.EmptyFlinkScalaKryoInstantiator.newKryo(FlinkScalaKryoInstantiator.scala:45)
        at org.apache.flink.runtime.types.FlinkScalaKryoInstantiator.newKryo(FlinkScalaKryoInstantiator.scala:82)
        ... 10 more
    Exception in thread "Thread-9" java.lang.IllegalArgumentException: classLoader cannot be null.
        at com.esotericsoftware.kryo.Kryo.setClassLoader(Kryo.java:975)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.checkKryoInitialized(KryoSerializer.java:477)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:337)
        at org.apache.flink.api.common.typeutils.CompositeSerializer.deserialize(CompositeSerializer.java:151)
        at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextElementLastAccessTimestamp(RocksDbTtlCompactFiltersManager.java:202)
        at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextUnexpiredOffset(RocksDbTtlCompactFiltersManager.java:189)
Mohammad Hossein Gerami
  • 1,360
  • 1
  • 10
  • 26
  • Could you share a little example which reproduces the problem? – Till Rohrmann Mar 19 '20 at 09:09
  • No, I can't. But I show to me this message "INFO org.apache.flink.api.java.typeutils.TypeExtractor - Class class org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.". can be related this message? – Mohammad Hossein Gerami Mar 19 '20 at 09:36
  • I used jackson as input and output type – Mohammad Hossein Gerami Mar 19 '20 at 09:38
  • `ClassNotFoundException: scala/Function0` usually appears when you try launch compiled scala code in one scala version using another scala environment. for example 2.11 and 2.12 – Boris Azanov Mar 19 '20 at 10:32

1 Answers1

3

This is probably an issue of not having user class loader in the context of the native compaction thread.

As a workaround, I would suggest to use a list state with simple elements which type has a fixed byte length when serialized. That would be primitive types and POJOs, maybe implement a custom fixed length serializer if possible. Then the Kryo serializer does not have to be called in the TTL cleanup and requires the user class loader.

azagrebin
  • 390
  • 1
  • 8