ignite: possible starvation in striped pool with igniteCache

Question

there:

I got the error when use ignite cache. My system select a master node use zookeeper,and has many slave nodes.The master process ignite cache expired values and put in an ignite queue.The slave node provide data into ignite cache use streamer.addData(k,v) and consume the ignite queue. My code is:

ignite cache and streamer :

    // use zookeeper IpFinder
    ignite = Ignition.getOrStart(igniteConfiguration);
    igniteCache = ignite.getOrCreateCache(cacheConfiguration);
    igniteCache.registerCacheEntryListener(new MutableCacheEntryListenerConfiguration<>(
        (Factory<CacheEntryListener<K, CountValue>>)() -> (CacheEntryExpiredListener<K, CountValue>)this
            ::onCacheExpired, null, true, true));

    //onCacheExpired master resolve the expired entry and put in igniteQueue

    cacheConfiguration.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.ONE_MINUTE));

    igniteDataStreamer = ignite.dataStreamer(igniteCache.getName());
    igniteDataStreamer.deployClass(BaseIgniteStreamCount.class);
    igniteDataStreamer.allowOverwrite(true);
    igniteDataStreamer.receiver(StreamTransformer.from((CacheEntryProcessor<K, CountValue, Object>)(e, arg) -> {
        // process the value.
        return null;
    }));

master process the entry expired from the cache,and put in ignite queue:

    CollectionConfiguration collectionConfiguration = new CollectionConfiguration().setCollocated(true);
    queue = ignite.queue(igniteQueueName, 0, collectionConfiguration);

the slaves consume the queue.

but i got error log below after running hours later:

2017-09-14 17:06:45,256 org.apache.ignite.logger.java.JavaLogger warning
WARNING: >>> Possible starvation in striped pool.
    Thread name: sys-stripe-6-#7%ignite%
    Queue: []
    Deadlock: false
    Completed: 77168
Thread [name="sys-stripe-6-#7%ignite%", id=134, state=WAITING, blockCnt=0, waitCnt=68842]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
        at o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:935)
        at o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:850)
        at o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$700(CacheContinuousQueryHandler.java:82)
        at o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1.onEntryUpdated(CacheContinuousQueryHandler.java:413)
        at o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryExpired(CacheContinuousQueryManager.java:429)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:3046)
        at o.a.i.i.processors.cache.GridCacheMapEntry.onTtlExpired(GridCacheMapEntry.java:2961)
        at o.a.i.i.processors.cache.GridCacheTtlManager$1.applyx(GridCacheTtlManager.java:61)
        at o.a.i.i.processors.cache.GridCacheTtlManager$1.applyx(GridCacheTtlManager.java:52)
        at o.a.i.i.util.lang.IgniteInClosure2X.apply(IgniteInClosure2X.java:38)
        at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1007)
        at o.a.i.i.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:198)
        at o.a.i.i.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:160)
        at o.a.i.i.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:854)
        at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1073)
        at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
        at java.lang.Thread.run(Thread.java:745)

score 2 · Answer 1 · answered Sep 14 '17 at 09:58

2

Striped pool is responsible for messages processing. This warning tells you that there is no progress happening on some of the stripes. It may happen due to a bad network connection or when you put massive objects to a cache or a queue.

You may find more information about it in these threads:

http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-td14892.html

http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-message-td15993.html

answered Sep 14 '17 at 09:58

Denis

3,573
8
13

tks for answer,but those two case u give caused by different reason with my case.i start ignite cache on every node with CacheMode.PARTITIONED and every node cache will expired and notify each other node. – 茅破天 Sep 14 '17 at 14:30
But only master node process those expired entries,will this cause WAITING ? – 茅破天 Sep 14 '17 at 14:38
1

If you want to know for sure, try getting a thread dump and analyzing it when such warnings appear. The most common reason is a big size of objects in cache. It causes long pauses during serialization and deserialization. Thread dump will let you see what hung threads are doing. – Denis Sep 14 '17 at 14:39
ok,my object in cache was really not small,I use the cache to count words with hyperLogLog and i store hll object in cache. Let me do some optimization. – 茅破天 Sep 14 '17 at 14:45

ignite: possible starvation in striped pool with igniteCache

1 Answers1