Java ConcurrentSkipListMap getting stuck in some threads

Question

I have this container

private ConcurrentMap<Integer,SortedMap<Long,Object>> users;

Which gets initialized in constructor users = new ConcurrentHashMap<>();

and for each user it gets new ConcurrentSkipListMap<>()

I have this method which is called over and over again in threadpool

private void process(Object obj){
    // SOME CODE

    SortedMap<Long,Object> q = users.get( obj.userId);

    logger.debug("SECTION 1");

    q.headMap( someNumber ).clear();

    logger.debug("SECTION 2");

    q.put( obj.someId, obj );

    logger.debug("SECTION 3");
}

this method is called using executorService = Executors.newScheduledThreadPool(8);

executorService.execute(new Runnable() {
            @Override
            public void run() {
                process(obj);
            }
        });

sometimes it works fine but sometimes (I suppose on high load) for some user process methods (all calls) get stuck and never outputs SECTION 2

and if I swap q.headMap( someNumber ).clear(); and q.put( obj.someId, obj ); it still doesn't output SECTION 2

I also tried LinkedBlockingQueue instead of ConcurrentSkipListMap but I have same issue

it looks like a deadlock but I'm not using any synchronized statement on that structure.

please share your idea if you have one

Sounds like a stupid idea but, are you sure your logger instance is thread-safe? Could it be that the logger is actually causing the (potential) deadlock? Try putting each log statement in a synchronization block just for debugging to see if the problem disappears. — initramfs, Apr 16 '15 at 07:51
it's log4j2. SECTION 1 always gets output and it's getting stuck always after operation on "q" so I don't think that can be an issue but I'll try anyways thank you — MySqlError, Apr 16 '15 at 08:03

score 2 · Answer 1 · answered Apr 16 '15 at 07:56

2

There could multiple causes; the most obvious one is unsynchronized code. Take thread-dumps to easily identify the cause.

Some issues:

In ConcurrentSkipListMap JavaDocs it says at headMap() :

The returned map will throw an IllegalArgumentException on an attempt to insert a key outside its range.

So if obj.someId.compareTo(someNumber) < 0 you can get an IllegalArgumentException at put() which is not treated anywhere.

Also this can lead to race conditions:

q.headMap( someNumber ).clear();
q.put( obj.someId, obj )

if two threads have (thread1) someNumber = (thread2) obj.someId you get a race condition and who know what happens underneath. Moreover, as JavaDocs say clear() is not an atomical operation.

And you did not mention the other place where you use users referring object; how are users populated.

answered Apr 16 '15 at 07:56

Random42

8,989
6
55
86

Thank you but I'm not getting any Exception so we must assume that someNumber is correct. can race condition cause deadlock in thread-safe operations ? – MySqlError Apr 16 '15 at 08:00
@MySqlError It's hard to say without looking at the internals. Though the statement that `clear()` is not atomic just started a revelation in me (I had to check the javadoc myself). It is possible, especially for a SkipListMap, to have a thread attempting a put operation on a half-cleared map, resulting in data corruption (or possible deadlock). **However**, you did say you tried a `LinkedBlockingQueue`, which **does** have atomic clearing, suggesting your problem might be elsewhere (you still need to fix the race condition if you choose to use the SkipListMap though). – initramfs Apr 16 '15 at 08:08
1

@MySqlError But how do you know you have deadlock? Did you take thread-dumps? To answer your question: I do not think so (I believe JDK classes are built well enough not to deadlock, but it depends on internal implementation). Also how are you sure you don't get exceptions? It won't show up in logs unless you get the `Future` an treat it (which your code doesn't). See here for more details - http://stackoverflow.com/questions/2248131/handling-exceptions-from-java-executorservice-tasks – Random42 Apr 16 '15 at 08:11
You're right it should not be a deadlock because if it's deadlock than it should deadlock entire threadpool which is only 8 threads in my case. so Runtime Exception will be silent if it happens in threadpool ? if that's so I guess that might be a issue I'm dealing with – MySqlError Apr 16 '15 at 08:23

score 0 · Answer 2 · answered Apr 16 '15 at 11:56

It seams that problem is not that code gets stuck but it throws exception which is handled silently by theadpool. I tried to push custom ThreadFactory with logging enabling but it still doesn't works. solution is to provide proper try catching for RuntimeException s

So lesson I've learned is that if part of your code doesn't get executed inside threadpool it might be because of the RuntimeException and you should provide more try catching to find the cause

Java ConcurrentSkipListMap getting stuck in some threads

2 Answers2