0

If my heapdump Dominator Tree view looks like below, can I assume the major contributor (~1GB) to my heap is the Thread instance created by Weblogic? And in that Thread a reference to ByteArrayOutputStream is the reason for 1GB?

enter image description here

Follow up question: Can I also assume it is due to object serialization? Maybe triggered by Terracotta ehcahce (from the third line. Not because it is next to the ByeArrayOS, because in our code that is the only place serialization can happen)?

Kay Gee
  • 259
  • 1
  • 5
  • 15

2 Answers2

1

Ehcache does indeed rely on Java Serialization the moment your cache goes beyond heap.

However, it will only ever serialize what you put in the cache. So is it possible that some cached mappings have such huge value or even key?

Louis Jacomet
  • 13,661
  • 2
  • 34
  • 43
  • Thanks for your response Louis. We are using distributed cache (Terracotta ehCache). In this case does it serialize every time we add an element in the cache? To replicate the new element in all the nodes of the cluster? Rather than waiting for cache size going beyond the heap? – Kay Gee Nov 14 '17 at 14:54
  • That is correct, the heap is used as a local hot cache, but every write to the cache is immediately pushed to the cluster. Otherwise there would be no consistency guarantees. – Louis Jacomet Nov 15 '17 at 12:36
  • Thanks Louis. I have updated the image in in question to show next objects in the dominator tree. You could notice none of them are closer to the size of byte array. In fact the object that we put in the cache is ~2MB. Will it result in 1GB byte_array size when serialized? – Kay Gee Nov 15 '17 at 18:43
  • Sadly, answering more specifically would require seeing the code of you object. Is it possible that you hold a reference to some shared thing and that gets swallowed by the serialization. Also getting the Ehcache configuration would help. And finally, given the limited infromation, the issue might be something else altogether. – Louis Jacomet Nov 15 '17 at 19:16
  • It is interesting you asked about the shared thing. The object we are adding to the cache in a specific thread is also getting modified by another thread in parallel. This is not by design, but due to an unforeseen scenario. However should it really create an issue. Example, we have an Employee object with List. While one thread is adding the Employee object (e1) in cache, another thread somehow got the e1 reference and adding Tasks (limited numbers) . Maybe this where we have problem, but what exactly should make that 2 MB Employee object to 1 GB? – Kay Gee Nov 16 '17 at 05:42
  • Ah, then that's could be the culprit. Having concurrent modifications while serialization is in progress is tricky. With a HashMap you can get in an infinite loop if reads and writes happen in parallel. See https://stackoverflow.com/questions/13695832/explain-the-timing-causing-hashmap-put-to-execute-an-infinite-loop – Louis Jacomet Nov 16 '17 at 08:38
  • Maybe a stupid question: Shouldn't the first thread take the lock on the object it serializes and not let the other thread to modify? Should I not get ConcurrentModificationException error , like mentioned here https://stackoverflow.com/questions/39216656/ehcache-concurrent-modification-exception-using-spring-struts-application? BTW, our transactionalMode="off", is that the reason it is letting other thread to modify the object? – Kay Gee Nov 19 '17 at 16:47
  • Or since it is same JVM we need to get an explicit lock like mentioned here? http://www.ehcache.org/documentation/2.7/apis/explicitlocking.html Can copyOnRead="true" avoid this issue altogether? – Kay Gee Nov 20 '17 at 17:12
  • It is really up to you to not access non thread safe data structures concurrently. copyOnWrite="true" will not help you, because the issue will pop up while Ehcache is making the copy. This is really a more basic java concurrency issue, and maybe worth a question on its own. – Louis Jacomet Nov 20 '17 at 17:27
  • After making sure we are not making concurrent access to the object we retrieved from the cache, we did not see 1GB byte array issue. However, I may need to do some more reading to understand why concurrent access - T1 is serializing an object and T2 is modifying it, or concurrent reads and writes on HasMap would create problem. The blog you shared in the previous comment is talking about concurrent writes, mainly concurrent resize(), but not abt other scenario. But, like you said it is a separate topic. Thanks for your responses, it helped a lot in closing our issue. – Kay Gee Nov 27 '17 at 19:58
0

The Dominator tree is saying that you have a Weblogic thread holding a ByteArrayOutputStream (and a SerializerObjectOutputStream). The Weblogic thread is a classical worker thread currently processing a request. And it is currently stuck on something.

So, this is the equivalent of

ByteArrayOutputStream in = new ByteArrayOutputStream();
Thread.wait();

The thread is holding a ByteArrayOutputStream and it can't be garbage collected since the thread isn't done with it.

Seeing the serializer makes me think that you are currently deserializing from Ehcache disk or offheap.

Is it possible that you are putting pretty huge objects in your cache? As @louis-jacomet mentioned noticed.

Henri
  • 5,551
  • 1
  • 22
  • 29
  • Thanks Henri for the response. Can you help me understand how you do say it is deserializing from Ehcache? Is it not doing serialization (object to bytearry, than bytearray to object). I can see in this specific thread I am adding an object with the size of 1MB. But the thread size shows 1GB in the heap. Another information that might help is, overflowToOffHeap is set to false. Does ehcache still use offheap? – Kay Gee Nov 14 '17 at 17:32
  • This is Ehcache 2. I'm not an expert on this version so Louis will answer accurately. I'm saying that about serialization because I see the stream and serializer on the stack. Then, why it's so big depends on where the ByteArrayInputStream is coming from. – Henri Nov 15 '17 at 02:55