6

In my application, I've noticed that HornetQ 2.4.1 has been piling up message journal files, (sometimes into the thousands.) I'm using HornetQ via JMS Queues and we're using Wildfly 8.2. Normally, when starting the server instance, HornetQ will have 3 messaging journals and a lock file.

The piling up of message journal files has caused issues when restarting the server, we'll see a log that states:

HQ221014: 54% loaded

When removing the files, the server loads just fine. I've experimented some, and it appears as though messages in these files have already been processed, but I'm not sure why they continue to pile up over time.

Edit 1: I've found this link that indicates we're not acknowledging messages. However, when we create the session like so connection.createSession(false,Session.AUTO_ACKNOWLEDGE);.

I'll continue looking for a solution.

dimwittedanimal
  • 656
  • 1
  • 13
  • 29
  • 1
    Are you using XA transactions or JMS transactions? And how do you define your resource adapter in standalone.xml? – ewramner Dec 19 '17 at 20:54
  • I'm using XA Transactions. I'll get more info regarding the resource adapter in a bit. – dimwittedanimal Dec 19 '17 at 23:55
  • 1
    If you are using XA transactions it doesn't matter what you do with the session above. The messages should be acknowledged when the XA transaction is committed. I don't know about HornetQ, but in general this is an area where many products are a bit buggy. – ewramner Dec 20 '17 at 12:33
  • Something I found is that there are pileups of transactions in the "SCHEDULE_CANCEL" state. I have a feeling too many transactions. When ARJUNA is trying to cancel them, it only gets through one at a time before the Reaper Element wedges and cannot move on. I'm going to try to change the database Isolation level and see if that fixes our problem. – dimwittedanimal Dec 20 '17 at 13:48

1 Answers1

0

I've come to find out that this has been caused (for one reason or another, I currently believe it has something to do with server load or network hangs) by the failure of calling the afterDelivery() method. I'm addressing this by not hitting that queue so often. It's not elegant, but it serves my purpose.

See following HornetQ messages I found in the logs:

HQ152006: Unable to call after delivery
javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction. at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:87)

HQ222144: Queue could not finish waiting executors. Try increasing the thread pool size

HQ222172: Queue jms.queue.myQueue was busy for more than 10,000 milliseconds. There are possibly consumers hanging on a network operation
dimwittedanimal
  • 656
  • 1
  • 13
  • 29