5

I am currently building a Mule ESB server application which uses a request-response jms connector. Since it is being used in a highly concurrent environment, we enabled spring jms cache in our MQ config.

<spring:beans>
    <mule>
        <!-- MQ Factory -->
        <spring:bean id="testMsgMqFactoryBean1" name="testMsgMqFactory1" class="com.ibm.mq.jms.MQQueueConnectionFactory">
            <spring:property name="channel" value="${test.msg.mq.channel.1}" />
            <spring:property name="queueManager" value="${test.msg.mq.queueManager.1}" />
            <spring:property name="hostName" value="${test.msg.mq.hostName.1}" />
            <spring:property name="port" value="${test.msg.mq.port.1}" />
            <spring:property name="transportType" value="${mq.jms.transportType}" />
        </spring:bean>
        <spring:bean id="testMsgMqFactoryBeanCache1" class="org.springframework.jms.connection.CachingConnectionFactory">
            <spring:property name="targetConnectionFactory" ref="testMsgMqFactoryBean1" />
            <spring:property name="sessionCacheSize" value="${test.threading.profile.maxThreadsActive}" />
            <spring:property name="cacheConsumers" value="false" />
            <!-- <spring:property name="cacheProducers" value="false" /> -->
        </spring:bean>
        <!-- MQ Connector 1 -->
        <jms:custom-connector name="testMsgMqConnector.1" class="org.mule.transport.jms.websphere.WebsphereJmsConnector" doc:name="Custom JMS">
            <spring:property name="specification" value="1.1" />
            <spring:property name="connectionFactory" ref="testMsgMqFactoryBeanCache1" />
            <spring:property name="persistentDelivery" value="false" />
            <spring:property name="disableTemporaryReplyToDestinations" value="true" />
            <spring:property name="numberOfConsumers" value="${test.threading.profile.maxThreadsActive}" />
            <spring:property name="maxRedelivery" value="-1" />
            <receiver-threading-profile maxThreadsActive="${test.threading.profile.maxThreadsActive}" maxBufferSize="${test.threading.profile.maxBufferSize}" maxThreadsIdle="${test.threading.profile.maxThreadsIdle}"/>
            <reconnect frequency="${mq.jms.reconnection.frequency}" count="${mq.jms.reconnection.count}" blocking="false" />
        </jms:custom-connector>

        <!-- msgworks inbound and outbound MQ setup -->
        <!-- Rewards -->
        <jms:endpoint exchange-pattern="request-response" queue="${test.msg.mq.inbound.account.queue}" name="testQueue1" connector-ref="testMsgMqConnector.1" doc:name="JMS" />
    </mule>
</spring:beans>

This configuration runs fine when the client uses a static replyTo queue. However, we have some customers who are using dynamic/temporary replyTo queue. Since org.springframework.jms.connection.CachingConnectionFactory caches producers, for every temporary replyTo queue, a producer object is cached and never closed. After processing hundreds of requests, the application started throwing exceptions:

********************************************************************************
Message               : Failed to create and dispatch response event over Jms destination "queue://QMGR1/TESTret5a975v53AF980F2006BE02?targetClient=1". Failed to route event via endpoint: null. Message payload is of type: JMSTextMessage
Code                  : MULE_ERROR-42999
--------------------------------------------------------------------------------
Exception stack is:
1. MQJE001: Completion Code 2, Reason 2017 (com.ibm.mq.MQException)
  com.ibm.mq.MQQueueManager:2808 (null)
2. MQJMS2008: failed to open MQ queue TESTret5a975v53AF980F2006BE02(JMS Code: MQJMS2008) (javax.jms.ResourceAllocationException)
  com.ibm.mq.jms.MQQueueServices:398 (http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/ResourceAllocationException.html)
3. Failed to create and dispatch response event over Jms destination "queue://QMGR1/TESTret5a975v53AF980F2006BE02?targetClient=1". Failed to route event via endpoint: null. Message payload is of type: JMSTextMessage (org.mule.api.transport.DispatchException)
  org.mule.transport.jms.JmsReplyToHandler:173 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/transport/DispatchException.html)

After investigating the MQ error code (MQJE001: Completion Code 2, Reason 2017), I found that the reason behind this error is because we never closed producers, and the producers exhausted MQ Handles on the Queue Manager. The quick and easy fix is to uncomment the line in spring jms cache config to close Producers each and every time.

<spring:bean id="testMsgMqFactoryBeanCache1" class="org.springframework.jms.connection.CachingConnectionFactory">
    <spring:property name="targetConnectionFactory" ref="testMsgMqFactoryBean1" />
    <spring:property name="sessionCacheSize" value="${test.threading.profile.maxThreadsActive}" />
    <spring:property name="cacheConsumers" value="false" />
    <spring:property name="cacheProducers" value="false" />
</spring:bean>

Now I am not seeing the MQ issue, but came up with another performance issue, because no producers are cached, so a new producer is created every single time.

My question is, how to deal with this scenario? Since the client won't change the way they receive reply messages from temporary queues, how can we avoid exhausting MQ handlers without impacting the performance.

Thank you very much - Lei

davy_wei
  • 139
  • 13
  • This new application is going to replace the legacy application which uses message driven beans. The legacy application consumes the same clients' requests with dynamic replyTo queues, and it doesn't have the performance issue. The legacy application doesn't cache the senders (producers) either. Any ideas? – davy_wei Jul 02 '14 at 04:10
  • try { QueueSession session = mqSession.getSession(); replyToQueue = createReplyToQueue(session); sender = session.createSender(replyToQueue); TextMessage message = session.createTextMessage(invocation.getResponse()); message.setJMSCorrelationID(getJmsRequest().getJMSMessageID()); getLogger().logDebug(methodName, "Sending response to [" + replyToQueue + "]"); sender.send(message); getLogger().logDebug(methodName, "Sent response to [" + replyToQueue + "]"); setJmsResponse(message); return; } finally { JmsUtil.closeJmsResource(sender); } – davy_wei Jul 02 '14 at 04:14
  • Do you know why the producer is never closed? Are you creating that as part of your own code or does mule create it? If so, it might be worth reporting that to their issue tracker. – Stephane Nicoll Jul 02 '14 at 16:47
  • Stephane, it's not Mule doesn't close the producers, it's because we used spring jms CachingConnectionFactory. It caches session and proxies jms APIs for caching purposes (consumers, producers). – davy_wei Jul 02 '14 at 19:12
  • I am confused. You need to close the session at some point and that's going to clean up things. The fact that `CachingConnectionFactory` caches those as nothing to do with the problem. The only thing this does is to make sure to return you a cached producer for your current transaction but you (or Mule) are responsible to close the Session! (check `physicalClose()` in that class) – Stephane Nicoll Jul 03 '14 at 07:16

1 Answers1

1

This is a very interesting use case. However, I'm afraid there is nothing out of the box to fix this. There are the more obvious solutions: disable the caching or extend the spring cache provider.

Temporary queues and performance are definitely not two things that you can have at the same time. I would suggest another posibility:

If you are using the temporary queues to have the responses come back to a given consumer only, probably in adition to having old messages discarded on reconnection:

You could use a well know queue for the replies, using a combination of a header with the hostname that should receive the message plus a selector different on each consumer node and a TTL on the messages sent to make them dissapear after a while.

Víctor Romero
  • 5,107
  • 2
  • 22
  • 32