Rabbitmq queue is not automatically created on server start

Question

I get the following exception when starting my server on one of the environments. However, it works fine on the another environment.

Caused by: org.springframework.amqp.AmqpIllegalStateException: Fatal exception on listener startup
        at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:342)
        at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:363)
        at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
        ... 54 more
    Caused by: org.springframework.amqp.rabbit.listener.FatalListenerStartupException: Cannot prepare queue for listener. Either the queue doesn't exist or the broker will not allow us to use it.

    Caused by: java.io.IOException
            at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
            at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
            at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
            at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:790)
            at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:61)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
            at com.sun.proxy.$Proxy89.queueDeclarePassive(Unknown Source)
            at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:216)

        Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'blah.blah.queue' in vhost '/', class-id=50, method-id=10), null, ""}
                at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:474)
                at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315)
                at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
                at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
                at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)

Note: I have the RabbitAdmin defined in the spring xml definitions as follows:

<bean id="admin" class="org.springframework.amqp.rabbit.core.RabbitAdmin">
    <constructor-arg ref="rabbitConnectionFactory"/>
</bean>

I am not sure why the queues will get autocreated on one environment and not with the other.Any help with this will be great.

<rabbit:queue name="blah.blah.queue">
    <rabbit:queue-arguments>
        <entry key="x-ha-policy" value="all"/>
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:direct-exchange name="blah.blah.exchange">
    <rabbit:bindings>
        <rabbit:binding queue="blah.blah.queue" key="blah.blah.routing.key"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

*Added the queue declaration

Here are some more error logs:

2018-01-04 17:57:46,782 [instance=01] [SimpleAsyncTaskExecutor-1] ERROR org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer (SimpleMessageListenerContainer.java:562) - Consumer received fatal exception on startup
org.springframework.amqp.rabbit.listener.FatalListenerStartupException: Cannot prepare queue for listener. Either the queue doesn't exist or the broker will not allow us to use it.
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:231)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:527)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:790)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:61)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
    at com.sun.proxy.$Proxy89.queueDeclarePassive(Unknown Source)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:216)
    ... 2 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'blah.blah.queue' in vhost '/', class-id=50, method-id=10), null, ""}
    at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
    ... 11 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'blah.blah.queue' in vhost '/', class-id=50, method-id=10), null, ""}
    at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:474)
    at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315)
    at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
    at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)

.Adding some debug logs.. I have not yet been able to figure out the issue.

2018-01-18 16:16:21,583 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.connection.CachingConnectionFactory (CachingConnectionFactory.java:188) - Creating cached Rabbit Channel from AMQChannel(amqp://guest@10.221.57.217:5672/,1)
2018-01-18 16:16:21,596 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.core.RabbitTemplate (RabbitTemplate.java:625) - Executing callback on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://guest@10.221.57.217:5672/,1)
2018-01-18 16:16:21,596 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.core.RabbitAdmin (RabbitAdmin.java:387) - declaring Exchange 'blah.blah.exchange'
2018-01-18 16:16:22,439 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.connection.CachingConnectionFactory (CachingConnectionFactory.java:354) - Detected closed channel on exception.  Re-initializing: null
2018-01-18 16:16:22,652 [instance=01] [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.context.support.DefaultLifecycleProcessor (DefaultLifecycleProcessor.java:170) - Starting bean 'blahblahlistener' of type [class org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer]
2018-01-18 16:16:22,652 [instance=01] [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.aop.framework.JdkDynamicAopProxy (JdkDynamicAopProxy.java:117) - Creating JDK dynamic proxy: target source is SingletonTargetSource for target object [org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1@45ecefcc]
2018-01-18 16:16:22,652 [instance=01] [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer (AbstractMessageListenerContainer.java:361) - Starting Rabbit listener container.
2018-01-18 16:16:22,653 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.listener.BlockingQueueConsumer (BlockingQueueConsumer.java:198) - Starting consumer Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
2018-01-18 16:16:23,078 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.connection.CachingConnectionFactory (CachingConnectionFactory.java:354) - Detected closed channel on exception.  Re-initializing: null
2018-01-18 16:16:23,290 [instance=01] [SimpleAsyncTaskExecutor-1] WARN  org.springframework.amqp.rabbit.listener.BlockingQueueConsumer (BlockingQueueConsumer.java:222) - Reconnect failed; retries left=2
java.io.IOException
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:790)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:61)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
    at com.sun.proxy.$Proxy89.queueDeclarePassive(Unknown Source)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:216)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:527)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'blah.blah.queue' in vhost '/', class-id=50, method-id=10), null, ""}
    at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
    ... 11 more

.

This might help...just in case you didn't know what to search for. https://stackoverflow.com/questions/16370911/how-to-get-spring-rabbitmq-to-create-a-new-queue — leonardseymore, Jan 03 '18 at 12:55
Thanks for sharing the link but I have all thats mentioned there in place. Whats strange is that it works on one environment but not on the other environment. — Ankit Gupta, Jan 03 '18 at 14:03
There must be something different between the environments. I would suggest turn on DEBUG logging for all of `org.springframework` you should see bean creation and admin messages about the declarations. The compare the two logs to figure out what's different. — Gary Russell, Jan 03 '18 at 15:26
Added some more error logs. @GaryRussell: I tried setting DEBUG level on spring.amqp logs but that doesn't really give anything helpful. — Ankit Gupta, Jan 04 '18 at 12:44
That log is not helpful; it's just about the final state with the missing queue; you need to look at the logs to see the been creation and queue declaration and, as I said, compare them between the two environments. — Gary Russell, Jan 04 '18 at 14:19
I added a few more logs.. along with the DEBUG enabled logs.. I compared them between the two environments too.. but have not been able to figure this out yet.. Any help would be highly appreciated. — Ankit Gupta, Jan 19 '18 at 08:02

score 5 · Accepted Answer · answered Jan 19 '18 at 13:01

I was finally able to figure this out. Posting the answer here just in case it helps others who come across a similar situation:

I figured out that the app was declaring another pre-existing queue. However, this pre-existing queue had a dead-letter-routing-key defined in one environment and not in the other... inconsistency across environments. The xml did not have this argument defined. So it worked in the other environment and not in the 1st one. So before going ahead to create any of the non-existing queues, the rabbit channel was getting closed because the arguments mismatched.

Rabbitmq queue is not automatically created on server start

1 Answers1

I was finally able to figure this out. Posting the answer here just in case it helps others who come across a similar situation: