13

I am currently using EventingBasicConsumer from RabbitMQClient.dll C# client, we spawn a different thread to handle each message that is delivered to the consumer.

We encountered a strange behavior, the RabbitMQ server closes connections at times with the error missed heartbeats from client, timeout: 60s. Few moments later the client reports an error saying Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=541. I also see this error client unexpectedly closed TCP connection happening more frequently.

In some situations the clients may take more than 60 seconds to process one job request and this error happens under such conditions.

Is it required that a job be processed within 60 seconds ?, because for our process this can vary between 30 seconds to 5 minutes.

RabbitMQ server: 3.6.6 RabbitMQ.Client.dll (C# client): RabbitMQ.Client.4.1.1

Any insight into this issue is greatly appreciated.

M22an
  • 1,242
  • 1
  • 18
  • 35
  • 2
    You most likely handle all messages inside `Received` handler, which blocks every other activity on channel (I suppose), including heatbeats. Try to handle messages asynchronously: in `Received` event handler put them in some queue (like ConcurrentQueue) and process that queue in another thread. When done with certain message - ACK it on the channel. – Evk May 16 '17 at 13:40
  • I currently do processing like that, but why would that block the channel. I have a prefetch of 100 and I see 100 threads being spawned by the handler. I also see one thread actively listening to RabbitMQ, shouldn't that be taking care of the heartbeat ? – M22an May 16 '17 at 14:26
  • Well that thought came to my mind from description of your issue, I did not verify this myself. – Evk May 16 '17 at 14:54
  • Tried changing to async yesterday, still getting this errors intermittently :( – M22an May 23 '17 at 11:45

2 Answers2

4

I used to run much longer jobs (minutes) with EasyNetQ. It's more high-level client that wraps RabbitMQ.Client.

For me, the reason of these errors is something like Evk wrote in this comment. I would try EasyNetQ as it likely has fetching of messages decoupled from the handling process.

Community
  • 1
  • 1
starteleport
  • 1,231
  • 2
  • 11
  • 21
0

You can increase the TTL timeout in RabbitMq both per queue and per message 

IBasicProperties mqProps = model.CreateBasicProperties();
mqProps.ContentType = "text/plain";
mqProps.DeliveryMode = 2;
mqProps.Expiration = "300000" 

model.BasicPublish(exchangeName,
                   routingKey, mqProps,
                   messageBodyBytes);

Documentaion is at https://www.rabbitmq.com/ttl.html

But I think you're better off by rewriting it to Async pattern for the actual processing of the message.

This might give you inspiration for doing async message processing with RabbitMq

https://codereview.stackexchange.com/questions/42836/listen-to-multiple-rabbitmq-queue-by-task-and-process-the-message

And in this question there are quite allot of information too on async message consumption.

multi-threading based RabbitMQ consumer

Community
  • 1
  • 1
Archlight
  • 2,019
  • 2
  • 21
  • 34
  • 1
    Thanks for your response Archlight, but exactly increasing the TTL going to help me here ? Async pattern might help, I am yet to verify this. Will let you know how it goes – M22an May 18 '17 at 09:08
  • For an immediate pain relief increasing the TTL to over 5 minutes should stop you from getting the timeout. But I would absolutely recommend using Async – Archlight May 18 '17 at 12:25
  • Is the heartbeat duration related to ttl ? I see that there is a way to increase heartbeat timeout as well. but that will also just postpone the issue. – M22an May 18 '17 at 13:09
  • Seems like this would not help, as TTL counts until the message [is delivered](http://www.rabbitmq.com/ttl.html) to consumer. – starteleport May 21 '17 at 17:32