21

I've set up RabbitMQ in order to parse some 20.000 requests from an external API but it keeps timing out after a few minutes. It does get to correctly parse about 2000 out of the total 20.000 requests.

The log file says:

=INFO REPORT==== 16-Feb-2016::17:02:50 ===
accepting AMQP connection <0.1648.0> (127.0.0.1:33091 -> 127.0.0.1:5672)

=ERROR REPORT==== 16-Feb-2016::17:03:21 ===
closing AMQP connection <0.1648.0> (127.0.0.1:33091 -> 127.0.0.1:5672):
{writer,send_failed,{error,timeout}}

I've already increased the heartbeat value but I cannot figure out why it's timing out. Configuration is: Ubuntu 14.04, NGINX 1.8.1, RabbitMQ 3.6.0

I'd appreciate your time and input !

surfer01
  • 349
  • 1
  • 2
  • 11
  • There seem to be some details missing here: What is doing the "parsing"? Where is this log message taken from? Is the timeout happening while publishing messages *to* RabbitMQ, or while consuming them *from* it? Your tags mention PHP, so is there some relevant PHP code you could show us? – IMSoP Feb 16 '16 at 18:49
  • Thank you for replying ! Indeed, a consumer written in PHP is parsing some JSON data from an external API. The log messages have been taken from the main RabbitMQ log file: /var/log/rabbitmq/rabbit@hostname.log . I wouldn't say the PHP code is relevant to the error but it does write a brief output to a .txt file. – surfer01 Feb 16 '16 at 22:25
  • It might be worth posting (or creating a [mcve] of) your PHP code, just in case there's something you're doing that could be triggering the timeout. At the moment, we don't have much to go other than "there's a timeout error somewhere". Similarly, any code around the way the messages are published (presumably there's something picking items from the API and putting them into Rabbit somehow?) – IMSoP Feb 16 '16 at 23:10

2 Answers2

34

I've just solved a similar problem in python. In my case, it was solved by reducing the prefetch count on the consumer, so that it had fewer messages queued up in its receive buffer.

My theory is that the receive buffer on the consumer gets full, and then RMQ tries to write some other message to the consumer's socket and can't due to the consumer's socket being full. RMQ blocks on this socket, and eventually timeouts and just closes the connection on the consumer. Having a smaller prefetch queue means the socket receive buffer doesn't get filled, and RMQ is able to write whatever bookkeeping messages it was trying to do and so doesn't timeout on its writes nor close the connection.

This is just a theory though, but it seems to hold in my testing.

In Python, setting the prefetch count can be done like so:

subChannel.basicQos(10);

(Thanks to @shawn-guo for reminding me to add this code snippet)

tul
  • 1,709
  • 16
  • 15
13

Add more to @tul's answer.

subChannel.basicQos(10);

Reducing consumer prefetch count does eliminate this timeout exception.
The default prefetch count is unlimited.

Shawn Guo
  • 3,169
  • 3
  • 21
  • 28