Pika blocking_connection.py random timeout connecting to RabbitMQ

Question

i have a rabbit mq running on machine

both client and rabbitMQ are running on the same network
rabbitMQ has many clients
i can ping client from rabbitMQ and back
longest latency measured between the machine is 12.1 ms
network details : Standard Switch network (network of virtual machines running on single physical machine - using vmware VC)

im getting random timeouts when initializing RPC connection

/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py

problem is that the timeout isn't consistent and happens from time to time.

when manually testing this issue and running the blocking_connection.py 1000 times from the same machine that it fails no timeout accrue.

this is the error i get when failing :

2013-04-23 08:24:23,396 runtest-trigger.24397 24397 DEBUG      producer_rabbit initiate_rpc_connection Connecting to RabbitMQ RPC queue rpcqueue_java on host: auto-db1
2013-04-23 08:24:25,350 runtest-trigger.24397 24397 ERROR      testrunner go   Run 1354: cought exception: timed out
Traceback (most recent call last):
  File "/testrunner.py", line 193, in go
    self.set_runparams(jobid)
  File "/testrunner.py", line 483, in set_runparams
    self.runparams.producers_testrun = self.initialize_producers_testrun(self.runparams)
  File "/basehandler.py", line 114, in initialize_producers_testrun
    producer.set_testcase_checkout()
  File "/baseproducer.py", line 73, in set_testcase_checkout
    self.checkout_handler = pm_checkout.get_producer(self.testcasecheckout)
  File "/producer_manager.py", line 101, in get_producer
    producer = self.load_producer(plugin_dir, producer_name)
  File "/producer_manager.py", line 20, in load_producer
    producer = getattr(producer_module, 'Producer')(producer_name, self.runparams)
  File "/producer_rabbit.py", line 13, in __init__
    self.initiate_rpc_connection()
  File "/producer_rabbit.py", line 67, in initiate_rpc_connection
    self.connection = pika.BlockingConnection(pika.ConnectionParameters( host=self.conf.rpc_proxy))
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py", line 32, in __init__
    BaseConnection.__init__(self, parameters, None, reconnection_strategy)
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py", line 50, in __init__
    reconnection_strategy)
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/connection.py", line 170, in __init__
    self._connect()
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/connection.py", line 228, in _connect
    self.parameters.port or  spec.PORT)
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py", line 44, in _adapter_connect
    self._handle_read()
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py", line 151, in _handle_read
    data = self.socket.recv(self._suggested_buffer_size)
timeout: timed out

please assist

What's your network setup, where is the client and server located, have you measured the network latency between these two, do you know if the MQ has many clients? This all can drive timeout errors, and without these (and more) details, solving the issue will be difficult. — SpankMe, Apr 23 '13 at 07:31
thanks for the advise ive added the information in the question — Nimrod007, Apr 23 '13 at 07:39
On the same network ~12ms may a bit high, but you've not provided enough details on what kind of network that is. However, since you're obviously seeing timeouts, have you tried increating the `socket_timeout` value as in https://pika.readthedocs.org/en/latest/connecting.html ? — SpankMe, Apr 23 '13 at 07:43

score 1 · Answer 1 · edited May 23 '17 at 12:14

I had a similar issue. If everything looks fine, then you most likely have some sort of miss configuration, e.g. bad binding. If miss configured, then you'll get a timeout because the script can't reach where it thinks it needs to go, so the error can be miss leading in this case.

For my problem, I specifically had issues with both my rabbitmq.config file and my bindings and had to use my python solution shown in: RabbitMQ creating queues and bindings from a command line over the command line example I showed. Once updated and configured properly, everything worked fine. Hopefully this gets you in the right direction.

itzMEonTV · Answer 2 · 2015-01-23T05:27:47.447

Pika provides some time out issue when connecting different hosts.Solution is to pass a socket_timeout argument in connection parameter.Pika should upgrade to >=0.9.14

credentials = pika.PlainCredentials(RABBITMQ_USER, RABBITMQ_PASS)
connection = pika.BlockingConnection(pika.ConnectionParameters(
    credentials=credentials,
        host=RABBITMQ_HOST,
         socket_timeout=300))
channel = connection.channel()

Pika blocking_connection.py random timeout connecting to RabbitMQ

2 Answers2

Linked