27

Can you recommend what Python library to use for accessing AMQP (RabbitMQ)? From my research pika seems to be the preferred one.

A Lee
  • 7,828
  • 4
  • 35
  • 49
Meh
  • 7,016
  • 10
  • 53
  • 76

4 Answers4

26

My own research led me to believe that the right library to use would be Kombu, as this is also what Celery (mentioned by @SteveMc) has transitioned to. I am also using RabbitMQ and have used Kombu with the default amqplib backend successfully.

Kombu also supports other transports behind the same API. Useful if you need to replace AMQP or add something like redis to the mix. Haven't tried that though.

Sidenote: Kombu does currently not support the latest pika release (should you rely on it for some reason). Only 5.2.0 is currently supported, this bit me a while back.

dgorissen
  • 6,207
  • 3
  • 43
  • 52
  • You should update this response now that, according to the issue you link to, you discovered your mistake and that it was not Kombu that was broken. – Brandon Rhodes Mar 28 '11 at 15:16
  • 1
    @Brandon Craig Rhodes: done :) – dgorissen Mar 28 '11 at 15:44
  • https://stackoverflow.com/questions/48524536/can-anyone-please-tell-me-what-are-the-differences-between-pika-and-kombu-messag this provides a comparison between kombu and pika and will probably help future users more than just 'use this library' – Hansang Nov 08 '20 at 22:16
21

Pika is the RabbitMQ recommended library, and py-ampqlib is also mentioned. Depending on what you're using Rabbit for, you might also want to look at Celery (a client library dedicated to distributed queuing).

Again, depending on usage, you might also want to look at Apache's qpid which is a full AMPQ-based client-server alternative to RabbitMQ. One thing that attracted us to qpid was that it seemed to have better robustness on server crashes (queues are persisted in a distributed fashion).

Brandon Rhodes
  • 83,755
  • 16
  • 106
  • 147
SteveMc
  • 1,386
  • 8
  • 11
  • 6
    I just discovered that celery creates a queue per task which is a disappointing weakness http://celeryproject.org/docs/userguide/tasks.html#amqp-result-backend – Michael Dillon Apr 26 '11 at 07:29
  • We haven't found that to be an issue; our usage tasks get consumed quickly (we very rarely expect anything to be sitting around longer than a few seconds). It may be implementation-specific, too - they specifically mention RabbitMQ there. I'd be interested to know if it's causing you difficulties though. – SteveMc Apr 26 '11 at 19:24
  • Never tried celery because it just didn't seem to fit with the overall AMQP architecture. Message queueuing is not just for distributing tasks to pools of workers. – Michael Dillon Apr 27 '11 at 00:54
  • 1
    Ok, good to know. Yeah.. the main plus for celery was that it took a lot of coding away. If we find problems we may end up writing a thin wrapper around pika, as you have done. Good luck with your project! – SteveMc Apr 27 '11 at 02:43
  • 6
    There are a lot of problems on pika, such as publish large message fail, heart-beat timeout fail; – hupantingxue Sep 16 '14 at 08:04
  • Celery can be easily configured to use custom queues and send tasks to those queues instead of creating one queue per task. – danius Dec 04 '14 at 15:55
  • The old "amqp" result backend creates one queue per task, this was to have an amqp based result backend that behaves like a database (any process can retrieve the result). The old "amqp" result backend should probably never be used in production. The "rpc" result backend however is non-persistent, creates one queue per client and is to be used if you need RPC style calls. – asksol Apr 11 '16 at 18:43
3

Having looked at all these libraries, I am now convinced that the right answer is none of them. Instead, build an abstraction layer as a shim over whatever library you choose because you are bound to run into a situation where you have to change libraries.

But do remember, that if you stick to the same version of the AMQP protocol, these libraries do interoperate. Due to different libraries being tested, we had parts of a prototype application running pika, kombu and py-amqplib.

Read this blog about replacing amqplib with pika for a sense of why this is a good idea.

Michael Dillon
  • 31,973
  • 6
  • 70
  • 106
2

I am currently in the middle of making our Python app use SSL. I did not originally develop this application (nor am I a Python developer), so I don't know much about it, but we seem to use the AMQP Client in Twisted.

QPid also has one. Again, I don't know the quality.

the Tin Man
  • 158,662
  • 42
  • 215
  • 303
Travis Stevens
  • 2,198
  • 2
  • 17
  • 25