Using RabbitMQ is there a way to look at the queue contents without a dequeue operation?

Question

As a way to learn RabbitMQ and python I'm working on a project that allows me to distribute h264 encodes between a number of computers. The basics are done, I have a daemon that runs on Linux or Mac that attaches to queue, accepts jobs and encodes them using HandBrakeCLI and acks the message once the encode is complete. I've also built a simple tool to push items into the queue.

Now I want to expand the capabilities of the tool that pushes items into the queue so that I can view what is in the queue. I'm aware of the ability to see how many items are in the queue, but I want to be able to get the actual messages so I can show what movie or TV show is waiting to be encoded yet. The idea is that the queue manager would receive messages from the encoder clients when a job has completed and then refresh the queue list.

I know there is a convoluted way of keeping the queue manager's list in sync with the actual work queue but I'd like this to be "persistent" in that I should be able to close the queue manager and reopen it later to see the queue.

I received this via twitter - "no - RabbitMQ's queues are pure FIFO structures and there's no peek. However, look at basic.consume/get with acks" — Dustin, Jan 15 '11 at 15:49

score 47 · Accepted Answer · answered Feb 15 '12 at 01:56

Queue browsing is not supported directly, but if you declare a queue with NO auto acknowledgements and do not ACK the messages that you receive, then you can see everything in it. After you have had a look, send a CANCEL on the channel, or disconnect and reconnect to cause all the messages to be requeued. This does increment a number in the message headers, but otherwise leaves the messages untouched.

I built an app where message ordering was not terribly important, and I frequently scanned through the queue in this way. If I found a problem, I would dump the messages into a file, fix them and resubmit.

If you only need to peek at a message or two once in a while you can do that with the RabbitMQ management plugin.

In addition, if you only need a message count, you can get that every time you declare the queue, or on a basic.get command.

Just to expand on the message count getting, you can declare a queue with passive=True which will not create a queue if it doesn't exist. If it does, you'll get back number of messages and consumers. — Dave Foster, May 29 '12 at 13:48
RabbitMQ has a proprietary extension called Firehose Tracer which gives access to all posted messages. Details here: https://www.rabbitmq.com/firehose.html — Tom Pohl, Jul 06 '20 at 11:25

score 7 · Answer 2 · answered Sep 01 '17 at 06:27

7

@MichaelDillon based on your answer to make others life easier I am putting here a no_ack example:

#!/usr/bin/env python
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='Q.hello')


def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)
    # ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(callback, queue='Q.hello')

print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

answered Sep 01 '17 at 06:27

Tomasz

1,001
8
8

The `ch.basic_ack(delivery_tag=method.delivery_tag)` effectively works as a "delete from queue" command this case. +1 – AlejandroVD Sep 22 '17 at 21:24
2

This is super helpful, but I'm having trouble figuring out a way to do this that doesn't block indefinitely. Basically I just want to peek once at the queue at a fixed number of messages, and don't need to sit and wait for more messages to come in. Any ideas? – zjm555 Jul 08 '20 at 12:45

score 1 · Answer 3 · answered Jan 26 '11 at 20:09

1

What you want to do is called browsing the queue, although I gather from this that RabbitMQ does not yet support that.

answered Jan 26 '11 at 20:09

Lee at NCR

51
5

10

It is not supported directly, but if you declare a queue with NO auto acknowledgements and do not ACK the messages that you receive, then you can see everything in it. After you have had a look, send a CANCEL on the channel, or disconnect and reconnect to cause all the messages to be requeued. This does increment a number in the message headers, but otherwise leaves the messages untouched. – Michael Dillon Jun 11 '11 at 02:52
5

@MichaelDillon - That comment should be an answer. – cdeszaq Feb 14 '12 at 15:13
4

@cdeszaq: yes you are right. I wrote it as an answer with a bit more info. – Michael Dillon Feb 15 '12 at 01:56

Using RabbitMQ is there a way to look at the queue contents without a dequeue operation?

3 Answers3

Linked