Does RabbitMQ have any concept of message priority? I have an issue where some more important messages are being slowed down due to less important messages sitting before them in the queue. I want the high-priority ones to take precedence and move to the front of the queue.
I know I can approximate this using two queues, a "fast" queue and a "slow" queue, but that seems like a hack.
Does anyone know of a better solution using RabbitMQ?
The answers on this question are out-of-date. As of RabbitMQ 3.5.0, there is now in-core support for AMQP standard per-message priorities. The documentation has all the gory details, but in short:
More interesting caveats are in the docs. It's well worth reading them.
Rabbit has no concept of priority other than, as Brian succinctly puts it, the one in front gets there first. ;-)
I would suggest implementing a set of queues that serve to service your particular messaging need and have these queues model your prioritisation need by, say, calling them 'MyQueueP1', 'MyQueueP2' and so on and then have our consumer(s) check P1 before P2 (etc.) and service messages from there first.
If you then have a message that is high priority you would publish it to the appropriate priority queue by way of a suitable routing key and voila.
[update] Check this question: In a FIFO Qeueing system, what's the best way the to implement priority messaging
[update] As per recent RabbitMQ release 3.5.0 this answer is now outdated and should be considered valid for only versions prior to this release. https://stackoverflow.com/a/29068288/489888
IIRC RabbitMQ still uses the AMQP protocol version 0.9.1 (get the spec here). The spec definitely mentions message priority:
Messages may have a priority level. A high priority message is sent ahead of lower priority messages
waiting in the same message queue. When messages must be discarded in order to maintain a specific
service quality level the server will first discard low-priority messages.
And:
Note that in the presence of multiple readers from a queue, or client transactions, or use of priority fields,
or use of message selectors, or implementation-specific delivery optimisations the queue MAY NOT
exhibit true FIFO characteristics.
The spec says priority is a MUST, so I guess RabbitMQ should implement it, but you may want to consult its documentation.
Yes, RabbitMQ supports priority queues.
To make a queue work as a priority queue, supply property x-max-priority when declaring the queue.
Property x-max-priority defines the maximum priority number the queue supports.
In Java, you can do like below:
Map<String, Object> props = new HashMap<>();
props.put("x-max-priority", 10); // max priority number as 10
channel.queueDeclare(QUEUE_NAME, durable, false, false, props);
To publish messages of a particular priority, do like below:
String message = "My message with priority 7";
AMQP.BasicProperties.Builder basicProps = new AMQP.BasicProperties.Builder();
basicProps.contentType("text/plain")
.priority(7);
channel.basicPublish("", QUEUE_NAME, basicProps.build(), message.getBytes());
Here's a C# code sample that defines a queue with a range of priorities:
using RabbitMQ.Client;
public void Setup()
{
ConnectionFactory factory = new() { host = "", username = "", password = "" };
var connection = factory.CreateConnection();
var model = connection.CreateModel();
var args = new Dictionary<string, object>
{
{ "x-min-priority", 0 },
{ "x-max-priority", 9 }
};
model.QueueDeclare("Queue1", arguments: args);
}
And here's a function for sending messages with priority:
private static void Send(IModel model, string queue, string message, byte priority)
{
var body = Encoding.UTF8.GetBytes(message);
var basicProperties = model.CreateBasicProperties();
basicProperties.Priority = priority;
model.BasicPublish(exchange: "",
routingKey: queue,
basicProperties: basicProperties,
body: body);
}
As mentioned by other people, Rabbit doesn't strictly obey these priorities.
If you send a message outside the range defined for the queue, it's treated as the min or max value.
We could make rabbitmq a distributed priority queue by installing the plugin rabbitmq_priority_queue from https://www.rabbitmq.com/community-plugins.html . You have to download the plugin rabbitmq_priority_queue-3.3.x-72d20292.ez and put inside plugins folder of your rabbit mq installation directory. Restart the server. Now you can insert items into the queue with a priority and consume it accordingly , have pasted the sample code in How to poll the RabbitMQ to get messages in order of priority continuously? .
RabbitMQ / AMQP definitely has a concept of message priority - the message at the head of a queue gets priority over the one behind it, and that one gets priority over the one behind it, and so on, ad infinitum.
Can you change that model? Nope! :)
If it implemented prioritization, it would not be an MQ.
MQs are email for data. And in all data transmission preserving order is vital. If you switch order, deletes happen before inserts, updates get out of sequence. Nothing works properly.
You may have a valid implementation, there are a few exceptions, but I have found that most priority queues are designed because people are thinking shallow thoughts about their system architecture and the interactions of the parts therein. Preserving the order of things is almost always the right thing to do, both for in-entity, and inter-entity interactions.
In order to say an event A can have a priority higher than event B, the two events must be de-coupled always. And when that happens one wonders why they exist in the same queue structure at all. Then again, if it is payload related, the calculation effort for that payload is also going to impact the performance of the system, so deciding sooner, ie before making the payload makes sense.
