1

In RabbitMQ, is there a way to atomically toggle the binding of queues to exchange like the setup below. Q1 & Q2 are alternatingly bound.

X -> Q1
  -> Q2

By binding on an alternating basis, it creates the problem that messages will be either duplicated if Q1 & Q2 are bound at the same time during a toggle or lost if neither are. The requirement is to not lose or duplicate messages. I realise the window for lost messages will be very small, but I'd rather not deal with the logic issues that ensue.

Effectively what I want to achieve is a form of routing from exchange X to a queue (Q1 or Q2), but not make this a concern of the application that publishes the message, which publishes to an exchange X.

One way of achieving this is to have a process consuming message off a intermediate queue Q and implement the toggle routing in that process by giving Q1 and Q2 each an exchange of their own.

This strikes me as inefficient and introduces more moving parts to go wrong.

An alternate way would be to write a custom exchange that acts as a buffer/queue when no queues (X -> Q1 or X -> Q2) are bound, but my Erlang is non existent.

The easiest way would be to toggle the bindings between X -> Q1 and X -> Q2, but (afaik) there is no atomic toggling of bindings, messages could either get lost (unbind Q1, then bind Q2) or duplicated (Bind Q2, then unbind Q1).

The only workaround I can think of would be to temporarily make exchange X unavailable (ACL? rename?) prior to the binding toggle and hope the clients deal with the error conditions.

Is there an out-of-the-box way of achieving this cleanly? If not, what approach would you recommend?

Frederik
  • 2,921
  • 1
  • 17
  • 26
  • When I hacked up the toggling by just flipping the bindings around, I lost about 2000 messages out of a million, while toggling every 300ms. – Frederik Oct 03 '14 at 11:45

2 Answers2

2

Perhaps you want something like the consistent hashing exchange or the modulo hash exchange provided by the rabbitmq sharding plugin (also at the same link).

These plugins take care of partitioning the stream, which seems to be what you want to do actually.

old_sound
  • 2,243
  • 1
  • 13
  • 16
  • I'm not trying to partition the stream - I need the whole stream to be processed by a particular queue consumer at a certain time. Interesting plug-in though. – Frederik Oct 03 '14 at 11:42
1

Toggle bindings may be done with the help of Alternate Exchangs RabbitMQ extension:

  1. Create extra exchange, say X-AE and bind Q2 to X-AE.
  2. Setup X-AE as an Alternate Exchange on X exchange (to route all messages (that one that can't be handled) to Q2 queue).
  3. Unbind Q1 from X (messages from will flow to Q2 via X-AE).
  4. Bind Q2 to X.
  5. Remove alternate exchange X-AE policy from X exchange.
  6. Unbind Q2 from X-AE and remove X-AE exchange.

Note, if you have some other Alternate Exchanges on your X exchange exact steps may vary, but the idea still the same.

pinepain
  • 12,453
  • 3
  • 60
  • 65
  • This works well enough. In my tests I've noticed once that it possible to lose messages, but it is marginal: 1 message out of a million sent at 15kHz, toggling every 50ms (the intended toggle rate is once every hour). It's a bit of a pain to implement, C# Pivotal RabbitMQ client doesn't support policy setting/dropping, so you have to do this via REST on the management API. Also not found a nice way of setting AE policies per exchange. The API seems to require you to handle the AE policy definitions globally (not concurrency friendly) – Frederik Oct 03 '14 at 11:32