21

We're using a RabbitMQ server for messaging between applications. We have a need to create a central log for all amqp messages coming into the Rabbit server. Our purpose is not temporary debugging, but rather auditability. Ideally, I'd be able to log to a specified file to begin with, and later log to an external system such as Logly or Splunk.

I've explored turning on Firehose and using the tracing plugin, but the queues in question are not durable. I'm also not sure if these solutions will work if new queues and exchanges are added to the virtual host after logging hast started. These tools seem designed more for temporary debugging that what I need.

I'd love to hear your ideas. At this point I'm worried I'll have to setup a network monitor to intercept and log the messages before they reach Rabbit.

Chris McKenzie
  • 3,681
  • 3
  • 27
  • 36
  • 2
    Would exchange to exchange bindings help you in your scenario? http://www.rabbitmq.com/blog/2010/10/19/exchange-to-exchange-bindings/ – julio.g Nov 13 '13 at 03:03
  • That's a nice idea. Unfortunately I don't think it solves our problem as our exchanges are created dynamically. Our worst case scenario for an approach like this is to mandate that devs use a specific library to send messages to Rabbit and have the library duplicate messages to a logging exchange. This seems a bit draconian and requires a large number of application updates. Our ideal solution is to find a way to make it work on the server for any exchange that is created or will be created. – Chris McKenzie Nov 13 '13 at 16:53

2 Answers2

5

Why would you say that the firehose won't work? You can create the queues for listening as you want, i.e.: durable and so on. Then if you need every message, you can do a binding using the # character.

See: http://www.rabbitmq.com/firehose.html

Performance will drop as explained, but I wouldn't ditch the solution without testing performance.

old_sound
  • 2,243
  • 1
  • 13
  • 16
  • 1
    My experimentation showed that the tracing functionality did not survive a reboot of the server. It had to be setup and configured again. Most google searches on this topic are about people trying to log *to* queues, not logging *from* queues. The '#' binding is interesting--I looked for something like that but never found it. Where is that documentation? I've been over and over the Rabbit docs and haven't seen that. – Chris McKenzie Dec 19 '13 at 16:33
  • 1
    See here: http://www.rabbitmq.com/tutorials/tutorial-five-python.html it says "# (hash) can substitute for zero or more words." – old_sound Dec 26 '13 at 00:50
0

The RabbitMQ Tracing Plugin extends the functionality of the built-in trace feature, allowing you to send traced messages straight to a log file (which can be json format, if desired).

Regarding performance impact: In the original rabbitmq-tracing repo, the following was stated in 2015 or earlier: "On a few year old developer-grade machine, rabbitmq-tracing can write about 2000 msg/s to a log file."

So performance impact should not be an issue except in very high throughput situations.

Josh Withee
  • 9,922
  • 3
  • 44
  • 62