1

I am relative new to RabbitMQ, and found it is extremely handy and swift, I have used it for communicating small objects by using ruby + bunny gem.

Now I'm trying to pass object around 10~20MB each to exchange, and fanout to its subscribers.

It seemed worked fine, BUT is it a good practice to use RabbitMQ as a publisher? Or should I use something conjecture with RabbitMQ?

c2h2
  • 11,911
  • 13
  • 48
  • 60

2 Answers2

4

You should not send files via AMQP.

Message queues are not databases. Specifically, RabbitMQ was not built with the idea of storing large objects in the queues, because messages are not supposed to be large.

Think about the real world a bit - the postal service for years (not necessarily so much anymore), was optimized for processing letters. If your letter is too fat (heavy), they charge a pretty hefty fee for additional postage. Big messages cost more to move around and disrupt the system. Additionally, your mailbox won't hold large messages - they get left somewhere else - either in a separate package drop or your front door (where they sometimes go missing).

Message queues are the same way. A message typically contains a small piece of data describing an event or other meaningful thing that happened in your application. Usually the data conveyed by a message can be communicated in 100kB or less.

As I mention in this answer, the AMQP protocol (which underlies RabbitMQ) is a fairly chatty protocol. It requires large messages be divided into multiple segments of no more than 131kB. This can add significant of overhead to a large file transfer, especially when compared to other file transfer mechanisms (e.g. FTP, HTTP).

More importantly for performance, the message has to be fully processed by the broker before it is made available in a queue, and it ties up RAM on the broker while this is being done. Putting files in the broker may work for one client and one broker, but it will break quickly when scaling out is attempted. Finally, compression is often desirable when transferring files - HTTP supports gzip compression automatically, while AMQP does not.

What should you do? It is quite common in message-oriented applications to send a message containing a resource locator (e.g. URL) pointing to the larger data file, which is then accessed via appropriate means.

theMayer
  • 15,456
  • 7
  • 58
  • 90
0

If it works and doesn't cause you any problems then great. I would suggest that there may be a time cost for the conversion of each object to a byte array. Clearly the reverse at the consumer side is the case too. As each object is so large that may be consideration, unless speed is not your primary objective. Is is necessary to send such large objects?

One big problem with sending large objects is that they will block and entire connection so if you have more than one channel publishing on the same connection they will have to wait for each connection to finish sending this large object.

see here

robthewolf
  • 7,343
  • 3
  • 29
  • 29