20

I'm spec'ing a design right now that uses RabbitMQ as a message queue. The message are going to have a JSON body, and for one message in particular I'd like to add a small binary file.

What I'd like to know is, should the binary file's data be part of the JSON message, or can it be appended to the message separately?

riqitang
  • 3,241
  • 4
  • 37
  • 47

3 Answers3

30

Since RabbitMQ message payload is just a binary array you should encode your message body with 3 fields:

  1. File size
  2. Binary data of a file
  3. Json

I disagree with a previous answer about embedding a file in json. If you encode file data inside of json you will get wasted space because of json escaping + unnecessary CPU usage because of json encoding/decoding of the file data + you will need to read file data twice (once for json deserialization and once more to copy it where it needs to go)

cohadar
  • 4,709
  • 2
  • 20
  • 21
22

The message is a single byte stream, it also contains a header but that is less relevant. I suggest that you take one of two approaches. Your JSON object contains a field that is the byte stream of the binary file. Alternatively it gives the address where the file can be downloaded. If the binary file is small then the former should work fine. Otherwise you may prefer the latter.

robthewolf
  • 7,343
  • 3
  • 29
  • 29
  • Those were my thoughts exactly, it's reassuring to hear them come from somebody else. Thank you! – riqitang Feb 27 '14 at 15:47
  • 2
    Binary data needs to be escaped before it can be serialized into JSON. See http://stackoverflow.com/q/1443158/1196816. It is probably better to just pass the binary data as the body of the RabbitMQ publish call. – Mukul M. Aug 13 '15 at 21:09
5

We use RabbitMQ and file transfer. It's working a little slow, but for updating far away customers it is duable. I would recommend you following guidelines: Create a message structure for each block you send with command, offset and CRC32, data length, maximum 60kByte per block of data, give blocks a counter, do a sha256 at the end to make sure data is correct, make tar optional so the data can be much smaller and transmission is faster.. make a directory thing, to see which files need to update... use a broadcast event to see who's available and update 1 customer at a time with client.company.update orso, where a customer listens to .company. have fun! p.s. we also created a linux-service for this, which starts automatic at the customer.

Bart Mensfort
  • 995
  • 11
  • 21
  • 8
    _duable_ - doable or durable? – mbx Mar 27 '19 at 07:38
  • 1
    @mbx, Why not both? – Raja Anbazhagan Dec 31 '21 at 10:33
  • 2
    Sorry for the type, it's doable in my case. Here an additional trick. Since our Server is connected to our home network, our external IP may change a few times per year. Therefore, we put the IP-address in a file on our marketing website. So a customer first reads this file, then knows what IP address to connect. Also,, every configuration file where the IP-address/password is stored should be written encrypted, backwards, upside down, contain 90% rubbish data etc to fool hackers. – Bart Mensfort Jan 02 '22 at 18:05