1

We are setting up a database with a front end program that parses csv files and dumps them into the DB. Files are coming from many locations at the same time and need to be persisted if anything goes wrong.

Is a message broker like rabbitmq a good application for this? I need a queue of some sort that can be read from the back end.

bischoffingston
  • 637
  • 9
  • 27
  • 1
    See [Can rabbitMQ be used for avatars in a chat server?](http://stackoverflow.com/q/22842096), [Sending binary file through RabbitMQ](http://stackoverflow.com/q/22070639), [Send files through RabbitMQ](http://stackoverflow.com/q/27175543) questions for some highlights. – pinepain May 31 '16 at 19:04

1 Answers1

4

Is a message broker like rabbitmq a good application for this?

Yes, but I wouldn't recommend you to send large files to RabbitMQ especially if you'd like to use persistent messages. If this is the case, I'd save a file to a storage (cloud, cache, etc) and send its name/path in a message. A worker process would pick up a message, then download a file, parse it and save its contents to database. This way you can distribute files among worker processes and reduce load on broker. It would allow you to scale easily.

Alex G.
  • 909
  • 1
  • 9
  • 16