0

I have a scenario with these particular demands:

  • Production ready & stable.
  • Point to point connection, with the producer behind a firewall and a consumer in the cloud. It might be possible to split the traffic between a couple of producers\consumers, but all the traffic still has to traverse a single WAN connection which will probably be the bottleneck.
  • High throughput - something along the order of 300 Mb/sec (may be up to 1Gb!). Message sizes vary from ~1KB to possibly several MBs.
  • Guaranteed delivery a must - every message has to arrive at the consumer eventually, so we need to start saving messages to disk in the event of a momentary network outage or risk running out of memory.
  • Message order is not important, messages are timestamped and can be re-arranged at the consumer.
  • Highly preferable but not as important - should run on both linux & windows (JVM seems the obvious choice)

I've been looking at so many MQs lately, and I don't have any hands-on experience with any.
Thought it will be a better idea to ask someone with experience.
We're considering mostly Kafka, but I'm not sure it's the best for our use case, seems to be tailored to distributed deployment & mutliple topics\consumers\producers. Also, definitely not production ready on windows. What about Apache ActiveMQ or Apollo\Artemis? RabbitMQ seems not to be a good fit for our performance requirements. Or maybe there's some Java library that has the features we need without a middleman broker?
Any help making sense of this kludge would be greatly appreciated.

Tolstoyevsky
  • 452
  • 5
  • 13
  • Have a look at http://stackoverflow.com/questions/731233/activemq-or-rabbitmq-or-zeromq-or/5350026#5350026 - also the Confluent Platform provides a Rest Proxy to produce messages into kafka through HTTP – fhussonnois Jul 04 '16 at 16:39
  • Thanks, I will look at the question. Although I don't see how a REST proxy will help me, I intend to use Kafka's java API directly for the high performance TCP protocol and less middlemen. – Tolstoyevsky Jul 04 '16 at 17:24

1 Answers1

0

If anyone comes across this, we went with Kafka in the end. Its performance is impressive and so far it's very stable on linux. No attempt yet to run it on windows in production deployments.

UPDATE 12/3/2017: Works fine and very stable on Linux, but on Windows this is not usable in production. Old data never gets deleted due to leaky file handles, the relevant Jira is being ignored since 2013: https://issues.apache.org/jira/browse/KAFKA-1194

Tolstoyevsky
  • 452
  • 5
  • 13