2

At present, the trading system of our production environment is using Kafka. Because Kafka latency is too high, we hope to replace Kafka with Aeron. How can I use Aeron correctly?

mazaneicha
  • 8,794
  • 4
  • 33
  • 52
user13454562
  • 23
  • 3
  • 1
    Latency is too high? Please show your configs, certainly something is wrong... I assume you use Netflix or Uber? They use Kafka on a global scale. Go watch their conference talks. Also, Kafka is TCP, not UDP/IPC like Aeron – OneCricketeer May 02 '20 at 19:56

1 Answers1

5

Aeron isn't an out of the box replacement for Kafka although it does provide primitives that would allow you to replicate much of the functionality.

Kafka latencies are in the order of milliseconds whereas Aeron latencies are typically measured in microseconds.

What exactly you would need to build in Aeron very much depends on your use case.

One of the primary uses of Kafka is as a persistent queue.

To build a simple persistent queue for a single publisher use case. You would need:

Publisher

  • ArchivingMediaDriver - this component runs and Aeron MediaDriver which handles send/receiving messages over the network and and Archive which allows you to record and replay streams.
  • A Publication to send messages to be recorded by the Archive. See AeronArchive.addRecordedPublication.

Subsciber(s)

  • MediaDriver - this component handles send/receiving messages over the network.
  • A Susbcription that replays data from a specific position in the recorded stream of messages. See AeronArchive.replay.

There are examples of this in the aeron-samples.

Latency could be reduced further by having the publisher send messages over multicast/MDC and having the subscriber use ReplayMerge to seamlessly transition from the recorded stream to the live stream.

Worth noting that real-logic do provide commercial support.

James
  • 388
  • 3
  • 7
  • 1
    Thanks @james for the answer, great。 Because I use a `stream` to store all Kafka topic data, how to get the topic of each record without deserialization? thx – user13454562 May 04 '20 at 04:24
  • The trick here is to store the topic at a well known byte offset in your message. Then you can read it out without deserialising the rest of your message. – James May 04 '20 at 08:07