4

I'm trying to choose an appropriate architecture for a middle-frequency trading system I'm working on. Currently, I receive messages from Web Socket or Rest and process them right there. Sometimes it includes IO operations (i. e. additional rest requests), so it works very slowly and all other messages, I suppose, are getting buffered in Web Socket client's implementation. This naive approach doesn't look very scalable.

I've been reading into mature architectures for handling trading messages and currently, my choice has been narrowed down to Disruptor and Reactive programming. I would like to ask for your advice which one is a better choice. Specifically, I'm concerned about 2 scenarios:

  1. Logical dependency between message handlers. When I'm connected to a specific exchange, I need to receive balances and open orders before I can be able to process trading messages and make orders based on them. It seems to me Reactive is a better approach to handle this kind of situations that require flow control. Is it a problem for Disruptor?
  2. Long-running message handlers. Message handlers should be as fast as possible (not to block the following messages) but what is the right approach in case I need to make, let's say, a rest request to create an order as part of message handler?
Didix
  • 567
  • 1
  • 7
  • 26
SiberianGuy
  • 24,674
  • 56
  • 152
  • 266
  • I would highly recommend doing a simulation for both prototypes. Its a high risk to drive the architecture of your app purely from a stackoverflow answer aka: "Definitely Choose X maybe". – Cristian E. Mar 14 '18 at 10:15

1 Answers1

3

I think you should take a look at Apache's Kafka. Its design is very similar to Disruptor, and you can split your messages among several topics, with different configurations. Depending on whether you prefer low-latency or high-throughput. It also provides support for compression messages on the fly to reduce bandwidth usage, or will let you split messages within one topic among several partitions, each of which can potentially be hosted on a different machine. This is useful for load balancing. Of course, replication is supported, so if one of your machine crashes, the system will keep working reasonably.

To read and process the Kafka messages, you can then use multiple patterns. The default (at least when using the C++ librdkafka client) is to let you do polling, but you can easily setup a callback-based system on top of that. You can also have a reactive system instead, it maps quite naturally to the concept of topics/partitions that Kafka has.

In summary:

To handle your (1) scenario, you would split messages on different topics based on their urgency, and have higher-priority threads dealing with the more important messages (and also setup kafka to reduce latency on those topic)

To handle your (2) scenario, librdkafka (C++) provides a way to temporarily pause a topic, while your application catches up.

manuBriot
  • 2,755
  • 13
  • 21