3

I'm working with a "firm" real-time system (the result will be unuseful if served after the deadline) which requires maximum throughput and robustness to load.

More details about our system, we are processing telecommunication messages. The messages will go through several step with different concerns. However the messages order is very important. The steps are like these:

  1. Query something from Redis in-memory database
  2. Query something from another relational in-memory database
  3. Some business logic computional processing
  4. Update data into Oracle RDBMS
  5. Send messages to other server through Sockets

We are currently setup a system with a distributor and 8 processing units (which runs through the messages sequentially) on only 1 server. We decided to maximize the performance per unit first, before doing horizontal scale.

I've tried several approaches (i'm only passing the messages through the first two steps and some data collecting logic):

  1. Proactor pattern: by implementing the event-loop and high-level asynchronous API with callbacks by myself (using Java API). However, the performance is not as expected, the messages are not in order.
  2. SEDA: the overall performance is good when i'm using a thread-pool for each stage. However, to preserve the message order, I tried using only a thread per stage, the performance is even worse than the Proactor pattern.
  3. LMAX Disruptor: the "default" batching characteristic brings both the latency and throughput down because I find it a bit hard to apply batching to our current business logic. When I avoid using batching, the performance is ok now, however, it's only about 1.5x faster than the sequential approach (even i'm using a thread for each handler, so it's 3 thread)
  4. A "naive" multithreading approach: using a BlockingQueue + some worker threads, the performance is better than I expected. 1.75-2X faster than the LMAX Disruptor approach. I find it a little bit counter-logic here...

So, have i done anything wrong with my approaches above? Does the event-driven architecture really fit our need for a high throughput, firm real-time system?

P/S: It's my first time asking a question on Stackoverflow, so please give me advices if there're any problems with my questions :D. Thanks :D.

jxh
  • 69,070
  • 8
  • 110
  • 193
Nova
  • 186
  • 2
  • 5
  • 1
    I'd recommend looking at vert.x: http://vertx.io/ – duffymo Jun 16 '14 at 17:04
  • Thank you, I took a look at vertx before and I think it doesn't suit our needs. We need to find a way to handle messages concurrently to get maximum throughput, so I don't think asynchronous and non-blocking communications between the server and clients are crucial :D. – Nova Jun 16 '14 at 17:11
  • I think it could be worth a look on the server side as well. I wouldn't dismiss it out of hand before you've tried it out. – duffymo Jun 16 '14 at 17:15
  • 2
    You are performing operations on a number of databases over TCP, none of them are going to be particularly fast compared with say Disruptor or BlockingQueue. I would focus on how to get the most performance out of your data bases. BTW none of the database you mention are designed for Real Time, nor is Java, so I would give up on that notion. – Peter Lawrey Jun 16 '14 at 17:18
  • "Does the event-driven architecture really fit our need for a high throughput, firm real-time system" This bit does, the database not so much, but I assume you can't work around them. – Peter Lawrey Jun 16 '14 at 17:19
  • Our in-mem DB act as mostly read-only caches. No direct writes to these DB (they are synced with the Oracle DB after some hours). Because there are many other offline modules which shared the data with the system so we still need to keep the Oracle DB. However, can you explain why a "naive" multithread approach gives better performance than the "great" LMAX Disruptor framework if the bottleneck is about the DB, not the architecture? :( – Nova Jun 16 '14 at 17:25
  • Could you share some tips to achieve maximum performance with our current DBs? – Nova Jun 16 '14 at 17:30
  • When is the message order important? Steps 1-3 look order-independent, step 4 maybe (if message data is related) and step 5 is ambiguous when several sockets are used (if order needs to be guaranteed message 2 can only be send after message 1 is delivered). An a side-note: the "naive" multithreading approach has worked well for me but is hard to maintain if additional cases need to be supported like 'high priority messages' or 'message X must be processed after message Y and Z'. – vanOekel Jun 16 '14 at 20:11
  • It is exactly the case you mentioned "message X must be processed after message Y and Z". Such as for a "voice call", there's must be an opening message and a closing message, they need to be processed in order. Steps 1-2 are not order-dependent but the other ones are :(. – Nova Jun 17 '14 at 01:15
  • "message order is important" - but where are the messages? Do you consider results of database queries as messages? – Alexei Kaigorodov Jun 17 '14 at 04:03
  • No, the messages are the input for our system. As i said in the question :D "we are processing telecommunication messages". – Nova Jun 17 '14 at 04:06

0 Answers0