2

Requirement: Log events like Page Views and form Submits. Each page has ~1 second SLA. The application can have 100's of concurrent users at a time.

Log events are stored into the Database.

Solution: My initial thought was to use an async logging approach where the control returns back to the application and the logging happens in a different thread (via Spring's Thread pool task executor).

However someone suggested using JMS would be a more robust approach. Is the added work(Setting-up queue(s), writing to the queue(s), reading from the queue(s)) required when using this approach worthwhile?

What are some of the best practices / things to look out for (in a production environment) when implementing something like this?

Oliver
  • 104
  • 1
  • 9
  • Altough we have an interesting alternative path of thought (using JMS), and interesting other solutions, I would like to note that this is a duplicate of http://stackoverflow.com/q/3109140/744133 and http://stackoverflow.com/q/17018420/744133 – YoYo Apr 28 '15 at 04:08

3 Answers3

1

Using JMS for logging is a complete mismatch. JMS is a Java Abstraction for a Middleware Tool like MQ Series. That is complete overkill, and will let you go through a setup and configuration hell. JMS also lets you place messages in a transactional context, so you already get quickly the idea that JMS might be not much better than Database writes as @rjsang suggested.

This is not that JMS is not a nice technolgy. It is a good technology where it is applied properly.

For Assynchronous logging, you better just depend on a Logging API that directly supports it like Log4j2. In your case, you might be looking to configure a AsyncAppender with a JDBCAppender. Log4j2 has many more appenders as additional options, including one for JMS. However, by at least using a Logging abstraction, you make that all configurable and will make it possible to change your mind at a later time.

In the future we might have something similar to Asynchronous CDI Events, which should work similar to JMS, but would be much more lightweight. Maybe you can get something similar to work by combining CDI Events with EJB Asynchronous Methods. As long as you don't use EJB's with a remote interface, it should also be pretty lightweight.

YoYo
  • 9,157
  • 8
  • 57
  • 74
  • Async CDI Events are within JSR365 and that is currently before EDR1. Async Events will take a while until they are available. – mp911de Apr 28 '15 at 03:50
  • @mp911de Oy kidding me ... you are right. I was so sure it was there that I googled it up to find the [reference](https://blogs.oracle.com/theaquarium/entry/asynchronous_cdi_events) ... but this is documented as a wish list ... didn't notice... thanks. – YoYo Apr 28 '15 at 03:55
  • @mp911de how about that last addition of combining CDI events with EJB/Asynchronous? – YoYo Apr 28 '15 at 04:01
  • Works, but only in JEE environments. You trigger a regular CDI event and the observer method is an async EJB method. – mp911de Apr 28 '15 at 05:10
1

Both approaches are valid, but one is vulnerable if you app unexpectedly stops. In your first scenario, events yet to be written to the database will be lost. Using a persistent JMS queue will mean that those events will be read from the queue and persisted to the database upon restart.

Of course, if your DB writes are so much slower than placing a message of similar size on to a JMS queue, you may be solving the wrong problem?

rjsang
  • 1,757
  • 3
  • 16
  • 26
0

You could give it a try using fully async and external tooling if you want to. If you have to stick to your SLA at any price and resilience is important for you, you could try using either logstash or process your logs offline. With doing so, you decouple your application from the database and you are no longer depending on the database performance. If the database is slow and you're using async loggers, queues might run full.

With logstash using GELF the whole log processing is handled within a different (or even remote) JVM. Offline processing (e.g. you write CSV logs) allows you to load the log data afterwards into the database.

mp911de
  • 17,546
  • 2
  • 55
  • 95
  • Although I don't know anything about logstash/GELF, I particularly like that solution for high availability and high volume type of environment. Still there will be an overhead communicating with the remote log handler I would assume? Also in that case I might prefer integration with Log4j2, maybe again combining with a Asynchronous Appender – YoYo Apr 28 '15 at 04:05
  • So as per http://stackoverflow.com/a/26858968/744133, GELF seems to be - or contain - a Log4j2 appender. – YoYo Apr 28 '15 at 05:41
  • 1
    Well, logstash has a log4j input, that takes a serialized log4j message, but this input uses TCP and TCP can block an entire application. GELF is a protocol spec, not a tool which specifies compressed JSON messages over UDP. I wrote a suite of GELF appenders for log4j, log4j2, JUL, JBossAS7 and logback. UDP won't block your application on congestions, rather messages would be dropped. – mp911de Apr 28 '15 at 07:08
  • @mp911de The SocketAppender in Log4J2 uses tcp and since UDP is not reliable can I use SocketAppender along with AsyncAppender? – dilanSachi Sep 24 '19 at 08:42