-1

I need to log information with the following requirements:

  • it is for writing data to files, that will latter be processed
  • output lines have a predefined structure
  • has to be really fast
  • it should buffer the data and write it to files in background.
  • it should support rotating the file and also should allow manual file rotation whenever required

Do you happen to know any good library that supports this? Or would you recommend writing it on my own? I wouldn't mind implementing one on my own, but I would like to know what my options are before taking a decision.

Razvi
  • 2,808
  • 5
  • 31
  • 39
  • What have you been using so far? – Mukus Dec 07 '12 at 09:50
  • SLF-4j and log4j for logging errors and debug messages. But this part will be called very often (will never be turned off) and it needs to do the logging and return to the caller as fast as possible. – Razvi Dec 07 '12 at 10:02
  • Logging should be done asynchronously. It should not have priority over the main thread. In fact it should run when there is no active calculations/threads running. Therefore you might expect to see delayed output to the log file. If you're trying to free up CPU time from the main thread and write to log file, thereby getting faster output, you're doing it wrong. Your application slows down. If you use timestamps when the log is written you can always look at the log to see what happened at that point. Log being written after some delay does not mean the code execution did not reach there. – Mukus Dec 07 '12 at 10:10
  • This question can be useful to people thinking in the exact same way as the author. I am upvoting. – Mukus Dec 07 '12 at 10:15
  • @TejaswiRana: something to point out: asynchronous logging does imply a slightly higher overhead, so it will only be faster if not all CPU cores are saturated. I would suggest that the OP profiles their application if throughput is an issue, before trying any solution. – thkala Dec 07 '12 at 10:28
  • By asynchronous IO, I meant buffering logs. And having a separate thread write the data to the file from time to time. In what I meant, the logs will have the same order as when they were written (in my case, I don't care about the log lines order). – Razvi Dec 07 '12 at 11:56

2 Answers2

3

Most common Java logging libraries support most of what you are asking for, except perhaps for the asynchronous writing part. You might want to have a look at the following widely-used libraries:

  • Log4J - Log4J is extremely configurable, but it does have some issues that are supposed to be resolved in the new 2.x version series, which is about to be released.

  • Logback - This was designed as a successor to Log4J 1.2 - Log4J 2.x will include several improvements made by Logback.

  • SLF4J - This library can act as a proxy to a number of underlying logging libraries, allowing the logging framework to be switched as necessary. This might actually be the best choice to avoid binding yourself to a specific logging library, although it is probably not as efficient as using the underlying API directly.

As for the asynchronous writing requirement, if necessary it should be relatively easy to push all logging to a separate thread of your own.

EDIT:

Apparently, Log4J does have support for asynchronous logging.

Community
  • 1
  • 1
thkala
  • 84,049
  • 23
  • 157
  • 201
  • I need the asynchronous part so that it returns to the caller as fast as possible, without waiting for the IO operation. I'll look into the log4j asynchronous appender. Thanks :) – Razvi Dec 07 '12 at 10:05
0

I don't have any direct experience, but it seems log4j is a very popular choice. A quick check shows that it supports file rotation and is considered fast (at least, significantly faster than Java's built in facilities), which seem to me the two strictest of your requirements.

Links:

Main website

Some SO discussion

Community
  • 1
  • 1
David Duncan
  • 1,225
  • 8
  • 14