2

Suppose we have systems which are speed critical (like for example statistics/analytic's,socket programming etc), how do we design the traces and logs.

To be more specific, logs and traces generally reduce the performance (even if we have a switch off mechanism or verbose extension mechanism). In such scenarios, is there any reference guideline on how to 'put' logs/traces so that when the issue occurs (especially at production site) developer/post production team are able to pin-point the actual issue.

PS: I come from background where such applications are developed in C/C++ (run on Linux)

kumar_m_kiran
  • 3,982
  • 4
  • 47
  • 72
  • i guess async logging if you can afford to lose some entries; or something like disruptor from lmax, if you absolutely need consistency – driushkin Mar 19 '13 at 12:15
  • In case you are meaning logging with another thread, I am aware of that process. However, at times, even these will act as a drag. Wanted to know if there are any other "better" mechanisms to handle logging/tracing. – kumar_m_kiran Mar 19 '13 at 12:18
  • Are you looking for a system that adds as little overhead as possible in the case when there is nothing to log? Or are you looking for a way to minimise the overhead of actually writing to the log? If the latter, do the messages tend to cluster together with long gaps in between, or is there a more serious total throughput issue? – Ian Goldby Mar 19 '13 at 12:24
  • Let me put it the other-way round. I already have system (without inter-module trace, logs) supporting X speed in production environment. Now I am trying to find a way to add logs (all over the system) and traces and trying to minimize the degradation of X. – kumar_m_kiran Mar 19 '13 at 12:30

1 Answers1

1

You can accumulate logs inside a buffer which you can describe and implement using Google Protocol Buffers. You can have a different thread periodically(every 5minutes) empty this buffer to disk or sending it through a UNIX domain socket (or other Linux IPC mechanisms) to a daemon that listens and writes them to a persistent DB or simply writes them to disk.

If you don't want to hit the disk on the machine that produces logs, you can send them to a different machine through a regular socket and write them to disk on that machine.

If you are aggregating logs from multiple machines, consider using 0MQ or CrossRoads as message queues to pass your logs through the network to a machine where they are stored persistently. You can find some information about using 0MQ in conjuction with Google Protocol Buffers here.

Community
  • 1
  • 1
wsdookadr
  • 2,584
  • 1
  • 21
  • 44