0

I'm using syslog to log data to a file - the data is pretty intensive, the order of thousands of rows every few seconds. What I observe is that trace amounts of logs are being missed - less than 0.1 % most of the times - but they're still missing. I have no explanation for why this occurs.

It doesn't seem to correlate directly to the amount of data being written because increasing the amount of data being written did not increase the rate of missed logs.

I'm wondering of ways to debug this - how could we understand or confirm if it is indeed syslog which is dropping data and if so why?

egorulz
  • 1,455
  • 2
  • 17
  • 28
  • Just a guess since there's been no answer in two days: syslog uses UDP (try the freebsd equivalent of "strace logger somemessage" - ktrace?), which means that messages may be silently lost - see http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/unix-daemon-design-mistakes-to-avoid.html, and if it turns out to be the problem, using multilog from the daemontools (or runit, s6, ...) will solve it. The downside is that you'll have to adapt whatever scripts you're using to check your logs. – loreb Apr 19 '14 at 12:26
  • @loreb For local logging, the domain PF_LOCAL is used, not PF_INET, see `socket(2)`. UDP is only used for remote logging. If you start FreeBSD's `syslogd` with the `-ss` option (which is the safest option for only doing local logging) it doesn't even open a network socket. It does use datagram sockets as explained in my answer below. – Roland Smith Apr 22 '14 at 10:04
  • @RolandSmith yep, I guess in my mind SOCK_DGRAM==UDP, even if it's technically incorrect :) – loreb Apr 22 '14 at 13:11

1 Answers1

3

If you look at the source code for syslogd, you will see that the syslogd program only uses datagram sockets (type SOCK_DGRAM). These are by definition connectionsless but also not completely reliable in the sense that stream sockets are.

This is by design. Using stream sockets would mean that the syslog() call would have to wait for a confirmation that the message that it sent was received properly. So if syslogd was busy, every application that calls syslog() would block.

Syslogd was simply not designed with the volume of data that you are subjecting it to in mind. You could try enlarging the value of the sysctl variable kern.ipc.maxsockbuf, giving the logging socket a larger buffer.

If you want to make sure you capture everything, write to a file instead.

Roland Smith
  • 42,427
  • 3
  • 64
  • 94
  • Yes - for my particular requirements I guess I will have to create a custom logger which writes to a file. I was trying to understand what exactly is the limitations which prevents syslogd from processing the amounts of data I'm subjecting it to. – egorulz Apr 22 '14 at 10:28
  • Aren't UNIX Domain Sockets (SOCK_DGRAM or SOCK_STREAM) lossless? Doesn't the transmitter block if the buffers are full? – kjpires Oct 17 '19 at 13:42
  • @kjpires There are more causes of packet loss than full buffers. I'm not even sure if full buffers are often an issue on modern hardware. See [this answer](https://stackoverflow.com/a/10810040/1219295) for a good overview of the differences. – Roland Smith Oct 17 '19 at 19:06