How to send structured data through Syslog?

Question

I have structured data, key value pairs, that should be logged through syslog. In the end we want to see statistics about these metrics. How should we encode and then decode it on the reciever side?

One option that we parse the message part of the log in syslog and based on that parsing we insert it into a relational database table.

The second idea we had is to send the data in JSON and on the reciever side we treat the relational database table as a job queue, records must be parsed before inserted to a separate table.

In addition key value pairs may alter based on what we want to log.

I'd also like to know more about this. Windows has structured logging with ETW. What does Linux have? — Matt Johnson-Pint, Apr 02 '15 at 21:16
@MaciejLos - Not quite what this is about. That's just logging syslog data into mysql. This is talking about logging *structured data*. MySQL could be part of the solution, but it's not necessarily what this is about. — Matt Johnson-Pint, Apr 02 '15 at 23:32
@MattJohnson, i wasn't sure what this question is about, that's why i posted comment. Even now, i'm not sure what exactly OP wants to achieve and what he tried. — Maciej Los, Apr 03 '15 at 05:34

score 3 · Answer 1 · edited Oct 07 '21 at 11:04

The RFC 5424 for the syslog protocol defines a STRUCTURED-DATA field:

The syslog message has the following ABNF [RFC5234] definition:

    SYSLOG-MSG      = HEADER SP STRUCTURED-DATA [SP MSG]
    (...)
    STRUCTURED-DATA = NILVALUE / 1*SD-ELEMENT
    SD-ELEMENT      = "[" SD-ID *(SP SD-PARAM) "]"
    SD-PARAM        = PARAM-NAME "=" %d34 PARAM-VALUE %d34
    SD-ID           = SD-NAME
    PARAM-NAME      = SD-NAME
    PARAM-VALUE     = UTF-8-STRING ; characters '"', '\' and
                                     ; ']' MUST be escaped.
    SD-NAME         = 1*32PRINTUSASCII
                      ; except '=', SP, ']', %d34 (")
    (...)

(example here)

However, this RFC does not seem to be widely supported. You may need to devise your own protocol on top of the old RFC 3164. A JSON-encoded message part sounds like a very reasonable option.

If you have complete control over the whole chain, the option of logging directly into the target database should be assessed.

How to send structured data through Syslog?

1 Answers1