Systemd service and structured logging

Question

I have a systemd service which is of type simple.

I want my service to log key-value pairs.

Up to now my simple service just prints it log to stdout in a custom key-value syntax.

I would like to avoid my custom key-value syntax it use an official way to do structured logging.

Is there a way to use structured logs with systemd?

For example my service writes this to stdout:

{"key1": "value1", "key2": 1234}

It would be great of systemd could read the string as json.

Please give an example of the actual application logging and the expected output. — LMC, May 01 '19 at 00:42

LMC · Answer 1 · 2019-05-02T16:43:38.113

You can log structured data to systemd and then view it as is or in json format

Let's say we log key-value pairs in json format

systemd-cat -t "struct_logs" echo '{"key1": "value1", "key2": 1234}'

Then we read them as is

sudo journalctl -t 'struct_logs' --lines 1 --no-pager

Apr 30 21:46:14 linux-ar struct_logs[17455]: {"key1": "value1", "key2": 1234}

Or in json format

sudo journalctl -t 'struct_logs' --lines 1 --no-pager -o json-pretty
{
        "__CURSOR" : "s=b6d07ffffffffff2787cea140a0db88a5;i=db8;b=c9b2132ffffffffe5807625fe;m=9fa5b3f81;t=58fffffff87eab;x=6402818ea7e32a5b",
        "__REALTIME_TIMESTAMP" : "1556671574343339",
        "__MONOTONIC_TIMESTAMP" : "42854989697",
        "_BOOT_ID" : "fffffffffffff24ec7a237b5e5807625fe",
        "PRIORITY" : "6",
        "_MACHINE_ID" : "fffffffffff66c86aaaaaaaaaa",
        "_HOSTNAME" : "linux-ar",
        "_TRANSPORT" : "stdout",
        "_UID" : "1000",
        "_GID" : "100",
        "SYSLOG_IDENTIFIER" : "struct_logs",
        "MESSAGE" : "{\"key1\": \"value1\", \"key2\": 1234}",
        "_PID" : "17455"
}

Going further you can recover your exact log messages with the help of jq

sudo journalctl -t 'struct_logs' --lines 1 --no-pager -o json-pretty | jq -r '.MESSAGE'
{"key1": "value1", "key2": 1234}

It could be possible also to log custom keys provided that the user has the proper rights to do that. A service should be able to do that

sudo logger --journald <<end
MESSAGE_ID=67feb6ffbaf24c5cbec13c008dd72309
MESSAGE=The dogs bark, but the caravan goes on.
SYSLOG_IDENTIFIER=struct_logs
KEY=bark
VALUE=goes on
end

Hope this helps.

I use service type "simple". Is there a way to let the stdout of the process get to structured journalctl? — guettli, May 02 '19 at 11:10
If I understood your answer correctly, then the json is just a string for systemd/journalctl. — guettli, May 02 '19 at 11:14

score 3 · Answer 2 · answered May 03 '19 at 08:56

Have a look at Structured logging to journald from within a docker container; your application should use the sd_journal_send method if you want to specify custom fields.

If your application is unable to do that you can also output json - which will end up in the MESSAGE field you can then parse yourself. For example - I wrote SystemdJournal2Gelf to send the journal entries to Graylog and it also supports json output in the MESSAGE, sending it to Graylog as separate fields for filtering

If you don't want to modify the application you could use a separate script - something like this:

[Service]
ExecStart=/bin/sh -c '/bin/my-app | while read l; do echo "$l" | logger --journald; done'

You could even convert json to key/value pairs if you want, or use StandardOutput to send your standard-output to a socket where you have a separate service listening.

I have been thinking about such a wrapper, too. But I am unsure if signal handling for restarting still works if you wrap the real service. — guettli, May 06 '19 at 06:29

score 1 · Answer 3 · answered Feb 24 '20 at 16:53

Well, systemd does not make it easy to do structured logging from text stream. It want your to take a dependencies on its library and use sd_journal_send (see for example http://0pointer.de/blog/projects/journal-submit.html ). I have not found any "text form" standard that allow systemd to understand that your line should not be smushed together in the MESSAGE fields but rather should be interpreted as a key-value pair. I guess a solution would be to write a helper process that parse your stdout and then call the systemd logger. You would then pipe your process stdout process into

However, I did found this answer that pipe directly into the journald socket. With clever use of jq, you could write something that exploit that. But that seems hacky: I do not know if systemd really intend to keep any compatibility on that socket.

Systemd service and structured logging

3 Answers3