281

I am trying to redirect output of a systemd service to a file but it doesn't seem to work: 

[Unit]
Description=customprocess
After=network.target

[Service]
Type=forking
ExecStart=/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server
StandardOutput=/var/log1.log
StandardError=/var/log2.log
Restart=always

[Install]
WantedBy=multi-user.target

Please correct my approach.

kubanczyk
  • 5,184
  • 1
  • 41
  • 52
meallhour
  • 13,921
  • 21
  • 60
  • 117
  • I’m voting to close this question because From the tag: systemd questions should be for *programming questions* using systemd or its libraries. Questions about *configuring the daemon* (including writing unit files) are better directed to Unix & Linux: https://unix.stackexchange.com. Please delete this. – Rob Apr 03 '23 at 07:41
  • 2
    I would agree with Rob if the question was new, but it's been around for 7 years and has several highly upvoted answers; it's too late to delete it now imo. – Guntram Blohm Apr 03 '23 at 09:18

10 Answers10

357

I think there's a more elegant way to solve the problem: send the stdout/stderr to syslog with an identifier and instruct your syslog manager to split its output by program name.

Use the following properties in your systemd service unit file:

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=<your program identifier> # without any quote

Then, assuming your distribution is using rsyslog to manage syslogs, create a file in /etc/rsyslog.d/<new_file>.conf with the following content:

if $programname == '<your program identifier>' then /path/to/log/file.log
& stop

Now make the log file writable by syslog:

# ls -alth /var/log/syslog 
-rw-r----- 1 syslog adm 439K Mar  5 19:35 /var/log/syslog
# chown syslog:adm /path/to/log/file.log

Restart rsyslog (sudo systemctl restart rsyslog) and enjoy! Your program stdout/stderr will still be available through journalctl (sudo journalctl -u <your program identifier>) but they will also be available in your file of choice.

Source via archive.org

kubanczyk
  • 5,184
  • 1
  • 41
  • 52
Valerio Versace
  • 3,708
  • 1
  • 11
  • 10
  • 12
    Not working for me on Ubuntu 16.04. `journalctl -u` still works but nothing gets sent to the specified file. – Duncan Calvert Oct 05 '17 at 18:53
  • 2
    This works great on Debian stretch, however it complains that `~` is deprecated and `stop` should be used instead. Also note that the second line can be shortened to `& stop` if the two come after each other. – jlh Nov 06 '17 at 09:29
  • 85
    With systemd 236 or newer you can also write directly to a file using StandardOutput=file:/some/path https://github.com/systemd/systemd/pull/7198 – leezu Dec 18 '17 at 07:02
  • @leezu FYI i [wrote an answer that is similar to your comment](https://stackoverflow.com/a/48052152/52074). I wrote my answer before I saw your comment. – Trevor Boyd Smith Jan 01 '18 at 19:03
  • error during parsing file /etc/rsyslog.d/project.conf, on or before line 1: syntax error on token '==' [v8.32.0 try http://www.rsyslog.com/e/2207 ] – Király István Apr 01 '18 at 02:15
  • Source is no longer available. – Max Candocia Jun 06 '18 at 20:32
  • 11
    I got this working by changing `/etc/rsyslog.d/.conf` contents to: `:programname, isequal, "" /var/log/somelog.log` Here's documentation on the rsyslog filters: https://www.rsyslog.com/doc/v8-stable/configuration/filters.html And here's docs on the properties like `programname`: https://www.rsyslog.com/doc/master/configuration/properties.html – mbil Jul 17 '18 at 16:27
  • 5
    I had problem using this configuration until I found that `rsyslog` has its own user `syslog` and it has to have write access to the logs location. So use `chown` accordingly. Hope this helps somebody. – Imaskar Dec 13 '18 at 09:31
  • on ubuntu 16.04 this logs to /path/to/log/file.log and /var/log/syslog – Wakan Tanka Feb 26 '19 at 09:21
  • 1
    Is it possible to separate stdout from stderr using this method? – MacroMan Apr 12 '19 at 09:24
  • @Imaskar Thanks! This was the clincher for me on Xenial. – talonx Jul 26 '19 at 15:53
  • Ubuntu uses ReinerScript in its default configuration files. So changing contents of /etc/rsyslog.d/.conf to `if $programname == 'program_identifier' then { action (type="omfile" file="/var/log/my_program.log") }` worked out of the box without changing permissions of /var/log/my_program.log – user2910552 Sep 02 '19 at 14:23
  • 1
    Thanks, Test it on Centos 7.5 - Logs are appeared both on `journalctl -u SERVICE` and `/var/log/FILE.log` – Mohammed Almusaddar Oct 08 '19 at 11:28
  • The following rsyslog config worked for me: `:programname, isequal, "xxx" /var/log/xxx/xxx.log` – Zolbayar Apr 02 '20 at 02:49
  • 1
    the answer may provide a useful alternative solution, but it doesn't describe why the OP's approach doesn't work. I personally would like to know, as I've tried his approach myself, and it didn't work for me either. But why? – Greg Woods Jun 29 '20 at 22:05
  • 2
    @GregWoods at the time of writing, the StandardOutput setting simply didn't take a file descriptor. Now there's a way to do it by providing a "file:" value. Just read the manual: https://man7.org/linux/man-pages/man5/systemd.exec.5.html – Valerio Versace Jun 30 '20 at 23:08
  • syslog and process name are defaults in systyemd for `stdout` and `stderr`, so you do not need to clutter your config at all. – wick Dec 07 '20 at 22:28
  • 1
    Step 2 talks about /path/to/log/file.log then step 3 talks about chown that file - but it doesnt exist as no part of the steps create it – Al Grant Dec 10 '20 at 08:04
  • more complete answer: https://unix.stackexchange.com/a/479669 – Ali Sadeghi Dec 09 '21 at 11:09
  • Note that `StandardOutput`, `StandardError`, and `SyslogIdentifier` go in the `[Service]` section of the .service file. – Alex Henrie Aug 19 '22 at 17:27
  • N.b. this answer suggests a different strategy instead of directly answering the question, which other people have done. The submitter here may be happier with this suggestion, but there are plenty of situations where it is more appropriate to directly write to specified files without more dependencies, and people searching for that won't get helped by this answer. – Blaine Sep 19 '22 at 23:48
127

I would suggest adding stdout and stderr file in systemd service file itself.

Referring : https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=

As you have configured it should not like:

StandardOutput=/home/user/log1.log
StandardError=/home/user/log2.log

It should be:

StandardOutput=file:/home/user/log1.log
StandardError=file:/home/user/log2.log

This works when you don't want to restart the service again and again.

This will create a new file and does not append to the existing file.

Use Instead:

StandardOutput=append:/home/user/log1.log
StandardError=append:/home/user/log2.log

NOTE: Make sure you create the directory already. I guess it does not support to create a directory. This feature was introduced in systemd 240, but backported to earlier versions on e.g. RHEL8.

NodeUnderflow
  • 33
  • 3
Rajat jain
  • 1,715
  • 3
  • 12
  • 21
  • 23
    I think, I made it more direct\easy to understand. – Rajat jain Jan 29 '19 at 08:04
  • 1
    For me the `file:` route works on the first load of the service, but on subsequent restarts it no longer writes to the file. I tried `append:` from the docs and that didn't work at all. – rb- Jan 29 '19 at 16:53
  • 5
    Note that the docs make clear that `file:` writes to the start of the file every time, and does not truncate... further, `append:` seems to be a new addition (i.e. not present in the `man systemd.exec` page on Ubuntu 18.04). – cole Sep 26 '19 at 20:11
  • 12
    `append:` has been introduced in systemd version 240. To work around `file:` placing new log output at the beginning of the target file, something like this may help: `ExecStartPre=/bin/bash -c 'mv /var/log/my/logs.log /var/log/my/$$(date +%%T)-logs.log'`. Keeps a clean log and emulates log rotation effect in a way – SVUser Aug 19 '20 at 17:40
  • 3
    For anybody curious, Ubuntu 20.04 currently has systemd version 245, while Ubuntu 18.04 has 237. So unfortunately you have to have 20.04 to use `append:`. – nonrectangular Feb 26 '22 at 23:32
  • `truncate:` was added in `version 248`. It will truncate before opening. – rodvlopes Jul 20 '22 at 13:26
  • `append:` gives error `Failed to parse output specifier, ignoring:\home\path\log.log` – Siddharth Das Aug 18 '23 at 19:12
  • 1
    @SiddharthDas Make sure you have the correct path defined. Also to use `append:` you have to have `Ubuntu 20.04` – Rajat jain Aug 29 '23 at 11:54
126

If you have a newer distro with a newer systemd (systemd version 236 or newer), you can set the values of StandardOutput or StandardError to file:YOUR_ABSPATH_FILENAME.

Long story:

In newer versions of systemd there is a relatively new option (the github request is from 2016 ish and the enhancement is merged/closed 2017 ish) where you can set the values of StandardOutput or StandardError to file:YOUR_ABSPATH_FILENAME. The file:path option is documented in the most recent systemd.exec man page.

This new feature is relatively new and so is not available for older distros like centos-7 (or any centos before that).

Trevor Boyd Smith
  • 18,164
  • 32
  • 127
  • 177
  • 12
    Not working in ubuntu 1604 in 2018-03-20. The systemd version in ubuntu 1604 is only 229. – bronze man Mar 20 '18 at 02:00
  • Thanks, you said very clear. I just can not believe the systemd in ubuntu 1604 can not redirect output to a file just by config.I have to use the sh way to solve this problem. – bronze man Mar 22 '18 at 02:15
  • @bronzeman the feature request wasn't closed until 2017, while Ubuntu 16.04 came out in 2016. In a given major release of Ubuntu (e.g. 16.04, 16.10, 17.04, etc.), Ubuntu maintains ABI compatibility in its core system packages. So they won't upgrade systemd (or the Linux kernel, or glibc, or anything) unless it maintains the same ABI as when the Ubuntu version was first released. – villapx Nov 26 '18 at 21:42
  • 3
    FWIW: I've searched a bit but this feature doesn't appear to have provisions for log rotation, such as function to reopen the log file, with one having to use the likes of `copytruncate` in `logrotate`. – antak Jan 21 '19 at 05:17
  • 2
    The problem is, that file is always created as root:root, regardless of the unit's user and group... – bviktor Jan 13 '21 at 19:37
  • i'm using Ubuntu 18.04 and its `systemd` version is 237 but is does not work. The file is created but it is empty! – SuB Nov 07 '21 at 12:46
62

You possibly get this error:

Failed to parse output specifier, ignoring: /var/log1.log

From the systemd.exec(5) man page:

StandardOutput=

Controls where file descriptor 1 (STDOUT) of the executed processes is connected to. Takes one of inherit, null, tty, journal, syslog, kmsg, journal+console, syslog+console, kmsg+console or socket.

The systemd.exec(5) man page explains other options related to logging. See also the systemd.service(5) and systemd.unit(5) man pages.

Or maybe you can try things like this (all on one line):

ExecStart=/bin/sh -c '/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server 2>&1 > /var/log.log'
Community
  • 1
  • 1
Shuangistan
  • 1,053
  • 8
  • 20
  • 7
    Among the options, logging into the systemd journal is recommend. You view just the logs for your process in the journal by using `journalctl -u your-unit-name`. – Mark Stosberg Jun 21 '16 at 15:35
  • 6
    To specify a file, there's another cleaner option, as indicated by the documentation: `The fd option connects the output stream to a single file descriptor provided by a socket unit. A custom named file descriptor can be specified as part of this option, after a ":" (e.g. "fd:foobar").` – orion Jan 23 '17 at 10:25
  • 5
    Awsome answer, it's solved my problem. I just want to extend, because currently if the service restart overwrite the old logs, have to replace this part: `2>&1 > /var/log.log` to this: `2>&1 >> /var/log.log`. Thank you – PumpkinSeed Apr 10 '17 at 09:59
  • 10
    Frankly, calling shell with a command string in ExecStart sounds like the Really Wrong Way to do it. – David Tonhofer Jul 13 '17 at 10:51
  • The '/bin/sh -c' looks wrong but it works. It also works with 'Restart=always' . I also suggest you add 'StandardOutput=null' to the config to avoid some rubbish log go into /var/syslog – bronze man Mar 20 '18 at 02:11
  • The '/bin/sh -c' doesn't work for me, I think due to strange quoting rules in systemd. See e.g. https://stackoverflow.com/questions/23935637/systemd-string-escaping . Does anyone have this working? – Rich Jan 31 '19 at 12:00
  • 2
    "/bin/sh" is a great workaround, but you MUST use "exec", otherwise the service will not restart properly as SIGTERM will not get passed to the child process. See http://veithen.io/2014/11/16/sigterm-propagation.html – Rich Feb 14 '19 at 15:33
  • 1
    '/bin/sh -c' has one undesired consequence with not working "systemctl stop service" command. It just kills sh process and not touches the service – Roman Shishkin Feb 07 '20 at 10:07
  • 1
    `journalctl -u` is ridiculous and has numerous shortcomings. It needs redesigned with actual workflows and simple commands in mind. Every QA team member hates it (same with systemctrl not giving output). File output wasn't thought of from the start, which is just criminal; and, a lot of enterprise users still don't have it. This was a mistake and it hasn't been improved. This is one case where companies took over and left users struggling. – Derek White Jul 28 '20 at 23:04
  • To redirect stderr correctly, you must use ``>/var/log.log 2>&1``, not the other way round. – Marki Aug 27 '20 at 16:32
27

If for a some reason can't use rsyslog, this will do: ExecStart=/bin/bash -ce "exec /usr/local/bin/binary1 agent -config-dir /etc/sample.d/server >> /var/log/agent.log 2>&1"

Tigra
  • 2,611
  • 20
  • 22
  • What does the -e option of bash do? – Lamp May 07 '20 at 22:51
  • The only solution that worked in my case for not losing the log file at every service restart as append is not available in my current systemd version unfortunately. – pragmatic_programmer Sep 18 '20 at 18:12
  • @Lamp, I doubt you still need this but I was also curious about the -e option in bash. I found this: https://stackoverflow.com/a/41760745/5033594 – Allen Apr 28 '22 at 16:39
26

Short answer:

StandardOutput=file:/var/log1.log
StandardError=file:/var/log2.log

If you don't want the files to be cleared every time the service is run, use append instead:

StandardOutput=append:/var/log1.log
StandardError=append:/var/log2.log
Arnett Rufino
  • 444
  • 4
  • 9
  • 9
    Duplicate of [this answer](https://stackoverflow.com/a/54232531/485343), with less details – rustyx Jan 25 '20 at 13:16
  • 2
    Super helpful. I used this technique to restore logging to /var/log/tomcatX/catalina.out for Ubuntu 18.04 and 20.04. – Ted Cahall Aug 31 '20 at 01:16
  • Wrong about "file". Read the docs about what that does. It's "truncate" mode that actually does: _...cleared every time the service is run_ Also, there is no reason to specify StandardError as you do here, since default behavior is for the error stream to inherit the stream set up by StandardOutput. – Blaine Sep 19 '22 at 23:41
  • It does log trancating based on size automatically with this. Is there a way to change that? I don't seem to see anyone talking about that. – The Quantum Physicist Oct 07 '22 at 10:09
7

We are using Centos7, spring boot application with systemd. I was running java as below. and setting StandardOutput to file was not working for me. 

ExecStart=/bin/java -jar xxx.jar  -Xmx512-Xms32M

Below workaround solution working without setting StandardOutput. running java through sh as below. 


ExecStart=/bin/sh -c 'exec /bin/java -jar xxx.jar -Xmx512M -Xms32M >> /data/logs/xxx.log 2>&1'

enter image description here

Santhosh Hirekerur
  • 820
  • 8
  • 14
  • -1 for defining jvm parameters in wrong order. -Xmx512M must be defined before -jar . Also what you experience is expected. Systemd does not invoke services using shell – Sami Korhonen Dec 02 '19 at 21:42
  • 4
    @SamiKorhonen, I added my comments after testing this is working. Even I was thinking about order of -Xmx512M is smilar to you. Please test before adding blind comments. – Santhosh Hirekerur Dec 03 '19 at 00:56
4

Assume logs are already put to stdout/stderr, and have systemd unit's log in /var/log/syslog

journalctl -u unitxxx.service

Jun 30 13:51:46 host unitxxx[1437]: time="2018-06-30T11:51:46Z" level=info msg="127.0.0.1
Jun 30 15:02:15 host unitxxx[1437]: time="2018-06-30T13:02:15Z" level=info msg="127.0.0.1
Jun 30 15:33:02 host unitxxx[1437]: time="2018-06-30T13:33:02Z" level=info msg="127.0.0.1
Jun 30 15:56:31 host unitxxx[1437]: time="2018-06-30T13:56:31Z" level=info msg="127.0.0.1

Config rsyslog (System Logging Service)

# Create directory for log file
mkdir /var/log/unitxxx

# Then add config file /etc/rsyslog.d/unitxxx.conf

if $programname == 'unitxxx' then /var/log/unitxxx/unitxxx.log
& stop

Restart rsyslog

systemctl restart rsyslog.service
Hieu Huynh
  • 1,005
  • 11
  • 18
0
  • In my case 2>&1(stdout and stderr file descriptor symbol) had to be placed correctly,then log redirection worked as I expected
[Unit]
Description=events-server

[Service]
User=manjunath
Type=simple
ExecStart=/bin/bash -c '/opt/events-server/bin/start.sh my-conf   2>&1 >> /var/log/events-server/events.log'

[Install]
WantedBy=multi-user.target
Manju N
  • 886
  • 9
  • 14
0

Make your service file call a shell script instead of running the app directly. This way you have extra control. For example, you can make output files like those in /var/log/

Make a shell script like /opt/myapp/myapp.sh

#!/bin/sh
/usr/sbin/logrotate --force /opt/myapp/myapp.conf --state /opt/myapp/state.tmp
logger "[myapp] Run" # send a marker to syslog
myapp > /opt/myapp/myapp.log 2>&1 &

And your service file myapp.service contains:

...
[Service]
Type=forking
ExecStart=/bin/sh -c /opt/myapp/myapp.sh
...

A sample of log config file /opt/myapp/myapp.conf

/opt/myapp/myapp.log {
    daily
    rotate 20
    missingok
    compress
}

Then you will get myapp.log, and zipped myapp.log.1.gz ... for each time the service was started, and previous zipped.

Liyong Zhou
  • 71
  • 4