3

I want to output a logfile from a Docker container and stumbled across something that I don't understand. These two lines don't fail, but only the first one works as I would like it to:

tail --follow "/var/log/my-log" &
tail --follow "/var/log/my-log" | sed -e 's/^/prefix:/' &

Checking inside the running container, I see that the processes are running but I only see the output of the first line in the container output.

Dockerfile

FROM debian:buster-slim

COPY boot.sh /

ENTRYPOINT [ "/boot.sh" ]

boot.sh

Must be made executable (chmod +x)!

#!/bin/sh

echo "starting"

echo "start" > "/var/log/my-log"
tail --follow "/var/log/my-log" &
tail --follow "/var/log/my-log" | sed -e 's/^/prefix:/' &

echo "sleeping"

sleep inf

Running

  • Put the two files above into a folder.
  • Build the image with docker build --tag pipeline .
  • Run the image in one terminal with docker run --init --rm --name pipeline pipeline. Here you can also watch the output of the container.
  • In a second terminal, open a shell with docker exec -it pipeline bash and there, run e.g. date >> /var/log/my-log. You can also run the two tail ... commands here to see how they should work.
  • To stop the container use docker kill pipeline.

I would expect to find the output of both tail ... commands in the output of the container, but it already fails on the initial "start" entry of the logfile. Further entries to the logfile are also ignored by the tail command that adds a prefix.

BTW: I would welcome a workaround using pipes/FIFOs that would avoid writing a persistent logfile to begin with. I'd still like to understand why this fails. ;)

Ulrich Eckhardt
  • 16,572
  • 3
  • 28
  • 55
  • In the second command, you are only running the sed command to the background and not the initial tail command. Try (tail --follow "/var/log/my-log" | sed -e 's/^/prefix:/') & – Raman Sailopal Dec 31 '20 at 10:17
  • Hi @RamanSailopal! Thanks for the suggestion, but that doesn't work. Also, I wouldn't say that that is how pipelines work. After all, running e.g. `htop`, I verified that the processes are all running (i.e. both `tail` commands and `sed`), but only the standalone `tail` produces any output that exits the container. – Ulrich Eckhardt Dec 31 '20 at 11:38

1 Answers1

3

Based on what I have tested, It seems that sed is causing the issue where the output of this command tail --follow "/var/log/my-log" | sed -e 's/^/prefix:/' & does not appear while running the container. The issue can be solved by passing -u to sed which disables the buffering.

The final working boot.sh will be as follow:

#!/bin/sh

echo "starting"

echo "start" > "/var/log/my-log"
tail --follow "/var/log/my-log" &
tail --follow "/var/log/my-log" | sed -u -e 's/^/prefix:/' &

echo "sleeping"
sleep inf

And the output after running the container will be:

starting
sleeping
start
prefix:start

Appending more data to the log file will be displayed as expected too.

starting
sleeping
start
prefix:start
newlog
prefix:newlog

Also see: why cant I redirect the output from sed to a file

Mostafa Hussein
  • 11,063
  • 3
  • 36
  • 61
  • Yes, thank you, it's really that easy. :) Just for completeness, even without the `-u` flag, `sed` will at some point read input data and process it. – Ulrich Eckhardt Jan 19 '21 at 22:53