Python app does not print anything when running detached in docker

Question

I have a Python (2.7) app which is started in my dockerfile:

CMD ["python","main.py"]

main.py prints some strings when it is started and goes into a loop afterwards:

print "App started"
while True:
    time.sleep(1)

As long as I start the container with the -it flag, everything works as expected:

$ docker run --name=myapp -it myappimage
> App started

And I can see the same output via logs later:

$ docker logs myapp
> App started

If I try to run the same container with the -d flag, the container seems to start normally, but I can't see any output:

$ docker run --name=myapp -d myappimage
> b82db1120fee5f92c80000f30f6bdc84e068bafa32738ab7adb47e641b19b4d1
$ docker logs myapp
$ (empty)

But the container still seems to run;

$ docker ps
Container Status ...
myapp     up 4 minutes ... 

Attach does not display anything either:

$ docker attach --sig-proxy=false myapp
(working, no output)

Any ideas whats going wrong? Does "print" behave differently when ran in background?

Docker version:

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.2
Git commit (client): a8a31ef
OS/Arch (client): linux/arm
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.2
Git commit (server): a8a31ef

Answer 1

Finally I found a solution to see Python output when running daemonized in Docker, thanks to @ahmetalpbalkan over at GitHub. Answering it here myself for further reference :

Using unbuffered output with

CMD ["python","-u","main.py"]

instead of

CMD ["python","main.py"]

solves the problem; you can see the output now (both, stderr and stdout) via

docker logs myapp

---

why -u ref

- print is indeed buffered and docker logs will eventually give you that output, just after enough of it will have piled up
- executing the same script with python -u gives instant output as said above
- import logging + logging.warning("text") gives the expected result even without -u

what it means by python -u ref. > python --help | grep -- -u

-u     : force the stdout and stderr streams to be unbuffered;

Answer 2

In my case, running Python with -u didn't change anything. What did the trick, however, was to set PYTHONUNBUFFERED=1 as environment variable:

docker run --name=myapp -e PYTHONUNBUFFERED=1 -d myappimage

[Edit]: Updated PYTHONUNBUFFERED=0 to PYTHONUNBUFFERED=1 after Lars's comment. This doesn't change the behavior and adds clarity.

Answer 3

If you want to add your print output to your Flask output when running docker-compose up, add the following to your docker compose file.

web:
  environment:
    - PYTHONUNBUFFERED=1

https://docs.docker.com/compose/environment-variables/

Answer 4

See this article which explain detail reason for the behavior:

There are typically three modes for buffering:

• If a file descriptor is unbuffered then no buffering occurs whatsoever, and function calls that read or write data occur immediately (and will block).
• If a file descriptor is fully-buffered then a fixed-size buffer is used, and read or write calls simply read or write from the buffer. The buffer isn’t flushed until it fills up.
• If a file descriptor is line-buffered then the buffering waits until it sees a newline character. So data will buffer and buffer until a \n is seen, and then all of the data that buffered is flushed at that point in time. In reality there’s typically a maximum size on the buffer (just as in the fully-buffered case), so the rule is actually more like “buffer until a newline character is seen or 4096 bytes of data are encountered, whichever occurs first”.

And GNU libc (glibc) uses the following rules for buffering:

Stream               Type          Behavior
stdin                input         line-buffered
stdout (TTY)         output        line-buffered
stdout (not a TTY)   output        fully-buffered
stderr               output        unbuffered

So, if use -t, from docker document, it will allocate a pseudo-tty, then stdout becomes line-buffered, thus docker run --name=myapp -it myappimage could see the one-line output.

And, if just use -d, no tty was allocated, then, stdout is fully-buffered, one line App started surely not able to flush the buffer.

Then, use -dt to make stdout line buffered or add -u in python to flush the buffer is the way to fix it.

Answer 5

Since I haven't seen this answer yet:

You can also flush stdout after you print to it:

import time

if __name__ == '__main__':
    while True:
        print('cleaner is up', flush=True)
        time.sleep(5)

Answer 6

Try to add these two environment variables to your solution PYTHONUNBUFFERED=1 and PYTHONIOENCODING=UTF-8

Answer 7

You can see logs on detached image if you change print to logging.

main.py:

import time
import logging
print "App started"
logging.warning("Log app started")
while True:
    time.sleep(1)

Dockerfile:

FROM python:2.7-stretch
ADD . /app
WORKDIR /app
CMD ["python","main.py"]

Answer 8

If anybody is running the python application with conda you should add --no-capture-output to the command since conda buffers to stdout by default.

ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "my-app", "python", "main.py"]

Answer 9

I had to use PYTHONUNBUFFERED=1 in my docker-compose.yml file to see the output from django runserver.

Answer 10

As a quick fix, try this:

from __future__ import print_function
# some code
print("App started", file=sys.stderr)

This works for me when I encounter the same problems. But, to be honest, I don't know why does this error happen.

Answer 11

If you aren't using docker-compose and just normal docker instead, you can add this to your Dockerfile that is hosting a flask app

ARG FLASK_ENV="production"
ENV FLASK_ENV="${FLASK_ENV}" \
    PYTHONUNBUFFERED="true"

CMD [ "flask", "run" ]

Answer 12

When using python manage.py runserver for a Django application, adding environment variable PYTHONUNBUFFERED=1 solve my problem. print('helloworld', flush=True) also works for me.

However, python -u doesn't work for me.

Answer 13

Usually, we redirect it to a specific file (by mounting a volume from host and writing it to that file).

Adding a tty using -t is also fine. You need to pick it up in docker logs.

Using large log outputs, I did not have any issue with buffer storing all without putting it in dockers log.