Kubernetes "logs" does not show any output for a simple python program running in a container in a pod

Question

I have a very simple setup. My .yaml-Configuration file for Kubernetes:

apiVersion: batch/v1
kind: Job
metadata:
  name: stdout-test
spec:
  template:
    spec:
      priorityClassName: research-high
      containers:
        - name: container-stdout-test
          image: <here comes my secret image repo>
          imagePullPolicy: "IfNotPresent"
          resources:
            limits:
              nvidia.com/gpu: "1"
              cpu: "1"
              memory: "8Gi"
            requests:
              nvidia.com/gpu: "1"
              cpu: "1"
              memory: "4Gi"
          command: ["python3", "/workspace/main.py"]
          volumeMounts:
            - mountPath: /workspace 
              name: localdir 
      imagePullSecrets:
        - name: lsx-registry
      restartPolicy: "Never"
      volumes:
        - name: localdir
          cephfs:
            monitors:
              - <here come my secret monitors>
            user: <namespace>
            path: "/home/stud/nothelfer/stdout-test" 
            secretRef: 
              name: <my secret>

And my simple python program (main.py):

import time

for i in range(0, 1000):
    print(i)
    time.sleep(1.0)

Starting and running the job in kubernetes works fine. I get this output from kubectl get pods:

NAME                READY   STATUS    RESTARTS   AGE
stdout-test-hl6qs   1/1     Running   0          5s

But i dont get the expected output 0, 1, 2, ... from stdout by using kubectl logs -f stdout-test-hl6qs. Instead i just get a blank screen. There is just no output printed to the console. I was expecting to get the output of my simple python program. I have tried all possible command line arguments for kubectl logs but none of them make me see the output of my Python program in the command line. The image is set up correctly, the Python program runs fine in the container, I have already checked all that. Can anyone help me here?

My Dockerfile:

FROM pytorch/pytorch
RUN conda install matplotlib pandas numpy
RUN conda update --all

ENV PYTHONUNBUFFERED 1

RUN python -c "import torch, pandas, matplotlib"

score 4 · Accepted Answer · answered Apr 17 '21 at 15:11

After quite a few tries, I figured out how to see the stdout output. In the Kubernetes configuration file, I had to replace line command: ["python3", "/workspace/main.py"] with command: ["python3", "-u", "/workspace/main.py"], which configures the output from Python to unbuffered. For reasons that aren't entirely clear, it didn't work when I only set variable ENV PYTHONUNBUFFERED=1 or ENV PYTHONUNBUFFERED 1 in the Dockerfile. I can now see the output of the program whether I specified ENV PYTHONUNBUFFERED=1 or ENV PYTHONUNBUFFERED 1 in the Dockerfile or not.

PYTHONUNBUFFERED

score 2 · Answer 2 · answered Apr 17 '21 at 06:12

2

Add the following line in your Dockerfile:

ENV PYTHONUNBUFFERED=1

See this

answered Apr 17 '21 at 06:12

dishant makwana

1,029
6
13

I have tried both `ENV PYTHONUNBUFFERED=1` and `ENV PYTHONUNBUFFERED 1` but neither results in me seeing any output. I updated my question with my corresponding Dockerfile. – Lukas Nothhelfer Apr 17 '21 at 14:06
You are not copying your .py files in your Docker image. How would the `main.py` file run inside kubernetes? I believe you will need something like: `COPY . /workspace` in your Dockerfile – dishant makwana Apr 17 '21 at 14:14
I am using Kubernetes. In the config.yaml of Kuberentes I mount the paths and set up the start command. I am not working with docker directly. I can confirm that the program is running in the container, that is not the problem. – Lukas Nothhelfer Apr 17 '21 at 14:21

score 2 · Answer 3 · answered Dec 26 '21 at 06:47

2

You can fix this without rebuilding your docker image. Just add tty: true in your k8s configuration, i.e., add spec.containers.[YOUR CONATINER].tty: true

Ref: https://github.com/kubernetes/kubernetes/issues/57414#issuecomment-352880955

answered Dec 26 '21 at 06:47

petertc

3,607
1
31
36

score 0 · Answer 4 · answered Sep 29 '21 at 08:41

0

What helped me see my logs in the k8s logs (using Lens) was to add the following after my execution statement |& tee /proc/1/fd/1

So you would basically have something like this

$ python some_python_program.py |& tee /proc/1/fd/1

answered Sep 29 '21 at 08:41

paradox

634
5
13

Kubernetes "logs" does not show any output for a simple python program running in a container in a pod

4 Answers4