8

I'm using Kubernetes, and a training job runs on the cluster. I'm using TQDM as progress bar, but unlike what I've expected, the progress bar doesn't show up when I check Kubernetes Pod logs. Does anyone have a solution to this problem?

Piljae Chae
  • 987
  • 10
  • 23

2 Answers2

9

I don't have a good solution, but this one helped me:

tqdm progress bar will show as expected if you use:

kubectl attach <pod_name>

instead of:

kubectl logs <pod_name>
GalDude33
  • 7,071
  • 1
  • 28
  • 38
1

So far unfortunately I haven't found a satisfying answer to this either. Depending on whether you use kubectl or k9s for example or even how the iterable looks like inside tqdm, or what else you log via for example the logging module, the results can be wildly different from no progress bar shown at all, to sporadic flushes of dozens of iterations. The only way I have found to get consistent kubernetes logs is printing them yourself every iteration and redirecting tqdm output to somewhere else.

For example as follows:

import time
import sys
from tqdm import tqdm

iterator = tqdm(range(99), file=open("/dev/null", "w"))

for x in iterator:
    print(iterator.__str__())
    sys.stdout.flush()
    time.sleep(0.2)
RagingPixels
  • 57
  • 6