4

I am using Kubernetes to run a python script as a cron job. The issue is that I am not seeing the output of the script (Which can take a while to run) until after the job finishes. I suspect this is due to the logging level (--v option) but I cannot for the life of my find either the documentation for it (it default to --v=0). IF I want to increase the verbosity of what is outputted, does anyone know the value of 'INFO' or 'TRACE' (or what the values are/where they are defined)? Thank for any help in advance.

Edit: has anyone successfully gotten a python file to log to a Kubernetes pod while the pod was running? If so did you use print() or a different logging framework?

HeronAlgoSearch
  • 1,581
  • 2
  • 18
  • 35
  • Could it be the behavior of your python script? Are you sure it's going to stdout? – thisguy123 May 16 '17 at 06:07
  • @thisguy123 thank you. I think I am logging to stdout, I am using a standard print() statement in python (which to my knowledge always logs to stdout). Whenever I run the script locally it certainly logs to stdout. I also tried logging to stderr, but no use-still showing "The selected container has not logged any messages yet." – HeronAlgoSearch May 17 '17 at 06:17
  • @thisguy123 The odd part is that I do see the logs, after my script finishes running (I am doing a print() -> sleep in a for loop a finite number of times). When the pod encounters crashbackLoop (i.e. when the script finishes) the logs get dumped to the log file. Is there a way to have the logs show up while the container is running? – HeronAlgoSearch May 17 '17 at 06:24
  • For reference I created an extremely simple script to test this: ` import sys import time def do_run(): print("Hello world1") print("hello world error", file=sys.stderr) time.sleep(120) if __name__ == "__main__": do_run() ` – HeronAlgoSearch May 17 '17 at 06:34

2 Answers2

4

According to Kubernetes docs, 

If you don't see much useful in the logs, you could try turning on 
verbose logging on the Kubernetes component you suspect has a problem 
using --v or --vmodule, to at least level 4. See 
https://github.com/golang/glog for more details.
fylie
  • 1,675
  • 1
  • 10
  • 14
  • do you know if that is logging from the pod itself? Or just logging from a Kubernetes component (as I think it indicates)? I am trying to get logs from my pod while my pod is running. – HeronAlgoSearch May 17 '17 at 06:25
  • 1
    @fylie, the link you posted is outdated now. However, I found a copy on archive.org: https://web.archive.org/web/20160121210548/https://github.com/kubernetes/kubernetes/wiki/Debugging-FAQ – user674669 Apr 30 '19 at 20:28
1

Found the root cause. Specifically, found it at Python app does not print anything when running detached in docker . The solution is to set the following environmental variable: PYTHONUNBUFFERED=0 . It was not that the print statement was not being displayed, it was that the print statement was being buffered. Doing the above will solve the issue. Thank you @thisguy17 and @fylie for assisting.

Community
  • 1
  • 1
HeronAlgoSearch
  • 1,581
  • 2
  • 18
  • 35