7

I have a python application that is running in Kubernetes. The app has a ping health-check which is called frequently via a REST call and checks that the call returns an HTTP 200. This clutters the Kubernetes logs when I view it through the logs console.

The function definition looks like this:

def ping():
    return jsonify({'status': 'pong'})

How can I silence a specific call from showing up in the log? Is there a way I can put this in code such as a python decorator on top of the health check function? Or is there an option in the Kubernetes console where I can configure to ignore this call?

William Ross
  • 3,568
  • 7
  • 42
  • 73
  • 1
    How are you pulling those logs? And can you provide a portion? – Crou Jan 03 '19 at 15:47
  • They are pulled by running kubernetes console 'kubectl proxy'. The going to http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ in the browser. Going to Replica sets, clicking on the pod I want to examine, and then there is a log button at the top which shows the console and logs. – William Ross Jan 03 '19 at 15:50

3 Answers3

5

In kubernetes, everything in container which you have on stdout or stderr will come into the kubernetes logs. The only way to exclude the logs of health-check ping call remove from kubernetes logs is that, In your application you should redirect output of those ping calls to somefile in say /var/log/. This will effectively remove the output of that health-check ping from the stdout.

Once the output is not in stdout or stderr of the pods, pod logs will not have logs from that special health-check

You can also use the sidecar containers to streamline your application logs, like if you don't want all of the logs of application in kubectl logs output. You can write those file.

As stated in Official docs of kubernetes:

By having your sidecar containers stream to their own stdout and stderr streams, you can take advantage of the kubelet and the logging agent that already run on each node. The sidecar containers read logs from a file, a socket, or the journald. Each individual sidecar container prints log to its own stdout or stderr stream.

This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it’s hardly a significant overhead.

For more information on Kubernetes logging, please refer official docs:

https://kubernetes.io/docs/concepts/cluster-administration/logging/

Prafull Ladha
  • 12,341
  • 2
  • 37
  • 58
  • If the call is an HTTP request which returns JSON data, how would I redirect the output to a different channel? – William Ross Jan 09 '19 at 15:47
  • As @Prafull said, redirect the stdout from you app, to a path (maybe a storage) visible to both containers (appliction and sidecar). Then, in the sidecar container, you can create a filter to output only what you want to it container stfdout. – Mauro Baraldi Jan 09 '19 at 15:56
  • You can utilize `request` utility in Python to get request on url and store the response in response object. Now you can redirect that object to a file (storage) you want. http://engineering.hackerearth.com/2014/08/21/python-requests-module/ – Prafull Ladha Jan 09 '19 at 16:00
  • The health check looks that the call ping call returns an HTTP 200 response code. Can I have the function return a 200 somehow but still not log? – William Ross Jan 09 '19 at 16:44
  • You can again use `request` library. `r = requests.head("https://stackoverflow.com") print(r.status_code)` This will give you only status_code. https://stackoverflow.com/questions/1140661/what-s-the-best-way-to-get-an-http-response-code-from-a-url – Prafull Ladha Jan 09 '19 at 17:27
0

clutters the Kubernetes logs when I view it through the logs console.

Since you said "view", I know it might not be the most accurate way to do this, but works ok:

kubectl logs [pod] [options] | awk '!/some health-check pattern|or another annoying pattern to exclude|another pattern you dont want to see|and so on/ {print $0}'

Used awk for filtering the logs, if you're unaware, you can read about it here

  • !: means exclude the following pattern
  • /pattern/: enclose your pattern between // (standard awk syntax)
  • {print $0}: means print the current line (you can also use just print)

Note: There is always a risk of excluding something necessary. Use wisely.

nightfury
  • 74
  • 1
  • 7
-1

Just invert the logic log on fail in the app, modify the code or wrap with a custom decorator

ms4720
  • 327
  • 1
  • 11
  • There is no logging for the function that is added. Kubernetes is logging it because it gets called. The call itself is not failing, it's just a simple ping style health check that confirms the app is running. – William Ross Jan 07 '19 at 13:34
  • If it is just a ping to see if app is running then use a liveness or readiness check to check the app and have k8s manage restarts if you want. Also look at what log level you are running, --v=X, the higher the number the more logs you get – ms4720 Jan 09 '19 at 04:21