4

I have one application which servers for REST request and also is listening on a Kafka topic. I deployed the application to Kubernetes and configure the readiness probe like this

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

basically following the instruction from [configure-liveness-readiness-startup-probes]

After deployment is done, I can see the pod readiness probe fails

Readiness probe failed: cat: can't open '/tmp/healthy': No such file or directory

That is expected. Then I sent a kafka message to the topic . I observed that

1) the kafka message has been consumed by my application and saved to database.
2) the rest api can't be accessed.

I assumed if the pod's readiness probe is failed, the application can neither receive kafka message nor the rest request. But why in my test, the REST request and Kafka message are handled differently.

According to the Kubernete documentation:

The kubelet uses readiness probes to know when a Container is ready to start accepting traffic

But it doesn't say clearly what kind of traffic it really means. Does kubernetes only restrict the http traffic to the pod if readiness probe failes but not restrict tcp traffic (as Kafka is working over tcp)?

My actual intention is to make my service application (kafka consumer) able to control when to receive kafka messages (and REST request as well). E.g. if there is heavy opertion, my service will delete the /tmp/healthy file and thus make the pod not ready for recieving kafka message and Rest request. When the heavy operation is finished, the app write the healthy file to make the pod ready for receiving message.

Some more information, in my test, the kubernetes version is v1.14.3 and kafka broker is running in a separated vm outside of kubernetes.

Jonas
  • 121,568
  • 97
  • 310
  • 388
Shenghua Liu
  • 43
  • 4
  • 1. the kafka message has been consumed by my application and saved to database > how did you sent a message? Failed readiness probe causes endpoint controller remove pod from balancing in case you have a k8s service in front of your pods (e.g. kafka.kafka.svc.cluster.local) – Konstantin Vustin Nov 21 '19 at 12:53
  • @KonstantinVustin, I used another producer application running in the same kubernetes to send the kafka message to the topic. After click one button from UI, it will send the kafka message to the topic by using kafka producer lib. The kafka broker and zookeeper are installed in a separated VM outside the kubernetes cluster. How could check if there is k8s service running in front my pods? Running `kubectl get services` shows there is kubernetes with type ClusterIP. – Shenghua Liu Nov 21 '19 at 15:15
  • which URL do you use for sending? – Konstantin Vustin Nov 21 '19 at 15:30
  • The Kafka Producer object uses "bootstrap.servers" property to send message to kafka topic. The values is like `ip_of_service_vm:9092`. The ip of the service vm is just normal IPV4 address. My consumer app is using the same format for kafka Consumer configuration. Producer app and consumer app are running in different Pods. – Shenghua Liu Nov 21 '19 at 19:08
  • Use Kafka's Pause and Resume APIs and call them in readiness controller. – user3153014 Jul 07 '23 at 04:40

1 Answers1

1

This is two very different things:

  • Receiving requests: An external service is sending a request and expect a response.
  • Sending requests: Your service is sending a request and waiting for a response.

ReadinessProbe

When a ReadinessProbe fails, no new requests will be routed to the pod.

Kafka consumer

If your pod is a Kafka consumer, then your pod is initializing requests to Kafka, to retrieve messages from the topic.

Check for required directory

can't open '/tmp/healthy': No such file or directory

If the directory /tmp/healthy is needed for your service to work correctly, your service should check for it on startup, and exit(1) (crash with an error message) if the required directory isn't available. This should be done before connecting to Kafka. If your application uses the directory continually, e.g. writing to it, any operations error codes should be checked and handled properly - log and crash depending on your situation.

Consuming Kafka messages

My actual intention is to make my service application (kafka consumer) able to control when to receive kafka messages ( and REST request as well). E.g. if there is heavy opertion, my service will delete the /tmp/healthy file and thus make the pod not ready for recieving kafka message and Rest request.

Kafka consumers poll Kafka for more data, whenever the consumer want. In other words, the Kafka consumer ask for more data whenever it is ready for more data.

Example consumer code:

 while (true) {
     ConsumerRecords<String, String> records = consumer.poll(100);
     for (ConsumerRecord<String, String> record : records) {
         // process your records
     }
 }

Remember to commit the records that you have processed so that the messages aren't processed multiple times e.g. after a crash.

Jonas
  • 121,568
  • 97
  • 310
  • 388
  • Thanks @Jonas. My actual intention is to make my service application (kafka consumer) able to control when to receive kafka messages ( and REST request as well). E.g. if there is heavy opertion, my service will delete the /tmp/healthy file and thus make the pod not ready for recieving kafka message and Rest request. When the heavy operation is finished, the app write the healthy file to make the pod ready for receiving message. – Shenghua Liu Nov 21 '19 at 20:14
  • One thing I am still not clear is that when my consumer app sending request and waiting response, how could the response be routed back to my consumer app if the pod is not ready? – Shenghua Liu Nov 21 '19 at 20:24
  • @ShenghuaLiu "how could the response be routed back to my consumer app if the pod is not ready" - The request is supposed to be routed to another replica of your service. – Jonas Nov 21 '19 at 20:34
  • Yes, your answer is most helpful so far. Thanks a lot. So basically it is not possible to prevent my consumer app getting kafka message dueto kafka's polling mechanism even though the pod is not ready, right? I read from some article e.g. [link] (https://dzone.com/articles/kafka-consumer-delivery-semantics), when consumer polls the messages, it actually reads the message from topic's partition. From consumer's point of view, reading is incoming traffic. But it looks like kubernetes doesn't block this incoming traffic even though the readiness probe fails. – Shenghua Liu Nov 22 '19 at 08:24
  • Back to my actual intention stated before, then I have to explicitly stop polling kafka message when there is heavy operation on going. Do you have any suggestion? – Shenghua Liu Nov 22 '19 at 08:25
  • @ShenghuaLiu, your app only poll (get more data) when it is ready to process more data. See my example... first the app has nothing to do, so it ask for more data (poll) then it process all data... until it has processed all data and has nothing to do again... and then ask for more data again (another poll in the loop). – Jonas Nov 22 '19 at 16:40