0

My setup is spring-boot service running in kubernetes pods. I have set terminationGracePeriodSeconds to 30 seconds and also configured graceful shutdown time in my spring boot service as spring.lifecycle.timeout-per-shutdown-phase: 30. I am trying with below test controller -

@GetMapping("/hello")
    public String hello(@RequestParam("time") long time) throws InterruptedException, IOException {
        System.out.println("Sleeping for " + time +  " milliseconds...");
        Thread.sleep(time);

        System.out.println("Done Sleeping");

        OkHttpClient client = new OkHttpClient();

        Request request = new Request.Builder()
                .url("http://www.google.com")//My server address will go here
                .build();
        Response response = client.newCall(request).execute();
        System.out.println("*********************** " + response.code());
        return "Hello World";
    }

I tried two scenarios

  • Locally ran spring boot service. Hit above endpoint and stopped the server using - kill pid (SIGTERM). I am able to hit google.com endpoint and get the response back.

  • Deployed these changes on K8 pod. Hit above endpoint and deleted the pod using - kubectl delete pod my-pod -n my-namespace. In this case, i successfully get the "Done Sleeping" log but as soon as it tries to make call to google.com it fails with -

SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception
java.net.ConnectException: Failed to connect to www.google.com/172.217.27.196:80
    at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183)
    at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
    at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
    at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
    at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
    at
......
......
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
    at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
    ... 116 more

It looks like during graceful shutdown time period of kubernetes, we cannot hit external endpoint. I cannot find any documentation around it. Is this correct ?

wut
  • 19
  • 4

1 Answers1

0

During terminationGracePeriodSeconds period, your Pod is in the Terminating phase, which means that it can complete the requests it already received, but any service will not pass new requests to this Pod.

Read more at the related SO question: Do Kubernetes pods still receive requests after receiving SIGTERM?

Rafał Leszko
  • 4,939
  • 10
  • 19
  • I understand that the pod won't take any new request but my question is is it allowed to make any external calls during that graceful shutdown period. – wut Dec 03 '21 at 09:21
  • I think the pod is allowed to make external calls in the `Terminating` phase. – Rafał Leszko Dec 03 '21 at 09:50
  • @wut were you able to find a solution to this? I am in the same boat. I try to uplaod some logs to aws after sigterm is recieved in my k8 container but the outbound calls to aws are failing for me while the container is still in terminationGracePeriodSeconds. I get "Could not connect to the endpoint s3://" – java_doctor_101 May 07 '23 at 18:10