3

I have a Spring Boot application with several endpoints declared like this:

@RestController
public class MyRestController {

    @PostMapping("/someRequest")
    public void doSomething(final @RequestBody MyRequest request) {
       // ...
    }

}

How can I find out whether or not the socket used by this controller has TCP keepalive turned on or not?

Update 1: I started the application with libdontdie, i. e. sudo DD_DEBUG=1 DD_TCP_KEEPALIVE_TIME=4 DD_TCP_KEEPALIVE_INTVL=5 DD_TCP_KEEPALIVE_PROBES=6 LD_PRELOAD=/usr/lib/libdontdie.so java -jar myapp.jar --spring.config.location=myapp-config.yaml &. Need to wait until tomorrow to see whether or not it works.

Glory to Russia
  • 17,289
  • 56
  • 182
  • 325
  • 3
    Sorry, what do you mean not working after a long time of inactivity ? The endpoint should work anytime as long the server is up and running. What is the problem and how is this related to keepalive ? – s7vr Mar 02 '20 at 13:24
  • @user2683814 That endpoint is used by a webhook of another application (JIRA). Let's say I start my application. When JIRA sends a request to my application immediately after start, it works (the request is received and processed). Then I let my application run for many hours during which it does not receive any requests. Then, JIRA sends a request to that endpoint. This request is not processed by my application. It is processed only after I restart my application. – Glory to Russia Mar 02 '20 at 13:27
  • FYI: The application runs on Ubuntu 18.04 in an AWS (EC2) cloud. It is perfectly possible that the issue is OS-specific. – Glory to Russia Mar 02 '20 at 13:28
  • Okay. Is there anything in the logs ? Assuming your application never received the second request it could be something at the network level. do you use any service discovery like consul or eureka in the middle ? do you have any health checks set up on app scanned by some external monitor ? – s7vr Mar 02 '20 at 13:35
  • @user2683814 Re *do you use any service discovery like consul or eureka in the middle ? do you have any health checks set up on app scanned by some external monitor ?*: No to both questions. – Glory to Russia Mar 02 '20 at 13:38
  • 3
    Thanks. I still don't think at this point this is related to tcp keep alive. Could you confirm if your app was still up when the second request came ?. If no the issue could be with your app. Have you tried to profile your app and see if application memory requirements are met and there are no memory leaks ? I'm not familiar with aws aspect of this so it could be problematic too. If you have aws logs you should check them too. – s7vr Mar 02 '20 at 14:18
  • Can you draw a rough picture of how your application looks like ? Where and how is this controller hosted? How is it called ? What logs do you see (both application and aws) when the POST request is received after presumably the endpoint has 'died' ? – Saif Asif Mar 03 '20 at 14:57
  • Also which server do you have running under the hood via spring boot ? Do you have some custom Servlet loading logic/configuration ? – Saif Asif Mar 03 '20 at 15:06
  • if youre running springboot with tomcat and default properties the keep-alive options is true. https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#server-properties – Alejandro Venegas Feb 20 '20 at 19:06

2 Answers2

2

It is very much possible that one of the following is happening. This might not be related to keepalive.

Cause

  1. Your AWS resource is facing a cold start or getting into hibernation due to idle CPU. This could be due to AWS
  2. Your /someRequest endpoint is dependent on some downstream resources like Database, another restful service, or may be something to do with I/O is in idle state.
  3. If it is connecting to database, make sure you have the reconnect enabled. This could depend on how you have setup your connection. See Spring is losing connection to the DB and does not recover or reconnect for how to do it for your situation.

Recommendation to resolve

  1. Check the health of the instance on AWS console and see if it has hit IDLE. Try to get the node to IDLE.
  2. Using Spring Boot Actuator enable test for health of downstream dependencies and ensure your health is good before making any API call. See, https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html
  3. I do not want to recommend this, but all your above options dry then create a dummy endpoint like /info or /health and keep polling it periodically. This is not a great solution but expect to work in most cases.
reflexdemon
  • 836
  • 8
  • 21
0

Polling is the solution. Keep polling it after a specific interval.

vaibhavkhot
  • 502
  • 3
  • 3