The issue
I've stumbled upon the issue:
Error message: org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
General info about the issue
It's a Spring Boot app (2.4.5) running on Reactive (WebFlux) stack. The app also uses Playtika OSS reactive Feign Client (starter 3.0.3) for synchronious REST API communication. Underlying web client is Netty. There are no any special Feign or WebClient configs in the app.
All the other microservice parties are running on embedded Tomcat with default Spring Boot autoconfigurations.
All apps are running in Kubernetes cluster.
The error log observed from time to time (not every day).
Guesses
After some investigation, my best guess would be that some long-lived connections are being dropped from the pool on certain conditions. This causing the error log.
This thought is based on Instana that connects the error log to the span that spans accross a lot of subcalls.
Also no data losses/ other inconsistencies were noticed so far xD
Questions
- Does Feign has a connection pool by default?
- How to know if those are live or idle connections from the pool being closed?
- How the connection pool can be configured or dropped to avoid long-running connections ?
- Is it possible that Kubernetes can somehow close these connections?
- What else can close connections?
