HTTP requests held up during Tomcat shutdown

Question

I am having an issue with Tomcat shutdown. Requests arriving while the servlet container is shutting down seem to be held up by the connector until container shutdown finishes.

The problem is, our servlet can take up to a minute to shut down, meaning connections arriving during this period are held for up to a minute.

Is this behavior correct/expected?

Wouldn't ideal Tomcat shutdown behavior be to

1. let existing requests finish (as it already does), but
2. reject new connections (instead of accepting and holding them)?

Answer 1

Apparently, you are not the only person to notice / comment on this behaviour; see this "tomcat-user" posting:

• http://grokbase.com/t/tomcat/users/144b4jka4t/http-requests-held-up-during-tomcat-shutdown.

(Indeed, the similarities between that posting and your Question suggest that there is some connection between yourself and Andry Eng ...)

So clearly, this is "expected" behaviour in some sense.

Whether it is "correct" is a matter of opinion. And it is moot if the Tomcat developers don't agree with your opinion.

And, for what it is worth, posting rhetorical questions here about the "ideal" behaviour is also moot. Apparently, the Tomcat developers either don't hang out in StackOverflow / "tomcat-users" ... or they are not willing to debate the issue.

---

So what possible solutions do you have?

• This linked Q&A explains a workaround - How to process servlet requests during long shutdown

• You could open an issue on the Tomcat issue tracker. The snag is that your issue could sit on the list for months or years before it is addressed. Or it could be "dismissed".

  However, it doesn't look like anyone has submitted this as an issue in the past.

• You could figure out how to modify Tomcat to reject new requests during shutdown. Then submit your (tested) changes as patches.

• You could hire someone to do the work for you. For example, take a look at the people / organizations listed here: http://wiki.apache.org/tomcat/SupportAndTraining.

Answer 2

This is correct behavior. Client would be sending request as long as it sees server is up at given port.

Server will accept request as long as it is up. So some of the requests which were received during shutdown process, will be received but won't be processed completely.

You will need to implement your own mechanism, if you want to handle these requests gracefully. You can use either JMS queue or filters. First request receiving filter would first store(db/serialize) request and then process it. Later when server restarts, you can check for requests which were not processed, you can either process them or notify client that it failed.

Answer 3

This is hardly desirable behavior. When shutting down it should stop accepting new connections but I guess there was some technical challenges in implementing it. Moot, anyway as noted.

How we got past it is this:

1. We have web servers in front of our tomcats. Get this in place, plan for long term if you do not, is my advice.
2. have a way to tell the web server to drop a node (tomcat instance) from its list of round robin instances.
3. after 3-4 minutes that tomcat node should be free (no new connections and old ones completed). We have a custom web server based on http://www.quickserver.org/ and it lets the back-end tomcats complete in process connections, though stops giving it new ones.
4. when maintenance is up, we have another service on the agent to add the tomcat back to the web server(s) list of active servers.

In our case we had a way to tell the server to reloads their configs. Other servers seem to have same https://serverfault.com/questions/108261/how-to-make-modification-take-affect-without-restart-nginx