9

We need to implement a graceful shutdown mechanism into our Servlet application.

EDIT: We want to make it simple as possible, which would be handling a kill signal sent via operating system's function. This would allow system admins to use built in shell utilities (kill or taskkill on Windows), otherwise they would have to install another utility just to "talk" with server.

This mechanism works in two phases:

  1. upon shutdown request, deny certain critical activities
  2. block until previously initiated critical actions are completed; these may take several hours

Phase #1 is implemented in our DAO layer. Phase #2 is implemented in our ServletContextListener#contextDestroyed method

Our problem is that once contextDestroyed is called the Servlet container stops servicing further HTTP requests.

EDIT: contextDestroyed is called when someone is calling the operating system's kill function on server's process.

We would like to let the application alive during Phase #2, notifying the users that some activities are unavailable.

Adrian Herscu
  • 738
  • 1
  • 6
  • 19
  • Good question. If the activities that are to be denied are implemented in specific servlets, you could always make these (and only these) return a status in the 4xx range if a DB lookup suggests that a shutdown is in progress? – Anders R. Bystrup Aug 26 '13 at 12:55
  • Some of the activities are initiated by UI actions and some are initiated by a scheduler -- that's why I am thinking about controlling them at the DAO layer. – Adrian Herscu Aug 26 '13 at 17:42

1 Answers1

5

Use a filter to keep a list of all critical requests.

When the "prepare shutdown" request is received, the filter should start denying some requests.

Write a servlet that tells you how many critical jobs are still left in the queue.

In the shutdown tool, send the "prepare shutdown". The poll the servlet for the number of critical jobs. When this reaches 0, send the actual shutdown command.

To make this happen, create a service in the business layer which orchestrates this. Note that everything must happen before contextDestroyed() is being called! Your special application shutdown doesn't fit into the J2EE view of the world, so you have to manage it yourself.

The service should be able to tell interested parties when a shutdown is in progress, how many critical jobs are still running, etc. Servlets and filters can then use this service to deny requests or tell how many jobs are left.

When all jobs are done, deny all requests except access to the "shutdown info" servlet which should then tell that the app is now ready for death.

Write a tool which gives the administrators a nice UI to initiate shutdown of your app.

[EDIT] You may feel tempted to prevent the OS from shutting down your application. Don't do that.

What you should do is write a special tool to shut down your application using the two phase process that I described above. This should be the standard way to shutdown.

Yes, administrators will complain about it. On Unix, you can hide this tool by putting it into the init script, so no one will notice. There might be a similar solution on Windows.

Killing the server should always be possible to be able to stop it in case of (un)expected circumstances like: bugs in your shutdown code, emergency shutdown during power failure, bugs in your application code, or when Murphy happens.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
  • Some of the activities are initiated by UI actions and some are initiated by a scheduler. A filter won't work with actions initiated by the scheduler. – Adrian Herscu Aug 26 '13 at 17:44
  • The filter will need to keep the list/queue in a global place (static variable or an entry in the application scope). That makes it possible to implement it in such a way that the scheduler can see it. If everything else fails, use a DI framework. – Aaron Digulla Aug 27 '13 at 08:10
  • That would make the business logic layer "know" about the presentation layer, wouldn't it? – Adrian Herscu Aug 27 '13 at 16:19
  • Actually, the shutdown logic is part of the business layer. The presentation layer just feeds it with information when to start. Denying requests is the PL asking the BL whether shutdown is in progress. Showing the number of jobs is the PL displaying info from the BL. So I don't see how that would invert the relation. – Aaron Digulla Aug 28 '13 at 07:10
  • Since there tasks which are initiated by users (through the presentation layer) and there are tasks which scheduled to run at various intervals and/or events we cannot detect critical activities at the presentation layer (as suggested by you). We can detect these at the data access layer (DAO). The detection and blockage of the aforementioned activities is not the issue. The issue is keeping the context, servlets and filters alive after the context the contextDestroyed method has been called by the container. I could not find anything in the Servlets API that would allow such thing.Am I wrong? – Adrian Herscu Aug 28 '13 at 18:04
  • No, your're right: That's not something Java EE can do. You have to look outside of the box. Or you can keep your servlets up and running as I suggested by having a two-phase shutdown. – Aaron Digulla Aug 29 '13 at 07:37
  • I decided to go with JMX and it seems that it is working. I will mark this as an answer. – Adrian Herscu Aug 29 '13 at 12:17