4

When something fails on the server side because the database and the application are out of sync instead of getting an error and the application crashing spring/tomcat seems to swallow the exception and pretend nothing has happened.

Call me crazy but if the program fails catastrophically I want it to actually fail catastrophically! Is there anyway to switch this behaviour off? It's really slowing development down when the server pretends that everything is fine when it's just thrown up into the logs.

If this isn't the spring/tomcat default then what else might be causing it? We are using a boatload of libraries and frameworks unfortunately. Spring would be the usual suspect but it could be something else.

Update

It's a sql server database which we are connecting to using SqlServerDataSource. Hibernate is in use in some parts of the project but is used to query the database at login time. On the client side we are using extjs and we are also using ExtDirectSpring to annotate methods for the client side to talk to. To translate the data going across the wire there's Jackson, which then gets wrapped by the extdirect json handler.

There's some AOP stuff going on thats to do with logging exceptions but deleting that code results in the same behaviour.

Further update

Ok its not a good idea to let your sever crash! See my answer below for my proposed middle ground.

JonnyRaa
  • 7,559
  • 6
  • 45
  • 49
  • Do you know that there are exceptions, or do you just know that no exceptions are propagating? If the latter, the issue may be in the database access layer - e.g., Hibernate or other JPA provider - because the beans defined in that layer tolerate and ignore extra database columns, and can end up erasing any of those extra columns in updates. If the former, I'd just say "don't use Spring", but that may be easier said than done. – Warren Dew Jun 27 '14 at 16:47
  • 1
    Give an example of a catastrophical failure that does not emit any errors or warnings – Jukka Jun 27 '14 at 16:55
  • 3
    I very much doubt Spring is to blame for this. What kind of database? Give us more context details. – Sotirios Delimanolis Jun 27 '14 at 17:02
  • 1
    Do you really think that tomcat (because Spring has nothing to do with that) should crash if a web application let go an uncaught exception ??? If it did, it will not be as used as it actually is... – Serge Ballesta Jun 27 '14 at 19:52
  • @warrendew there are definitely exceptions. Hibernate is being used in some parts of the project but not for the new stuff. – JonnyRaa Jun 30 '14 at 08:50
  • @SotiriosDelimanolis I've updated my post a bit. Does that help? – JonnyRaa Jun 30 '14 at 09:00
  • @SergeBallesta possibly not (there could be other apps in it etc) but I would expect it to let my application crash. Or at least for that to be configurable – JonnyRaa Jun 30 '14 at 09:02
  • Spring's AOP could be responsible for swallowing exceptions in some very specific advice cases. If you say you removed it and the same behavior occurred, then I don't know what could be the problem. You'd have to debug further. – Sotirios Delimanolis Jun 30 '14 at 14:27
  • I have a feeling that OP wants the JVM process to be terminated (or webapp to be undeployed)... so question to OP -> is that really, I mean REALLY, what you want? Please tell me I have misunderstood the question. Maybe describe what is the expected and actual behavior a bit better. – Pavel Horal Jun 30 '14 at 14:29
  • @PavelHoral I'm from a desktop application environment. That's the behaviour I'd expect in a desktop application - any exception (albeit not where you are dealing with unpredictable things like the file system) would crash the application. I dont see why an application should be allowed to more unreliable just because it doesn't all run on the same machine. See [this post](http://stackoverflow.com/questions/2416316/why-is-the-catchexception-almost-always-a-bad-idea) or [this article](http://research.microsoft.com/en-us/people/mickens/ToWashItAllAway.pdf) for a more humorous take – JonnyRaa Jun 30 '14 at 16:08
  • Letting the server exit just because some library throw runtime exception would be pretty bad idea. If a user request results in erroreous state, there is close to zero reason to take down the whole server with that. You just stop request processing, log error and return error response if possible. There is very good chance that the problem might have been just in that single request. Usually the only thing which is capable of bringing server down are [Errors](http://docs.oracle.com/javase/7/docs/api/java/lang/Error.html) (e.g. OutOfMemoryError). – Pavel Horal Jun 30 '14 at 20:17
  • @PavelHoral turns out you guys were right! Taking down the server is not a good thing to do. I've decided to take a different route instead - I've posted in my answer below – JonnyRaa Jul 03 '14 at 09:39

2 Answers2

2

If you really want to do that (but IMHO you should not ...) you can use a filter that will block the application once it let an uncaught exception go. It could be something like :

public class CrashFilter implements Filter {
    private boolean crashed = false;
    private String msg = "Major problem : application stopped";

    @Override
    public void doFilter(ServletRequest sr, ServletResponse sr1, FilterChain fc) throws IOException, ServletException {
        if (crashed) {
            HttpServletResponse resp = (HttpServletResponse) sr1;
            resp.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, msg);
            return;
        }
        try {
            fc.doFilter(sr, sr1);
        }
        catch (Exception ex) {
            crashed = true;
            throw new ServletException(ex);
        }
    }
    // init and destroy omitted from brevity
}
Serge Ballesta
  • 143,923
  • 11
  • 122
  • 252
  • thanks for the response. what is resp in the above example? It doesnt compile on my machine - I've tried both sr and sr1 and neither seem to have a sendError method. – JonnyRaa Jun 30 '14 at 14:01
  • 1
    @JonnyLeeds : sorry, I forgot one line ... Post edited. You should also write it in one single line `((HttpServletResponse) sr1).sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, msg);` – Serge Ballesta Jun 30 '14 at 14:17
  • You know you can always log `Fatal error occured.` and call [`System.exit(1);`](http://docs.oracle.com/javase/7/docs/api/java/lang/System.html#exit%28int%29)... but any reason for doing that would be questionable (together with developer's sanity). – Pavel Horal Jun 30 '14 at 14:26
  • Just checked Servlet specification and there is a better standard way to take down web application -> throw [UnavailableException](http://docs.oracle.com/javaee/1.3/api/javax/servlet/UnavailableException.html). – Pavel Horal Jun 30 '14 at 20:32
2

Ok so I did this in the end. I've basically used the ideas above but thought there was enough extra to post my own answer.

It turns out you really shouldn't do this as other people suggested and I've added a bit at the bottom to say why!

Here's my filter:

public class FailOnErrorFilter implements Filter
{
    @Override
    public void init(FilterConfig config) throws ServletException
    {

    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain filterChain) throws IOException, ServletException
    {
        try {
            filterChain.doFilter(request, response);
        }
        catch (Exception exception) {
            System.exit(1);
        }
    }

    @Override
    public void destroy()
    {

    }
}

To get this working you have to modify the web.xml:

<filter>
    <filter-name>failingFilter</filter-name>
    <filter-class>fullyQualified.FailOnErrorFilter</filter-class>
    <async-supported>true</async-supported>
</filter>

<filter-mapping>
    <filter-name>failingFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

The top one defines the filter + the second one says use it everywhere.

ExtDirectSpring

After doing the above I realised that ExtDirectSpring was a further culprit. It's default behaviour is to swallow all exceptions in server side methods.

I was worried I was going to have to patch the library but thankfully someone else had already complained about this and it got fixed in 1.3.6. Initially I tried upgrading to the latest version but it broke a load of code! What a great library. Anyway 1.3.6 added in the ability to switch off the suppression of errors by doing the following:

//this is to switch off blanket switching of exceptions in extdirect spring
@Component
public class NoExceptionHandling implements RouterExceptionHandler
{
    @Override
    public Object handleException(MethodInfo methodInfo, BaseResponse baseResponse, Exception e, HttpServletRequest httpServletRequest)
    {
        throw new RuntimeException(e);
    }
}

As the name suggests extdirectspring uses spring and so doesn't make its dependencies obvious to calling code, however if you go digging (its on github). You'll see in RouterController it calls the following method in the catch

private Object handleException(MethodInfo methodInfo, BaseResponse response, Exception e, HttpServletRequest request) {
    return configurationService.getRouterExceptionHandler().handleException(methodInfo, response, e, request);
}

Where router controller does this:

@Autowired(required = false)
private RouterExceptionHandler routerExceptionHandler;

public RouterExceptionHandler getRouterExceptionHandler() {
    return routerExceptionHandler;
}

It sets up a default one if you dont provide one.

Update - why you shouldn't do this

It turns out you really shouldn't call System.exit in a tomcat application. Not only does it bring down your application it also causes the server to exit. This brings down any other applications running aswell!

It's also not appropriate for a number of other reasons:

  • if the first in a series of tests throws an exception then all subsequent tests will fail
  • Its time consuming restarting the server and you have to be the person who breaks it to see the exception
  • if you are running a manual test deployment on a seperate machine then you have to restart the server if something caused a problem somewhere.

Likewise:

  • In production it will take everyone's application down and most users wont be in a position to restart the server.

What I'm doing instead

The errors were already being written the the tomcat logs + the database.

  • In debug we now also going to redirect to an error page with the stacktrace
  • In production we are going to just redirect to a 'something went wrong' page. We're also going to set up an email service that notifies us of exceptions.
  • For UI/selenium tests - it'll work the same as debug
  • For headless Js tests the server rejects subsequent requests until the next test resets the error state of the server

Just to make things more complicated unsurprisingly the original webapp is too flaky to not mask errors so I've kept the old error suppression in place for that as we're not actively developing/fixing it at the moment.

JonnyRaa
  • 7,559
  • 6
  • 45
  • 49