Android: Handle critical exception at startup

Question

So here's a weird issue. While developing my app, I've been using an uncaught exception handler to log any exceptions that will cause my app to fail. Sometimes I have a fail that occurs during the app's startup. Obviously I resolve those issues, but right now I'm concerned with my error handling.

When I'm using the UncaughtExceptionHandler, the exceptions end up being caught and logged, but the app doesn't shut down. Instead, in the logs it loops, constantly trying to start again before blowing up with the same exception, etc etc etc. On the device, a white screen that never goes away is all that appears.

I want to catch and log exceptions, but I also want the app to close and crash when that happens. What am I doing wrong?

Uncaught exception handler:

@Override
    public void uncaughtException(Thread thread, Throwable throwable) {
        Log.e(TAG, String.format("Uncaught exception in thread %d", thread.getId()), throwable);
        Process.killProcess(Process.myPid());
        System.exit(10);
    }