How to log why WELD SE Container is shutting down

Question

EDIT: It turns out that my docker container was closing because of an errant health check failing. This caused the kubernetes probe to fail too many times and killed the pod. Since the command came from outside the JVM, it is likely not possible for the code to know when the container around the JVM was killed. Leaving this question up in case someone finds an answer.

I have a Java SE project that reads in arguments to process files. I am using Weld-SE for injection. After a few minutes of running, the weld container shuts down mid progressing without explanation. The following is a skeleton of my code. Here in my Main Class

@ApplicationScoped
public class Main {

  @Setter //Lombok annotation
  private Scheduler scheduler;

  public void schedule(String[] filenames) throws InterruptedException {
    scheduler.schedule(filenames);
  }

  @PreDestroy
  public void cleanUp() {
    log.info("PreDestroy - Main");
    //clean up omitted
  }

  public static void main(String[] args) {
    Weld weld = new Weld();
    Main main = null;
    try (WeldContainer container = weld.initialize()) {
      main = container.select(Main.class).get();
      Scheduler scheduler = CDI.current().select(Scheduler.class).get();
      main.setScheduler(scheduler);
      main.schedule(args);
      log.info("Main - completed processing");
    } catch (Exception e) {
      log.info("Exception occurred: " + e.getMessage(), e);
    } finally {
      log.info("Main Finally - Shutting down Weld");
      CDI.current().destroy(main);
      weld.shutdown();
    }
  }
}

Here is my Scheduler class

@Named
public class Scheduler {

  private ThreadPoolExecutor executor;
  private Map<Future<?>, Importer> beanMap = new HashMap<>();
  private static final Duration sleepDuration = Duration.ofSeconds(45);

  @PostConstructor
  public void init() {
    //setup executor
  }

  @PreDestroy
  public void cleanUp() {
    log.info("PreDestroy - Scheduler");
    //clean up omitted
  }

  public void schedule(String[] filenames) throws InterruptedException {
    for(String filename : filenames) {
      Importer caller = CDI.current().select(Importer.class).get();
      caller.setFilename(filename);
      Future<?> future = executor.submit(caller);
      beanMap.put(future, caller);
    }

    while(!isDone()) {
       Thread.sleep(sleepDuration.toMillis());
       log.info("Remaining threads: {}, getRemaining());
       destroyCompleted();
    }
  }

  //Returns true only when all threads are completed
  private boolean isDone() { //code omitted }

  //Returns number of not completed threads
  private int getRemaining() { //code omitted }

  //Clean up any completed Importers
  private boolean destroyCompleted() { //code omitted }
}

Here is my Importer code

@Named
public class Importer implements Callable<String> {

  @Setter //lombok annotation
  private String filename;

  public void cleanUp() {
    log.info("PreDestroy - Importer");
    //clean up omitted
  }

  public String call() {
     List<String> content = //extract file content
     for(int i = 0; i < content.size; i++) {
       if (i+1 % 10000 == 0) {
         log.info("Processed {} rows in file {}", i+1, filename);
       }
       //rest of loop omitted
     }
  }
}

The file test.txt has around 100,000 lines so I can tell that the crash happens mid file processing. So after my code runs for a few minutes I get the following at the end of my log

Processed 50000 rows in file test.txt
Processed 60000 rows in file text.txt
PreDestroy - Scheduler
PreDestroy - Importer
PreDestroy - Main
WELD-ENV-002001: Weld SE container 03128098-039d-46db-97e5-8538a52a38cc shut down

I am logging "org.jboss" at DEBUG level. I can see that the container was shutdown midway through processing a file, but I don't see why it occurred. Is there some type of listener that I can extend/implement to see why shut down command happened?

Answer 1

For future reference: You can define an observer method for the BeforeShutdown event which you can use to at least close all open files and such before shutdown. It won't necessarily give you a reason for why it's shutting down, but it's a start.

If you say the JVM process was terminated from outside, then there's probably an InterruptedException thrown somewhere which you can catch. Note that catching InterruptedException and not re-interrupting the thread is a bug most of the time. In your case, you should react to it with an orderly shutdown when calling Thread#sleep. If the thread gets interrupted, someone wants to cancel the execution. They probably have a good reason for doing so; don't think you know better than them!

At the very least you can periodically check the interrupt flag of your threads at some convenient point during execution to initiate an orderly shutdown of your task.