0

I've spent the past couple days working on this, and at this point I am super stuck. I have a Java program that must be run not as a service. This program must also be capable of updating itself when a new file is given for updating.

As a result, I have a script that is started with Linux that starts the Java application and then checks every 5 seconds if the application has been terminated. If the application has been terminated, it should check if there is an update and then start appropriately.

This is the code for that script:

#!/bin/bash
JAVA_HOME=/usr/lib/jvm/java-16-openjdk-amd64
WORKING_DIR=~/Data
LOG=$WORKING_DIR/logs/Bash.log

rm $LOG
echo "Script started" > $LOG

while true; do
  source $WORKING_DIR/Server.pid

  if ! kill -0 $AppPID; then

    echo "App must be started" >> $LOG

    source $WORKING_DIR/UpdateStatus
    if [ "$UpdateReady" -eq "1" ]; then
      echo "Moving files for update" >> $LOG

      mv $WORKING_DIR/Server.jar $WORKING_DIR/old.jar
      mv $WORKING_DIR/new.jar $WORKING_DIR/Server.jar
    fi

    nohup ${JAVA_HOME}/bin/java -jar ${WORKING_DIR}/Server.jar &
    echo AppPID="$!" > $WORKING_DIR/Server.pid
    echo "Server started" >> $LOG

    if [ "$UpdateReady" -eq "1" ]; then
      echo "Checking for safe update" >> $LOG
      source $WORKING_DIR/Server.pid
      echo UpdateReady="0" > $WORKING_DIR/UpdateStatus

      sleep 5;

      if kill -0 $AppPID; then
        echo "Update successful" >> $LOG
        rm $WORKING_DIR/old.jar
      else
        echo "Update failed, restarting old jar" >> $LOG
        rm $WORKING_DIR/Server.jar
        mv $WORKING_DIR/old.jar $WORKING_DIR/Server.jar
        nohup ${JAVA_HOME}/bin/java -jar ${WORKING_DIR}/Server.jar &
        echo AppPID="$!" > $WORKING_DIR/Server.pid
      fi

    fi

    echo "Server start process finished, going into idle state" >> $LOG
  fi

  sleep 5
  echo "5 seconds idle passed" >> $LOG
done

To initialize the update, I have tried a couple of different things, both with the same result. First I had set UpdateReady="1" through Java, then used exit(0);. I have also tried having Java call a Bash script which also sets UpdateReady="1" but uses kill $AppPID to shutdown the java application.

The result is that both the Java application and the Bash script stop executing causing the update and restart to fail! I have looked through a significant amount of Stack Overflow questions and answers finding things such as nohup, all to no avail.

I will once again state that the Java application cannot be run as a service. No packages other than what is included in Java or made by Apache can be used, and no programs can be installed to Linux. I would prefer to solve the problem with Bash.

Upon testing some things mentioned in comments, I may have missed something that turns out to be important. While all other runs of the startup script will be run by the startup applications manager, the initial run is not.

The install is taken care of remotely with an SSH connection sending the following command string:

cd /home/UserName; unzip -u -o Server.zip; chmod 777 install.sh; bash install.sh &; exit

install.sh is as follows:

#!/bin/bash
INSTALL_DIR=~/Data

mkdir ${INSTALL_DIR}
mkdir ${INSTALL_DIR}/logs
mkdir ${INSTALL_DIR}/data

cp Server.jar ${INSTALL_DIR}/Server.jar
cp service-start.sh ${INSTALL_DIR}/service-start.sh

chmod 777 ${INSTALL_DIR}/service-start.sh

rm Server.jar
rm service-start.sh
rm Server.zip

nohup bash $INSTALL_DIR/service-start.sh &

Upon rebooting my machine, I noticed that this problem goes away! This means that there must be a problem with the initial setup script. When the shell command is run, it does seem to be sticky and not actually let go after the bash install.sh &. I have tried putting nohup at the beginning of this, however the entire line will not run in that case for reasons I am not able to determine.

I would prefer to not have the user be forced to restart after install and can't seem to find any way to force the Startup Application manager to start an application at any time other than startup.

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
  • Did you know that you can run `bash -x yourscript` to get a debug log of what the script does? – that other guy Aug 12 '21 at 22:59
  • On my system, `kill -0 ...` does nothing so far as I can tell; per the kill manpage: "If sig is 0, then no signal is sent, but error checking is still performed." On your system, does it do something different? – ruakh Aug 12 '21 at 23:07
  • You should probably be doing this with a systemd timer or cron. In any case, the running app and its 'watcher' should be separate entities. It must be said that something that checks a process every five seconds *is* a service – g00se Aug 12 '21 at 23:17
  • 1
    @ruakh The error checking it performs includes whether or not the process exists. This is a typical way of checking whether a pid you started is still alive. – that other guy Aug 12 '21 at 23:19
  • @g00se I mean a service in terms of a process that is controlled by systemd. In that situation one part of the Java application will not run correctly. I am trying to make the running app and the watcher separate entities, but that is what I am failing to do – Leone Shamoth Aug 12 '21 at 23:53
  • @thatotherguy I did not know this. The log file is still more useful in my case due to the nature of the system – Leone Shamoth Aug 12 '21 at 23:54
  • *In that situation one part of the Java application will not run correctly.* Why so? – g00se Aug 12 '21 at 23:57
  • @g00se due to a GUI that the java application starts. I tried for about 3 days prior to this to get a systemd service to allow for the GUI to appear to no avail. – Leone Shamoth Aug 12 '21 at 23:59
  • Ah right. Still, the 'watcher' could run as a systemd timer – g00se Aug 13 '21 at 00:01
  • @g00se Potentially yes, however I have a strong feeling that it will have the same problems as a systemd service. I will try this tomorrow though. – Leone Shamoth Aug 13 '21 at 00:04
  • Have you checked what process corresponds to `$AppPID`? Java won't kill its parent shell spontaneously. And indeed, I don't think it even knows what its parent shell's PID is if you use `nohup`. But ... if `$AppPID` is actually the shell itself, and Java is killing *that* ... – Stephen C Aug 13 '21 at 00:07
  • @LeoneShamoth Sorry, I was unclear. I meant "Please run the script with `bash -x yourscript` to get a debug log. This will likely show you if/how/why the script exits". If the problem is not immediately obvious, please include this log – that other guy Aug 13 '21 at 00:20
  • @StephenC Yes I have made sure that $AppPID does not correspond to Bash. There are two completely separate items in System Monitor and I have confirmed that $AppPID is for the Java process. – Leone Shamoth Aug 13 '21 at 00:21
  • Post has been updated to include new information that appears to be relevant. – Leone Shamoth Aug 13 '21 at 00:45
  • @g00se I do not see any information on the internet on using a systemd timer to do anything other than start a systemd service. Do you have any resource which shows otherwise? – Leone Shamoth Aug 13 '21 at 13:24
  • I'll see what I can find – g00se Aug 13 '21 at 13:51
  • Update: I thought I could find some workaround, but it seems that systemd really doesn't play well with GUIs. Maybe a systemd maven can offer some advice. I suspect it's not impossible – g00se Aug 13 '21 at 15:32
  • @thatotherguy: Oh, I see; thanks! – ruakh Aug 14 '21 at 07:51

1 Answers1

0

Well, after a lot of searching and some prompting from the comments, I found that the issue lied with how the program was initially being started.

As mentioned in the update, the first run is always started by an ssh connection. I knew there was a slight problem with this ssh connection, as it seemed to hold onto the connection no matter what I did. It turns out that this was causing the problem that resulted in the Bash instance and the Java instance remaining attached.

The solution for this problem was found here: jsch ChannelExec run a .sh script with nohup "lose" some commands

After managing to get the initial setup to start with nohup properly, the issue has gone away.