18

I am running bash scripts from java programs on a server. I just uploaded a new version of the script intending the next run of the script to use the version. I did not mean to interrupt the existing, running instances of the script. However, I just received over 100 notifications of crashes from my 300 servers. I'm guessing replacing the running bash script with a new version caused this. However, this would require that the running bash script is reading from the disk as it gets to each new step. Is this how it works?

The running versions of the bash script run some ray tracing software. Each run takes 2 hours. Sub-steps take between 5 minutes and 1.5 hours. The script always reports crashing after finishing a step in the script. It never reports crashing an already running sub-step. Some crashes report not finding commands that I cannot find in the script. Different crashes report different places.

help!

EDIT: I copied the script to all 300 servers using scp. The file was replaced on the file system. This is not a shared-file.

Tim Perry
  • 3,066
  • 1
  • 24
  • 40

4 Answers4

18

SiegeX is half right - bash will load an entire script into memory, so a script can continue to run even if it's source file is deleted while the process is running. But bash will also check whether the source file is updated while the script is running. If it has been, bash will reload it and continue running it from the current position reopen the file, seek to the current position of the script, and continue running the script from that point.

Here's a proof-of-concept script:

# If you modify a script, will it change the behavior of
# processes that are currently running that script?
# Does this script print "Foo" or "Bar"?

cat >foo.sh <<EOF
sleep 5
echo Foo
EOF

bash foo.sh &
sleep 2

cat >foo.sh <<EOF
sleep 5
echo Bar
EOF

wait

So the upshot is don't modify the source files of bash scripts if you care about the processes that are currently running that script.

(This script, however, displays "Foo". The "current position" of the bash script is always at the beginning or end of a line.)

echo "sleep 5 ; echo Foo" > foo.sh
bash foo.sh &
sleep 2
echo "sleep 5 ; echo Bar" > foo.sh
wait
mob
  • 117,087
  • 18
  • 149
  • 283
  • Wow. That is exactly what I ran into. I was fairly sure something like this was happening based on behavior, but I thought SiegeX was correct. Thanks for taking the time to respond. – Tim Perry Jan 21 '11 at 06:02
  • No, mob's explanation is wrong, stating that it reads all at invocation and checks updates. See [my answer](http://stackoverflow.com/questions/8335747/emacs-workflow-to-edit-bash-scripts-while-they-run/8926090#8926090) to the question [Emacs workflow to edit Bash scripts while they run](http://stackoverflow.com/questions/8335747/emacs-workflow-to-edit-bash-scripts-while-they-run). (Even if Emacs is irrelevant to you. =) It also states how you can modify a running shell script. – teika kazura Dec 01 '12 at 04:58
1

I'm not sure if things have changed since mob's answer, which has been accepted, but in bash 4.3.46 (such as comes with ubuntu 16.04), it is true that bash monitors the script file for changes, but this is broken if the file is deleted.

So a slight modification to his script does the 'right' thing:

# If you modify a script, will it change the behavior of
# processes that are currently running that script?
# Does this script print "Foo" or "Bar"?

cat >foo.sh <<EOF
sleep 5
echo Foo
EOF

bash foo.sh &
sleep 2

rm foo.sh

cat >foo.sh <<EOF
sleep 5
echo Bar
EOF

wait

This now prints Foo.

Tom
  • 7,269
  • 1
  • 42
  • 69
1

Don't update a running system if you can avoid it

Deleting the script is one thing, but modifying it may produce more "interesting" results.

Also, changing a file that is replicated and/or network-mounted introduces behavior specific to the filesystem and deployment protocols. These are not going to be modelled accurately by a simple test on a local hard mount or one where a network mount is modified on the same system that is reading the file.

Furthermore, "uploading" this file to 300 servers introduces all kinds of wonderful complexity that us overflowians probably don't have nearly enough information to analyze.

ISTM that your issues probably are related to the update. I think the mystery commands may come from bash reading part of a script from the old version and part from the new version. I do know that you should probably shut down the subsystem, if possible, while updating.

DigitalRoss
  • 143,651
  • 25
  • 248
  • 329
  • Well...I scp'ed the file to all 300 servers separately so this isn't a network file sharing problem. Shutting them down and brining them back up is non-trivial, but probably the correct thing to do. – Tim Perry Jan 21 '11 at 01:54
  • While I think the opening advice is solid, and for a different question I might vote it up, I don't think it's useful for answering this specific question. And what the heck is an "overflowian"? – Bryan Oakley Jan 21 '11 at 01:58
0

Upon invocation, bash will fork off a new process and load the script into memory. Unless the script references itself, it should still work. Here is an example of a script both deleting itself from disk and still working until it tries to access itself via $0

Input

#!/bin/bash

echo "I can't take it anymore"
echo "Goodbye World!"

rm -f "$0"

echo "Wait, I change my mind!"

sed 's/Goodbye/Hello/' "$0"

Output

$ ./suicide.sh
I can't take it anymore
Goodbye World!
Wait, I change my mind!
sed: can't read ./suicide.sh: No such file or directory
SiegeX
  • 135,741
  • 24
  • 144
  • 154
  • Thanks. That is what I thought. My problem must be elsewhere. – Tim Perry Jan 21 '11 at 01:44
  • No, this explanation is wrong. See [my comment to mob's answer](http://stackoverflow.com/questions/4754592/could-replacing-a-bash-script-with-a-new-version-cause-a-running-instance-of-the?lq=1#comment18740353_4755281). (It's true that the running bash can survive the deletion of the script.) – teika kazura Aug 07 '13 at 08:50