0

I am trying to get a shutdown script to execute using a Google Cloud compute VM.

I see this output when running gcloud compute connect-to-serial-port startup-test-v

Apr  8 22:01:25 startup-test-v shutdown-script: INFO Starting shutdown scripts.
Apr  8 22:01:25 startup-test-v shutdown-script: INFO Found shutdown-script in metadata.
Apr  8 22:01:26 startup-test-v shutdown-script: INFO shutdown-script: No change requested; skipping update for [startup-test-v].
Apr  8 22:01:27 startup-test-v shutdown-script: INFO shutdown-script: Return code 0.
Apr  8 22:01:27 startup-test-v shutdown-script: INFO Finished running shutdown scripts.

I create the preemptible instance from the command line and shut it down in the GUI.

gcloud compute instances create $INSTANCE_NAME \
    --zone=$ZONE \
    --image-family=$IMAGE_FAMILY \
    --image-project=deeplearning-platform-release \
    --maintenance-policy=TERMINATE \
    --machine-type=$INSTANCE_TYPE \
    --boot-disk-size=50GB \
    --metadata="install-nvidia-driver=True" \
    --preemptible \
    --scopes="storage-rw,cloud-platform" \
    --metadata-from-file="shutdown-script=gce/shutdown_test.sh"

shutdown_test.sh is simply:

#!/bin/bash
echo "+++ Shutdown test +++"
exit 0

Startup scripts are working as expected. I've tried swapping the --metadata-from-file flag to --metadata-from-file shutdown-script=gce/shutdown_test.sh as well, with no change.

Ideas? It seems GCE is finding the shutdown script, but not executing it.

Andrew Schreiber
  • 14,344
  • 6
  • 46
  • 53

2 Answers2

2

Turns out that images can overwrite the CLI-defined shutdown-script metadata

In my case, pytorch-latest-gpu image changes the shutdown-script metadata to point at its own shutdown script. It does this during first startup.

If you edit that script - defined at /opt/deeplearning/bin/shutdown_script.sh - you can get whatever shutdown behavior you like. Otherwise you can edit the metadata to point at your script. Your shutdown scripts will appear in serial output logs.

Apr  9 23:26:46 new-test-d shutdown-script: INFO Starting shutdown scripts.
Apr  9 23:26:47 new-test-d shutdown-script: INFO Found shutdown-script in metadata.
Apr  9 23:26:47 new-test-d shutdown-script: INFO shutdown-script: ++++++++++++++ Shutdown test +++++++++++++++++

You can prevent the image script from changing shutdown-script by not giving the instance the permission to add metadata via omitting cloud-platform in --scope. Or you can edit the shutdown-script in the GUI after startup. You may also be able to re-edit it back via a startup script.

Andrew Schreiber
  • 14,344
  • 6
  • 46
  • 53
  • A good finding and surprising stuff. It shouldn't be like that. In this case the vendor has stepped into the customer area from the IaaS model perspective. Instead of using custom metadata the vendor's shutdown script should have been embedded into the image. You can create an issue on [Issue Tracker](https://issuetracker.google.com/) regarding this. – mebius99 Apr 15 '20 at 08:12
1

Shutdown script does not provide any output neither to serial console or current SSH terminal console because it is running as root user and it's stdout is not routed neither to /dev/tty* or /dev/ttyS*.

To ensure that shutdown script does actually work, you might put into the VM metadata key shutdown-script something like below: 

#!/bin/bash 
ofile=/var/tmp/shutdown.txt 
echo "+++ Running shutdown script +++"
echo "id = $(id)" > $ofile 
echo "script_file path = $(realpath $0)" >> $ofile 
echo "script_file rights, user, group = $(stat -c "%A %U %G" $0)" >> $ofile

Next connect to both the Serial Console and SSH to get ready to watch the absence of any output provided by the shutdown script.

Then stop and start the VM.

In the end you will see that shutdown script has actually worked as root user and left a log file: 

$ ls -l /var/tmp/shutdown.txt 
 rw-r--r-- 1    root    root    165     Apr  9 18:40    shutdown.txt 
$ cat /var/tmp/shutdown.txt  
 id = uid=0(root) gid=0(root) groups=0(root) 
 script_file path = /tmp/metadata-scripts196132089/shutdown-script 
 script_file rights, user, group = -rwxr-xr-x root root

Shutdown script works for preemptible instances the same as for normal instances. The difference is that the former have shorter shutdown period (30 seconds).

mebius99
  • 2,495
  • 1
  • 5
  • 9
  • This approach did not work for me - `ls -l /var/tmp` would show `0 items`. – Andrew Schreiber Apr 09 '20 at 23:54
  • Could a custom script clean up /var/tmp/ on boot? On a default VM instance the /tmp/ directory is cleaned, but /var/tmp/ is not. You might try other destination for $ofile to check. – mebius99 Apr 10 '20 at 06:54