How to automatically shutdown a GCE VM after the docker container finishes?

Question

I have a batch process (packaged in a docker image) that needs to run once a day somewhere in the cloud. Google Compute Engine (GCE) seems like a simple and easy to use option for this with it's container-optimized OS VM. This works really well, except that I cannot find an easy way to automatically shutdown the VM after the docker container finishes. You can specify a startup script but GCE seems to execute it before the container is started, so doing docker wait there does not work. I don't care that it's a docker-optimized VM. One of the other basic Linux OSs (like Debian) is fine as long as it can easily be setup with docker.

Is there an easy way of automatically shutting a Linux VM down after the docker container finishes?

Inside your container execute a program that shuts down the virtual machine. This could be a REST API or a simple Python (any language) program. My guess is that you are using Compute Engine Container Optimized OS, so you are limited in what you can do within the instance OS. — John Hanley, Mar 02 '22 at 23:15
Yes, it is Compute Engine Container Optimized OS, running Python within the image. What is the code that would shutdown the GCE VM? — Derrick, Mar 02 '22 at 23:44

score 3 · Answer 1 · answered Mar 03 '22 at 00:01

As requested in the question comments, Python code to shutdown a Compute Engine instance.

Note: The Google Compute Engine Metadata server can provide the REPLACE variables in the script. Also, you do not need to wait for the results to be STOPPED, just verify that the script did not fail. You can test this program locally as well.

See this answer for tips on COS container credentials. The program below uses ADC (Application Default Credentials).

from pprint import pprint
import time

from googleapiclient import discovery
from oauth2client.client import GoogleCredentials

credentials = GoogleCredentials.get_application_default()

service = discovery.build('compute', 'v1', credentials=credentials)

# Project ID for this request.
project = 'REPLACE_WITH_PROJECT_ID'

# The name of the zone for this request.
zone = 'REPLACE_WITH_COMPUTE_ENGINE_ZONE'

# Name of the instance resource to start.
instance = 'REPLACE_WITH_COMPUTE_ENGINE_INSTANCE_NAME'

request = service.instances().stop(project=project, zone=zone, instance=instance)
response = request.execute()

pprint(response)

print('Waiting for operation to finish...')
print('Name:', response['name'])

while True:
    result = service.zoneOperations().get(
        project=project,
        zone=zone,
        operation=response['name']).execute()

    print('status:', result['status'])

    if result['status'] == 'DONE':
        print("done.")
        break;

    if 'error' in result:
        raise Exception(result['error'])

    time.sleep(1)

score 1 · Accepted Answer · answered Mar 14 '22 at 00:14

1

I was able to avoid shutting down the VM from within the docker container by simply deploying a regular Debian VM on Compute Engine. Steps:

First, create a Debian VM on Compute Engine and pass a startup script that installs docker. Startup script:

#!/bin/bash
# Install docker. Taken from the docker documentation.
# Passed into the gcloud command that creates the VM instance.
apt-get -y remove docker docker-engine docker.io containerd runc
apt-get update
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get -y install docker-ce docker-ce-cli containerd.io
echo Done installing docker.

Second, replace that startup script with one that pulls and runs the docker image:

#!/bin/bash
...
# Get authorization to pull from artifact registry
gcloud -q auth configure-docker us-east1-docker.pkg.dev
# Pull the image and run it, then shutdown.
docker pull $image
docker rm $container
docker run -v $vol_ini:$app_ini -v $vol_log:$app_log --name $container $image
shutdown now

Now this VM can be scheduled to run the docker image on a daily basis and automatically shut itself down.

answered Mar 14 '22 at 00:14

Derrick

323
2
10

This answer will shock many people who follow it at the end of the month. Be careful assuming that the Linux **shutdown** command stops the VM instance. Billing still occurs as the VM instance is still running. You must stop the VM instance via a Google Cloud interface (CLI, API, etc.). Go back and follow my answer. – John Hanley Mar 14 '22 at 01:20
Thanks John. It works fine for me at least. I have been verifying via the console that it does stop as expected, and so far it has never failed to shutdown. However, maybe there are cases where it will not work. Do you have specific details? I definitely do not want to get caught by surprise. – Derrick Mar 14 '22 at 18:06
Check your billing statement next month. However, your answer does **NOT** answer the question you asked **How to automatically shutdown a GCE VM after the docker container finishes?** – John Hanley Mar 14 '22 at 18:39
I explicitly check the VM state on Google Console and it definitely shows the VM as stopped. This does exactly what the question asks: `docker run` starts and waits for the container to finish, after which `shutdown now` causes the VM to stop. – Derrick Mar 14 '22 at 23:29
Your question's comment states **Compute Engine Container Optimized OS**. Your answer is for Debian on Compute Engine and not for COS. – John Hanley Mar 15 '22 at 00:20
Ahh, I see. Edited for clarification. The shutdown seems to be working fine. I'll give some feedback in a few weeks. – Derrick Mar 16 '22 at 00:11
1

After running this for a couple months I can confirm that the `shutdown now` command does indeed terminate the VM. This approach works perfectly for my use case since the daily batch process is very short (only a few minutes), resulting in monthly charges of well under one dollar. In fact, I have been able to reduce this to less than 10 cents by using a pre-emptible VM. The risk of pre-empting is minimal due to the short run time, and (in my case) the rare pre-emption is acceptable. – Derrick Jun 04 '22 at 12:57

How to automatically shutdown a GCE VM after the docker container finishes?

2 Answers2