7

I setup my CD with following commands to deploy container to gce:

gcloud compute instances stop my-awesome-app
gcloud compute instances update my-awesome-app --no-shielded-integrity-monitoring
gcloud beta compute instances update-container my-awesome-app --container-image=docker.io/my-docker-hub-user/my-awesome-app:${IMAGE_TAG}
gcloud compute instances start my-awesome-app

The instance will then stop, replace container image and start.

If I create new instance with the latest image, everything works fine.

If I deploy with my CD pipeline, the instance ALWAYS hang forever (after receiving start event, no new logging is coming), NEVER pulls the new image, and NEVER goes up and running.

Anyone can tell me what's the problem?

gocreating
  • 161
  • 2
  • 9

3 Answers3

17

I don't start and stop my instance to update the container. I just:

gcloud compute instances update-container $instanceName2 --zone $instanceZone --container-image $registry:$BUILD_ID

But before pulling a new image I always do a docker prune to avoid my disk getting filled with images: 

gcloud compute ssh $instanceName2 --zone $instanceZone --command 'docker system prune -f -a'

For me, a better way was to use a template pointing to my repository, and from that template to build a Managed-Instance-group. This way you can have zero downtime deployments and still have only one machine up 99% of the time. Details

TudorIftimie
  • 1,050
  • 11
  • 21
2

I hope the document [1] is helpful for you from where you could have steps for updating a container on a VM in Google Cloud.

When you want to update a VM running a container, Compute Engine performs below two steps:

  1. Updates container declaration on the instance. Compute Engine stores the updated container declaration in instance metadata under the gce-container-declaration metadata key.
  2. Stops and restarts the instance to actuate the updated configuration, if the instance is running. If the instance is stopped, updates the container declaration and keeps the instance stopped. The VM instance downloads the new image and launches the container on VM start.

Please follow 1 to 5 steps as the document [1], Compute Engine saves the changes and restarts the instance automatically to make the update. After the VM restarts, it downloads the new image and starts the container with the updated configuration.

[1] https://cloud.google.com/compute/docs/containers/deploying-containers#updating_a_container_on_a_vm_instance

Shafiq I
  • 406
  • 2
  • 7
2

Thanks for above answers!

I finally figured out my own way. 1. Every deploy will create a new managed instance with container. 2. Then my reserved IP address is pointed to the new instance. (So I don't need to update DNS) 3. Finally, shutdown the old deployment when possible.

The downside is obviously there will be down time during deploying, but I think the deploy pipeline is good enough for my current project.

# install dependencies
apt-get --yes install jq

# Create new instance
NEW_INSTANCE="$INSTANCE_NAME_PREFIX-build-$CIRCLE_BUILD_NUM"
gcloud beta compute instances create-with-container "$NEW_INSTANCE" \
    --machine-type=f1-micro \
    --container-image=docker.io/xxx/yyy:$IMAGE_TAG \
    --container-env="ANSIBLE_VAULT_PASSWORD=${ANSIBLE_VAULT_PASSWORD}" \
    --tags http-server,https-server \
    --zone ${GOOGLE_COMPUTE_ZONE}

IP_ADDRESS=$(gcloud compute addresses list --filter="name=('$RESERVED_IP_NAME')" --format=json | jq '.[0].address' --raw-output)
OLD_INSTANCE=$(gcloud compute instances list --filter="EXTERNAL_IP=('$IP_ADDRESS')" --zones asia-east1-a --format=json | jq '.[0].name' --raw-output)

# Remove ephemeral IP address from new instance
echo "Removing ephemeral IP address from instance: $NEW_INSTANCE"
gcloud compute instances delete-access-config "$NEW_INSTANCE" \
    --access-config-name "external-nat" \
    --zone ${GOOGLE_COMPUTE_ZONE}

# Remove reserved IP address from old instance
# Ignore error if there is no access config present
if [ "$OLD_INSTANCE" != "null" ]; then
    echo "Removing reserved IP address from instance: $OLD_INSTANCE"
    gcloud compute instances delete-access-config "$OLD_INSTANCE" \
        --access-config-name "external-nat" \
        --zone ${GOOGLE_COMPUTE_ZONE} || true
fi

# Assign reserved IP address to new instance
gcloud compute instances add-access-config "$NEW_INSTANCE" \
    --access-config-name "external-nat" --address "$IP_ADDRESS" \
    --zone ${GOOGLE_COMPUTE_ZONE}

# Shutdown old instance
if [ "$OLD_INSTANCE" != "null" ]; then
    gcloud compute instances stop "$OLD_INSTANCE" --zone ${GOOGLE_COMPUTE_ZONE}
fi
gocreating
  • 161
  • 2
  • 9