I am trying to follow for the most part this guide to deploy a Google Workflow to launch a container optimized VM on compute engine, run a long running task, and then clean up the VM. I am also using the GCP docs in Creating a Workflow with Terraform.
I have my image built. The image successfully runs locally and also runs on the container optimized VM when I launch it through the console. The image is built in my CI/CD pipeline on Cloud Build and uploaded to Google Container Registry.
However, when I try to launch the container via the Cloud Workflow console, I get an error that seems to me to be related to authentication to GCR. Here is the stack trace from Cloud Logging.
I can't explain this because both the VM launched from the console and the one launched via workflows use the same service account with the same scopes. The guide I am following suggests taking the REST-equivalent of a known working VM and converting that into a YAML config to be used in the Terraform code. This is what I have done. And upon comparing the two machines in the console.. they look identical to me.
This is the Terraform code used to create the workflow. I pass the Commit SHA into the Terraform script to reference the container. I have confirmed that the container built on Cloud Build can be downloaded and run on my local machine.
resource "google_workflows_workflow" "workflows_example" {
name = "scraper-workflow"
region = "us-central1"
description = "Scraper workflow"
service_account = "scraper-workflow-executor@rdmops-219503.iam.gserviceaccount.com"
source_contents = <<-EOF
# FYI, In terraform you need to escape the $$ or it will cause errors.
- init:
assign:
- commitSHA: ${var.SCRAPER_IMAGE_COMMIT_SHA}
- projectId: $${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
- projectNumber: $${sys.get_env("GOOGLE_CLOUD_PROJECT_NUMBER")}
- zone: "us-central1-a"
- machineType: "e2-medium"
- ticker: $${args.ticker}
- instanceName: $${ticker+"-scraper"}
- create_and_start_vm:
call: googleapis.compute.v1.instances.insert
args:
project: $${projectId}
zone: $${zone}
body:
canIpForward: false
confidentialInstanceConfig:
enableConfidentialCompute: false
deletionProtection: false
shieldedInstanceConfig:
enableIntegrityMonitoring: true
enableSecureBoot: false
enableVtpm: true
tags:
items:
- http-server
- https-server
name: $${instanceName}
labels:
- "container-vm": "cos-stable-97-16919-29-40"
machineType: $${"zones/" + zone + "/machineTypes/" + machineType}
disks:
- initializeParams:
diskSizeGb: "10"
diskType: "projects/rdmops-219503/zones/us-central1-a/diskTypes/pd-balanced"
sourceImage: "projects/cos-cloud/global/images/cos-stable-97-16919-29-40"
boot: true
autoDelete: true
deviceName: $${instanceName}
# Needed to make sure the VM has an external IP
networkInterfaces:
- accessConfigs:
- name: "External NAT"
networkTier: "PREMIUM"
stackType: "IPV4_ONLY"
subnetwork: "projects/rdmops-219503/regions/us-central1/subnetworks/default"
# The container to run
metadata:
items:
- key: "google-logging-enabled"
value: "true"
- key: "gce-container-declaration"
value: '$${"spec:\n containers:\n - name: scraper-workflow\n image: gcr.io/" + projectId + "/scraper-workflow:" + commitSHA + "\n stdin: false\n tty: false\n restartPolicy: Never\n"}'
# Needed to be able to pull down and run the container
serviceAccounts:
- email: 937088654099-compute@developer.gserviceaccount.com
scopes:
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring.write
- https://www.googleapis.com/auth/servicecontrol
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/trace.append
- log_wait_for_vm_network:
call: sys.log
args:
data: $${"Waiting for VM network to initialize"}
- wait_for_vm_network:
call: sys.sleep
args:
seconds: 10
- get_instance:
call: googleapis.compute.v1.instances.get
args:
instance: $${instanceName}
project: $${projectId}
zone: $${zone}
result: instance
- extract_external_ip_and_construct_urls:
assign:
- external_ip: $${instance.networkInterfaces[0].accessConfigs[0].natIP}
- base_url: $${"http://" + external_ip + "/"}
- start_url: $${base_url + "start"}
- poll_url: $${base_url + "poll"}
# Redacted rest of workflow for simplicity
EOF
}
And this is the REST-equivalent of the instance I create through the console. This works fine. The SA looks the same to me. And the scopes... And all of the other info...
POST https://www.googleapis.com/compute/v1/projects/rdmops-219503/zones/us-central1-a/instances
{
"canIpForward": false,
"confidentialInstanceConfig": {
"enableConfidentialCompute": false
},
"deletionProtection": false,
"description": "",
"disks": [
{
"autoDelete": true,
"boot": true,
"deviceName": "instance-1",
"initializeParams": {
"diskSizeGb": "10",
"diskType": "projects/rdmops-219503/zones/us-central1-a/diskTypes/pd-balanced",
"labels": {},
"sourceImage": "projects/cos-cloud/global/images/cos-stable-97-16919-29-40"
},
"mode": "READ_WRITE",
"type": "PERSISTENT"
}
],
"displayDevice": {
"enableDisplay": false
},
"guestAccelerators": [],
"keyRevocationActionType": "NONE",
"labels": {
"container-vm": "cos-stable-97-16919-29-40"
},
"machineType": "projects/rdmops-219503/zones/us-central1-a/machineTypes/e2-medium",
"metadata": {
"items": [
{
"key": "ssh-keys",
"value": "XXXXX"
},
{
"key": "gce-container-declaration",
"value": "spec:\n containers:\n - name: instance-9\n image: gcr.io/rdmops-219503/workflow-scraper:test3\n args:\n - ''\n stdin: false\n tty: false\n restartPolicy: Never\n# This container declaration format is not public API and may change without notice. Please\n# use gcloud command-line tool or Google Cloud Console to run Containers on Google Compute Engine."
}
]
},
"name": "instance-9",
"networkInterfaces": [
{
"accessConfigs": [
{
"name": "External NAT",
"networkTier": "PREMIUM"
}
],
"stackType": "IPV4_ONLY",
"subnetwork": "projects/rdmops-219503/regions/us-central1/subnetworks/default"
}
],
"reservationAffinity": {
"consumeReservationType": "ANY_RESERVATION"
},
"scheduling": {
"automaticRestart": true,
"onHostMaintenance": "MIGRATE",
"preemptible": false,
"provisioningModel": "STANDARD"
},
"serviceAccounts": [
{
"email": "937088654099-compute@developer.gserviceaccount.com",
"scopes": [
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring.write",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/service.management.readonly",
"https://www.googleapis.com/auth/trace.append"
]
}
],
"shieldedInstanceConfig": {
"enableIntegrityMonitoring": true,
"enableSecureBoot": false,
"enableVtpm": true
},
"tags": {
"items": [
"http-server",
"https-server"
]
},
"zone": "projects/rdmops-219503/zones/us-central1-a"
}
