1

My Dataflow job fails when it tries to access a secret:

"Exception in thread "main" com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/REDACTED/secrets/REDACTED/versions/latest' (or it may not exist)."

I launch the job using gcloud dataflow flex-template run. I am able to view the secret in the console. The same code works when I run it on my laptop. As I understand it, when I submit a job with the above command, it runs under a service account that may have different permissions. How do I determine which service account the job runs under?

user100464
  • 17,331
  • 7
  • 32
  • 40

2 Answers2

2

Since Dataflow creates workers, they create instances. You can check this on Logging

  • Open GCP console
  • Open Logging -> Logs Explorer (make sure you are not using the "Legacy Logs Viewer")
  • At the query builder type in protoPayload.serviceName="compute.googleapis.com"
  • Click Run Query
  • Expand the entry for v1.compute_instances.create or any other resources used by compute.googleapis.com
  • You should be able to see the service account used for creating the instance. This service account (boxed in red) is used anything related to the running the dataflow job.

Take note that I tested this using the official dataflow quick start.

enter image description here

Ricco D
  • 6,873
  • 1
  • 8
  • 18
0

By default the worker nodes of dataflow run with the compute engine default service account (YOUR_PROJECT_NUMBER-compute@developer.gserviceaccount.com) lacking of the "Secret Manager Secret Accessor" rights.

Either you need to add those rights to the service account or you have to specify the service account in the pipeline options:

gcloud dataflow flex-template run ... --parameters service_account_email="your-service-account-name@YOUR_PROJECT_NUMBER.iam.gserviceaccount.com"
sudo
  • 747
  • 6
  • 19