Secret manager access denied despite correct roles for service account

Question

I'm writing a cloud function in Nodejs (10), and trying to access a secret like so:

const [secret] = await new SecretManagerServiceClient().accessSecretVersion({
    name: `projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/latest`
})

I created the secret in the web console and the name used in code matches that of the existing secret. On the page for the cloud function details, it states that the service account is PROJECT_ID@appspot.gserviceaccount,com, so I added the secretmanager.secretAccessor role to it. However, I'm still getting the same error every time:

Error: 7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/latest' (or it may not exist).

It makes no difference if I specify a concrete version or just use latest.

Should be working, double check secret name and service account name, then try removing secretAccessor role and adding it back. If this doesn't help, try redeploying the function. — Emil Gi, Jun 18 '20 at 08:02
Just to be sure: what are you using as the PROJECT_NUMBER? Code looks fine to me and I think the only additional permission I added on my test container was Secret Manager Secret Accessor. Just for reference here is my code that is definitely working on my test container: const dbURI = secret.readSecret("projects/tasklist-XXXXXX/secrets/dev-db-connectionstring/versions/latest").then((connectionString) => { mongoose.connect(connectionString); }); — Ville Rinne, Jun 18 '20 at 08:29
@VilleRinne Where are you getting `tasklist-XXXXXX` from? Mine is just numbers, no letters; perhaps I'm taking the wrong value? (except it's the same path as listed on the secrets web page) — Alex Meuer, Jun 18 '20 at 08:42
Tasklist-XXXXXX is just the full project identifier for my project. So when I created the project I entered tasklist as the name and Google appended the numbers at the end to make it unique. You can see the identifiers for every project in the console when you click to change the project and with gcloud. — Ville Rinne, Jun 18 '20 at 08:53
This is just project id, while in your case you use project number, it doesn't matter and should work fine with both. @VilleRinne please do not post project ids on public resources, use placeholders. — Emil Gi, Jun 18 '20 at 08:56
Yeah this is a test project that will get deleted in the next few days, but you are correct. — Ville Rinne, Jun 18 '20 at 09:00
Alright, I can see that the service account does indeed have the secret accessor role, and the code is correct. I don't have IAM persmissions myself so I'll have to reach out to the admin to see if they can spot something amiss. I'll update this post with an answer if I get it resolved. — Alex Meuer, Jun 18 '20 at 09:13
Have you tried removing and adding role to service account? Just to rule out [this](https://cloud.google.com/iam/docs/understanding-service-accounts#deleting_and_recreating_service_accounts) possibility. — Emil Gi, Jun 18 '20 at 09:25
The service account is a default and has never been deleted (heck, we could probably do with deleting some of our unused service accounts!). It did not have any secret manager permissions before the role was added, and did have them after. I'll talk to the admin and try it regardless. — Alex Meuer, Jun 18 '20 at 09:28
I realized that I am also using a customer service account on my test project, and I hav given it owner permissions. So maybe you could do one more test with really loose permissions to rule out that you aren't missing something? — Ville Rinne, Jun 18 '20 at 09:36
And by customer service account I of course meant custom :-) — Ville Rinne, Jun 18 '20 at 09:44
A service account with only this role should work fine. May be try creating a new service account, give it only this role and redeploy your function with this service account and try again. Maybe there is something wrong with the default one. — Emil Gi, Jun 18 '20 at 10:17
How are you deploying the function? Both project_id and project_number are acceptable for the secret name. — sethvargo, Jun 18 '20 at 11:57

Lin Du · Accepted Answer · 2022-08-11T02:40:11.830

HTTP cloud function code:

const { SecretManagerServiceClient } = require('@google-cloud/secret-manager');

const secretManagerServiceClient = new SecretManagerServiceClient();
const name = 'projects/shadowsocks-218808/secrets/workflow/versions/latest';

exports.testSecretManager = async (req, res) => {
  const [version] = await secretManagerServiceClient.accessSecretVersion({ name });
  const payload = version.payload.data.toString();
  console.debug(`Payload: ${payload}`);
  res.sendStatus(200);
};

Deploy:

gcloud functions deploy testSecretManager --runtime nodejs10 --trigger-http --allow-unauthenticated

Deploying function (may take a while - up to 2 minutes)...done.                                                                                                                                                                                                                        
availableMemoryMb: 256
entryPoint: testSecretManager
httpsTrigger:
  url: https://us-central1-shadowsocks-218808.cloudfunctions.net/testSecretManager
ingressSettings: ALLOW_ALL
labels:
  deployment-tool: cli-gcloud
name: projects/shadowsocks-218808/locations/us-central1/functions/testSecretManager
runtime: nodejs10
serviceAccountEmail: shadowsocks-218808@appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-43476143-b555-4cb2-8f6f-1b2d1952a2d7/42c4cda4-98a8-4994-a3be-d2203b9e646a.zip?GoogleAccessId=service-16536262744@gcf-admin-robot.iam.gserviceaccount.com&Expires=1596513795&Signature=kbLw5teN8EoYmj4fEweKKiIaakxcrhlUg2GGHV4jWJjvmeEfXePpRNOn9yz2zLn%2Fba0UqM9qdJMXujs5afBk%2BVBmywPEiptAZe2qgmldpr%2BsYejFu0woNgsPHVqtJ0NoWDo6W2dq4CuNNwO%2BaQ89mnhahUUQTInkJ55Y3wCIe9smk%2BqWtcvta3zICiToA7RQvPKY5MS6NViyj5mLxuJtDlTY9IKPL%2BqG6JAaQJSFYKYVgLyb6JfirXk8Q7%2FMvnHPpXPlhvsBLQksbF6jDPeefp2HyW4%2FSIQYprfpwKV3hlEIQyRQllz5J9yF83%2FxDPh%2BQPc5QmswKP5XAvYaszJPEw%3D%3D
status: ACTIVE
timeout: 60s
updateTime: '2020-08-04T03:34:32.665Z'
versionId: '2'

Test:

gcloud functions call testSecretManager --data '{}'

Got error same as you:

error: |-
  Error: function terminated. Recommended action: inspect logs for termination reason. Details:
  7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/shadowsocks-218808/secrets/workflow/versions/latest' (or it may not exist).

solution:

You can find the serviceAccountEmail: shadowsocks-218808@appspot.gserviceaccount.com from the deployment information details of cloud function.

go to IAM & Admin web UI, click ADD ANOTHER ROLE button, add Secret Manager Secret Accessor role to this service account.

Test again:

> gcloud functions call testSecretManager --data '{}'

executionId: 1tsatxl6fndw
result: OK

Read the logs for testSecretManager cloud function:

gcloud functions logs read testSecretManager

You will see the logs for the secret payload string.

For those looking today, you have to go under IAM & Admin > IAM, not the Service Accounts section. Wasted a huge amount of time trying to add permissions from under the Service Accounts section... they should really make this clearer. — arao6, Oct 11 '20 at 22:05
@arao6, you must answer this question with your comment. Your comment is the correct answer. Thanks! — Abner Escócio, Mar 18 '21 at 14:41
Thank you for the tip. I just added the Secret Manager Secret Accessor role to Editor under my App Engine default service account and it worked perfectly. — Steve Klock, May 11 '21 at 19:24

score 29 · Answer 2 · answered Aug 08 '20 at 20:29

29

I had the same issue and to solve it, I just had to:

Find the Service Account under General of my Google Cloud Function.

It looked like <project-name>@appspot.gserviceaccount.com
In IAM Admin, Add Secret Manager Secret Accessor Role to this Service Account.

After this, everything worked!

answered Aug 08 '20 at 20:29

binarytrails

516
4
14

1

was the key thing here. As I had many service accounts in there – M-sAnNan Sep 07 '20 at 11:40

score 15 · Answer 3 · answered Apr 30 '21 at 09:22

15

I have had similar issues working with secretmanager and the python google-cloud-secretmanager library (2.4). Specifically, after creating a secret and giving my service account the secretmanager.secretAccessor role on this secret (and nothing else, following the principle of least privilege), I was getting the following error when trying to access it:

details = "Permission 'secretmanager.versions.access' denied for resource 'projects/projectid/secrets/keyname/versions/latest' (or it may not exist)."

I could only make it work by also adding the secretmanager.viewer role at the project level, which as far as I can tell is not described in the documentation.

answered Apr 30 '21 at 09:22

Pascal Delange

417
5
17

7

OMG THANK YOU, finally, it took me so long to find out the accessor role is not enough – Patric Feb 19 '22 at 13:48
2

I'm struggling with the same. How did you add the role "at the project level"? – akauppi Jul 18 '22 at 08:56
2

When adding roles in GCP, you can typically add them on the whole project (you do this by going to the projects IAM page and giving the permission to the user/service account), or with more granularity on the ressource level (you do this by visiting the relevant ressource's page, here a Secret manager secret, and setting the permission there). Sometimes it's a bit more confusing and you need a mix of both project-wide and ressource-level permissions to make it work - as is the case here. – Pascal Delange Jul 18 '22 at 13:42
in my case i didn't need to do it at project level but did go into secret manager --- clicked into my secret --- went to permissions tab --- and added App Engine default service account the following permissions: (was already there) = Editor + 1) Secret Manager Secret Accessor + 2) Secret Manager Viewer = now works – Yev Guyduy Mar 14 '23 at 15:41

score 1 · Answer 4 · edited Jul 28 '20 at 06:47

A bit late, but maybe this answer could be useful for future users. I encountered the same behavior only with Python. I tried lots of things but only thing that worked was creating new service account with zero roles(if I granted it secretmanager.secretAccessor role immediately, I got the same error). Then when empty service account is created, in IAM tab I press +Add, copy my empty service account adress and ONLY then I add secretmanager.secretAccessor role to it. Then I use this account as the account that will execute particular function. You of course may need to add other roles depending on what your function is intended to accomplish.

Samuel · Answer 5 · 2021-06-17T10:34:29.787

I had similar problem using terraform under gitlab. I must add two authorizations to the service account which runs the pipeline:

resource "google_project_iam_policy" "gitlab" {
  project     = "secret_owner_project_id"
  policy_data = data.google_iam_policy.iam.policy_data
}

data "google_iam_policy" "iam" {
  binding {
    role = "roles/secretmanager.secretAccessor"
    members = [
      "serviceAccount:project_accessing_secret@XYZ.iam.gserviceaccount.com",
    ]
  }
  binding {
    role = "roles/viewer"
    members = [
      "serviceAccount:project_accessing_secret@XYZ.iam.gserviceaccount.com",
    ]
  }
}

score 0 · Answer 6 · answered Dec 30 '21 at 06:41

OAuth scope plays an important role here and please make sure the scope is defined correctly.

To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the cloud-platform OAuth scope. If you receive an error with the following message, it means the instance or node was not provisioned with the correct OAuth scopes.

Request had insufficient authentication scopes

The required OAuth scope to use Secret Manager is:

https://www.googleapis.com/auth/cloud-platform

Example gcloud command to create dataproc with scope

gcloud dataproc clusters create xyz-pqr --region asia-south1 --subnet projects/xyz-pqr/regions/asia-south1/subnetworks/abc-serverless-vpc --zone asia-south1-b --master-machine-type n1-standard-4 --master-boot-disk-size 100 --num-workers 2 --worker-machine-type n1-standard-4 --worker-boot-disk-size 100 --image-version 2.0-debian10 --project xyz-development -scopes https://www.googleapis.com/auth/cloud-platform

Walker · Answer 7 · 2023-08-04T23:57:01.173

I was having the same issue when running Go program and was getting error - rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/...' on my machine. It did not work by setting role "Secret Manager Secret Accessor" on my Compute Engine default service account that looks something like 66666666-compute@developer.gserviceaccount.com, nor editing permissions from within Secret Manager , side panel permissions.

What helped was glcoud secrets list and gcloud secrets get-iam-policy my-secret as mentioned in this document. I am assuming it probably refreshed something locally and I was able to get through without getting error. https://cloud.google.com/sdk/gcloud/reference/secrets/get-iam-policy

Also gcloud secrets get-iam-policy my-secret will give which service account you are using with that secret.

Update -

I had issuer again and this time the command below helped.

gcloud auth application-default login

Hope this helps someone.

Secret manager access denied despite correct roles for service account

7 Answers7