gsutil copy returning "AccessDeniedException: 403 Insufficient Permission" from GCE

Question

I am logged in to a GCE instance via SSH. From there I would like to access the Storage with the help of a Service Account:

GCE> gcloud auth list
Credentialed accounts:
 - 1234567890-compute@developer.gserviceaccount.com (active)

I first made sure that this Service account is flagged "Can edit" in the permissions of the project I am working in. I also made sure to give him the Write ACL on the bucket I would like him to copy a file:

local> gsutil acl ch -u 1234567890-compute@developer.gserviceaccount.com:W gs://mybucket

But then the following command fails:

GCE> gsutil cp test.txt gs://mybucket/logs

(I also made sure that "logs" is created under "mybucket").

The error message I get is:

Copying file://test.txt [Content-Type=text/plain]...
AccessDeniedException: 403 Insufficient Permission               0 B

What am I missing?

Was the GCE VM created with full control or read/write GCS scope? — jterrace, Dec 03 '14 at 17:09
Thank you for pointing that out. I was not aware of that option indeed. I re-created the instance with the option enabled and it worked. If you could suggest to turn on the flag as an answer I would happily flag it. — Christophe, Dec 04 '14 at 09:54

score 188 · Accepted Answer · edited Dec 27 '22 at 03:15

One other thing to look for is to make sure you set up the appropriate scopes when creating the GCE VM. Even if a VM has a service account attached, it must be assigned devstorage scopes in order to access GCS.

For example, if you had created your VM with devstorage.read_only scope, trying to write to a bucket would fail, even if your service account has permission to write to the bucket. You would need devstorage.full_control or devstorage.read_write.

See the section on Preparing an instance to use service accounts for details.

Note: the default compute service account has very limited scopes (including having read-only to GCS). This is done because the default service account has Project Editor IAM permissions. If you use any user service account this is not typically a problem since user created service accounts get all scope access by default.

After adding necessary scopes to the VM, gsutil may still be using cached credentials which don't have the new scopes. Delete ~/.gsutil before trying the gsutil commands again. (Thanks to @mndrix for pointing this out in the comments.)

As of now, you can edit the scopes. Stop the machine - edit - then change the Cloud API access scopes. I believe this has only be available for about a month now. — Warren, Dec 13 '16 at 00:08

score 65 · Answer 2 · answered Feb 16 '17 at 11:33

65

You have to log in with an account that has the permissions you need for that project:

gcloud auth login

answered Feb 16 '17 at 11:33

Javier Giovannini

2,302
1
19
21

4

Do not forget to issue `gcloud auth revoke ` after you are finished. – ssasa Apr 12 '19 at 08:06

score 56 · Answer 3 · edited Aug 04 '19 at 18:25

56

gsutil config -b

Then surf to the URL it provides, [ CLICK Allow ]

Then copy the verification code and paste to terminal.

edited Aug 04 '19 at 18:25

Uli Köhler

13,012
16
70
120

answered Dec 19 '16 at 23:47

Dr. Tyrell

2,829
1
16
9

Note this doesn’t work if you run `gsutil` from `ssh`. – bfontaine Dec 30 '16 at 14:17
3

It "doesn't work" in so far as not opening your browser. Instead, it provides a url for you to manually copy and paste. – BuvinJ Sep 08 '17 at 19:34
6

This also works fine for everyone else _if you drop the `-b`_. That will not open a browser, but will simply spit out a url that can be opened outside of the shell. – oligofren Mar 23 '18 at 13:36
Nice - This is the first solution I have found which does not require restarting the VM (and thereby potentially losing data). – Jonathan Dec 17 '21 at 20:11
This solution worked for me without restart. Thank you!! – Ravi Feb 25 '22 at 19:06

Omar Bara · Answer 4 · 2021-12-07T21:51:49.297

30

Stop VM
goto --> VM instance details.
in "Cloud API access scopes" select "Allow full access to all Cloud APIs" then Click "save".
restart VM and Delete ~/.gsutil .

edited Dec 07 '21 at 21:51

answered Mar 26 '20 at 23:45

Omar Bara

409
4
5

1

You need to stop the instance before editing any permissions – kukroid Sep 09 '21 at 20:32
3

Thank you, this one worked perfectly for me on 2 Feb 2022. Looks like the best up to date answer. – TC Arlen Feb 03 '22 at 16:14
Excellent, worked in May 2022 – sapo_cosmico May 26 '22 at 22:06

score 13 · Answer 5 · edited Apr 22 '18 at 11:47

13

I have written an answer to this question since I can not post comments:

This error can also occur if you're running the gsutil command with a sudo prefix in some cases.

edited Apr 22 '18 at 11:47

Rann Lifshitz

4,040
4
22
42

answered Apr 22 '18 at 00:45

TheLoneDeranger

1,161
9
13

3

This is really a comment, not an answer. With a bit more rep, [you will be able to post comments](//stackoverflow.com/privileges/comment). – Lece Apr 22 '18 at 01:05
1

I have given the answer (+1) so that @TheLoneDeranger will be closer in reputation to the 'Post comment' privilege. – Rann Lifshitz Apr 22 '18 at 01:23
this saved my life! – Haris Hasan Dec 01 '21 at 19:23

score 4 · Answer 6 · answered Dec 06 '18 at 17:48

After you have created the bucket, go to the permissions tab and add your email and set Storage Admin permission.

Access VM instance via SSH >> run command: gcloud auth login and follow the steps.

Ref: https://groups.google.com/d/msg/gce-discussion/0L6sLRjX8kg/kP47FklzBgAJ

Yev Guyduy · Answer 7 · 2019-11-02T17:46:18.773

So I tried a bunch of things trying to copy from GCS bucket to my VM. Hope this post helps someone.

Via SSHed connection:

and following this script:

sudo gsutil cp gs://[BUCKET_NAME]/[OBJECT_NAME] [OBJECT_DESTINATION_IN_LOCAL]

Got this error:

AccessDeniedException: 403 Access Not Configured. Please go to the Google Cloud Platform Console (https://cloud.google.com/console#/project) for your project, select APIs and Auth and enable the Google Cloud Storage JSON API.

What fixed this was following "Activating the API" section mentioned in this link - https://cloud.google.com/storage/docs/json_api/

Once I activated the API then I authenticated myself in SSHed window via

gcloud auth login

Following authentication procedure I was finally able to download from Google Storage Bucket to my VM.

PS

I did make sure to:

Make sure that gsutils are installed on my VM instance.
Go to my bucket, go to the permissions tab and add desired service accounts and set Storage Admin permission / role.

3.Make sure my VM had proper Cloud API access scopes:

score 2 · Answer 8 · answered Jul 11 '19 at 10:20

From the docs: https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes

You need to first stop the instance -> go to edit page -> go to "Cloud API access scopes" and choose "storage full access or read/write or whatever you need it for"

Changing the service account and access scopes for an instance If you want to run the VM as a different identity, or you determine that the instance needs a different set of scopes to call the required APIs, you can change the service account and the access scopes of an existing instance. For example, you can change access scopes to grant access to a new API, or change an instance so that it runs as a service account that you created, instead of the Compute Engine Default Service Account.

To change an instance's service account and access scopes, the instance must be temporarily stopped. To stop your instance, read the documentation for Stopping an instance. After changing the service account or access scopes, remember to restart the instance. Use one of the following methods to the change service account or access scopes of the stopped instance.

1

Still valid in 2021 – derekbaker783 Mar 07 '21 at 15:45

score -10 · Answer 9 · edited Aug 25 '20 at 14:47

-10

Change the permissions of bucket.

Add a user for "All User" and give "Storage Admin" access.

edited Aug 25 '20 at 14:47

Lamanus

12,898
4
21
47

answered Aug 25 '20 at 12:00

Pratik Waware

1

5

This will give write access to everybody to the bucket. Pretty sure you don't want to do that :) – LachoTomov Mar 11 '21 at 16:42

gsutil copy returning "AccessDeniedException: 403 Insufficient Permission" from GCE

9 Answers9

Linked