Stack Overflow
Stack Overflow

Questions tagged [gsutil]

gsutil is a command-line program that allows you to store, share and manage data hosted at cloud-based storage providers.

gsutil allows you to store, share and manage data hosted at cloud-based storage providers. It provides a simple Unix shell command-like syntax, operating over buckets and objects named with a URIs and wildcards.

868 questions
102
votes
19 answers

gsutil ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket even though I'm loggedin in gcloud

I am trying to create an internal app to upload files to google cloud. I don't want each individual user or this app to log in so I'm using a service account. I login into the service account and everything is ok, but when I try to upload it gives…
Daniel Tranca
  • 1,374
  • 3
  • 13
  • 23
70
votes
7 answers

How to count number of file in a bucket-folder with gsutil

Is there an option to count the number of files in bucket-folders? Like: gsutil ls -count -recursive gs://bucket/folder Result: 666 files I just want an total number of files to compare the amount to the sync-folder on my server. I don't get…
user2811846
  • 703
  • 1
  • 5
  • 4
62
votes
10 answers

Fastest way to get Google Storage bucket size?

I'm currently doing this, but it's VERY slow since I have several terabytes of data in the bucket: gsutil du -sh gs://my-bucket-1/ And the same for a sub-folder: gsutil du -sh gs://my-bucket-1/folder Is it possible to somehow obtain the total size…
fredrik
  • 9,631
  • 16
  • 72
  • 132
44
votes
2 answers

gsutil / gcloud storage file listing sorted date descending?

Is there no way to get a file listing out from a Google Cloud Storage bucket that is sorted by date descending? This is very frustrating. I need to check the status of files that are uploaded and the bucket has thousands of objects. gsutil ls does…
Dhiraj Gupta
  • 9,704
  • 8
  • 49
  • 54
42
votes
4 answers

How can I use gsutil with multiple accounts?

I frequently use Google Cloud Storage with at least two accounts: personal@gmail.com and work@corp.com. I used gsutil config to create .boto files for both accounts, which I've renamed to personal.boto and work.boto. It is tiring to have to…
fejta
  • 3,061
  • 2
  • 17
  • 22
39
votes
3 answers

gsutil returning "no matches found"

I'm trying using gsutil to remove the contents of a Cloud Storage bucket (but not the bucket itself). According to the documentation, the command should be: gsutil rm gs://bucket/** However, whenever I run that (with my bucket name substituted of…
JohnGB
  • 1,906
  • 2
  • 20
  • 31
39
votes
7 answers

ResumableUploadAbortException: 403 Insufficient Permission

I am using Google Cloud Storage and Google Compute Engine and am trying to upload to my bucket from my VM. My bucket has read/write permissions and so does anyone that has access to it. They are labeled as Owner. I can download from my bucket fine.…
superuserdo
  • 1,637
  • 3
  • 21
  • 33
26
votes
4 answers

What is the difference between gcloud and gsutil?

I want to know difference between gcloud and gsuitl. Where do we use what? Why certain commands begin with gsutil while others with gcloud?
Kshitij Singh
  • 411
  • 1
  • 4
  • 7
26
votes
5 answers

Google GSutil create folder

How can u create a new folder inside a bucket in google cloud storage using the gsutil command? I tried using the same command in creating bucket but still got an error gsutil mb -l us-east1 gs://my-awesome-bucket/new_folder/ Thanks!
Bongsky
  • 493
  • 3
  • 12
  • 23
25
votes
6 answers

Gsutil - How can I check if a file exists in a GCS bucket (a sub-directory) using Gsutil

I have a GCS bucket containing some files in the path gs://main-bucket/sub-directory-bucket/object1.gz I would like to programmatically check if the sub-directory bucket contains one specific file. I would like to do this using gsutil. How could…
activelearner
  • 7,055
  • 20
  • 53
  • 94
22
votes
3 answers

Google Storage ArgumentException when setting CORS config

Following Google docs, when using GoogleCloud Storage console: mysef@myproject:~$ cat cors-json-file.json [ { "origin": ["*"], "method": ["GET"], "maxAgeSeconds": 3600 } ] then I get the following error: myself@myproject:~$…
comte
  • 3,092
  • 5
  • 25
  • 41
20
votes
3 answers

How to do gsutil cp -R while ignoring files like .git, .gitignore?

I am trying to automate the process of syncing my web assets with Google Cloud Storage. I basically need to copy everything in my development directory up to the cloud. However, I need to ignore the .git directory and some other irrelevant files. I…
Shahbaz
  • 10,395
  • 21
  • 54
  • 83
19
votes
2 answers

Rename a folder in GCS using gsutil

I can't rename an existing folder in GCS. How do I do this? As per the documentation, this should be: gsutil mv gs://my_bucket/olddir gs://my_bucket/newdir However, what happens is that olddir is placed under newdir, i.e the directory structure is…
knub
  • 3,892
  • 8
  • 38
  • 63
17
votes
3 answers

How to copy folders with 'gsutil'?

I've read the documentation on the gsutil cp command, but still don't understand how to copy folders to keep the same permissions. I tried this command: gsutil cp gs://bucket-name/folder1/folder_to_copy gs://bucket-name/folder1/new_folder But it…
Praytic
  • 1,771
  • 4
  • 21
  • 41
17
votes
2 answers

Securing GAE env variables by using gsutil builder to source app.yaml during build?

I have the same problem as the one mentioned here: Securely storing environment variables in GAE with app.yaml - namely: "I need to store API keys and other sensitive information in app.yaml as environment variables for deployment on GAE. The issue…
Andrew Bass
  • 336
  • 2
  • 8
1
2 3
…
57 58