19

I can't rename an existing folder in GCS. How do I do this?

As per the documentation, this should be:

gsutil mv gs://my_bucket/olddir gs://my_bucket/newdir

However, what happens is that olddir is placed under newdir, i.e the directory structure is like this (after the call to gsutil mv):

my_bucket
    newdir
        olddir

instead of (what I would expect)

my_bucket
    newdir

I've tried all four combinations of putting trailing slashes or not, but none of them worked.

knub
  • 3,892
  • 8
  • 38
  • 63
  • What is the output from the ```gsutil``` command when you run it and end up with the unwanted nested folder structure? Normally you would see a Copying... and Removing... line recursively for all the files contained under your ```olddir```. – yoape Aug 20 '18 at 11:59
  • Possible duplicate of: https://stackoverflow.com/questions/45896712/how-to-rename-an-object-in-google-storage-bucket ...which confirms my answer - while I even provided a viable solution approach to the problem, instead of just telling "it's not possible". – Martin Zeitler Aug 20 '18 at 12:11
  • @yoape: Yes, I see lots of "Copying gs://[..]" and "Removing gs://[..]" lines. – knub Aug 20 '18 at 12:50
  • Generally speaking, in object storage systems like GS and AWS S3, a folder is not a real first-class item the way a file is. GS seems to have fixed this bug, but in object management you should design around files, not folders. – Jack Parsons Feb 19 '20 at 21:11

2 Answers2

20

This is a confirmed bug in GCS, see https://issuetracker.google.com/issues/112817360

It actually only happens, when the directory name of newdir is a substring of olddir. So the gsutil call from the question actually works, but the following one would not:

gsutil mv gs://my-organization-empty-bucket/dir_old gs://my-organization-empty-bucket/dir
knub
  • 3,892
  • 8
  • 38
  • 63
2

I reproduced your case by having a bucket with a folder named olddir of which I want to move the content to newdir folder.

the following command:

 gsutils mv gs://<bucketname>/olddir gs://<bucketname>/newdir

moved the whole content of folder to the newly created newdir folder.

Olddir and newdir folders were then at the same level, in the bucket root. after that I just had to remove the folder called olddir.

Objects in a bucket cannot be renamed.

The gsutil mv command does not remove the previous folder object like the mv comand would do in Unix CLI.

I guess that if you have tried moving folders several times by using "/" characters placed differently, the structure and hierarchy of the folders will have changed after issuing the initial command.

Please try again from the beginning.

Bear in mind that once you have a subfolder inside a folder, objects will have to be moved one by one using the full path.

Tzigan
  • 155
  • 6
  • I double-checked everything (in a new empty bucket, with new folders/files) and I still see the wrong behaviour. This is very odd, seems not like a usage problem, but like a bug in GCS. – knub Aug 20 '18 at 13:07
  • https://pastebin.com/iksntwwE In case you're interested in the actual commands I used. – knub Aug 20 '18 at 13:17
  • It is because in your case the folder gs://my-organization-empty-bucket/test_data already exist when you run the mv command, so it copies the data_b folder inside the folder named test_data. If the destination test_data folder would not exist it would create it and move the content of data_b folder inside the test_data folder. – Tzigan Aug 20 '18 at 13:42
  • According to the script output: gs://my-organization-empty-bucket/test_data/text.txt $ gsutil rm -r gs://my-organization-empty-bucket/test_data Removing gs://my-organization-empty-bucket/test_data/text.txt#1534770322877415... / [1 objects] Operation completed over 1 objects. the file inside the folder was deleted but not the folder itself. when I reproduce on my side I have operation completed on 2 objects, the file and the folder. – Tzigan Aug 20 '18 at 13:49
  • The concept of a folder does not exist in GCS. When the last file is deleted, it also deletes the folder. Also, I'm calling `rm -r` on the folder. That aside, the bug also happens if I don't create the directory before. Apparently it's enough if the new directory name is a substring of the old directory name ... – knub Aug 20 '18 at 13:53
  • @knub Hi! I'm with GCP Support. I saw that you posted in the Public Issue Tracker. Any comments on the issue will be posted over there. Feel free to, later on, post anything from there in this question, for future reference of the community :) – Mangu Aug 21 '18 at 15:03