Python - Move files between GCP Buckets

Question

I'm trying to move csv files between two existing buckets on GCP. But I'm stuck on an 'Not Found' Error. Here is the code I'm using (This was built after reading GCP docs and SDK's):

from google.cloud import storage


def move_blob(bucket_name, blob_name, destination_bucket_name, destination_blob_name):
    """Moves a blob from one bucket to another with a new name."""
    storage_client = storage.Client()

    source_bucket = storage_client.bucket(bucket_name)
    source_blob = source_bucket.blob(blob_name)
    destination_bucket = storage_client.bucket(destination_bucket_name)

    blob_copy = source_bucket.copy_blob(
        source_blob, destination_bucket, destination_blob_name
    )

    print(
        "Blob {} in bucket {} moved to blob {} in bucket {}.".format(
            source_blob.name,
            source_bucket.name,
            blob_copy.name,
            destination_bucket.name,
        )
    ) 

def list_blobs():
    """Lists all the blobs in the bucket."""

    bucket_name = "BUCKET_1"
    destination_bucket_name = "BUCKET_2/folder"
    
    storage_client = storage.Client()
    
    blobs = storage_client.list_blobs(bucket_name)
    
    for blob in blobs:
        print(blob.name)
        
        destination_blob_name = blob_name = blob.name
        
        move_blob(bucket_name, blob_name, destination_bucket_name, destination_blob_name)

list_blobs()

And the content from the error message is:

'NotFound: 404 GET https://storage.googleapis.com/storage/v1/b/BUCKET_2/folder?projection=noAcl&prettyPrint=false: Not Found'

score 1 · Accepted Answer · answered Jun 03 '22 at 17:00

1

Your destination bucket name should just be the bucket name and should not have any prefixes. Just do the following: destination_bucket_name = "BUCKET_2" and add folder/ as a prefix to all blobs being transferred.

answered Jun 03 '22 at 17:00

Aishwary Shukla

450
1
7
21

1

Thanks man! It worked! I changed my code with your idea and I finally solved my issue. I changed the code to: blob_name = blob.name destination_blob_name = "folder/{}".format(blob.name) – Renan Saia Jun 07 '22 at 00:11

Andres Fiesco Casasola · Answer 2 · 2022-06-04T06:01:43.990

Here is a function I found (by dmlee8) that you can use when moving blobs between directories within the same bucket or to a different one:

from google.cloud import storage
import os
    
    os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="path_to_your_creds.json"
 
def mv_blob(bucket_name, blob_name, new_bucket_name, new_blob_name):
    """
    Function for moving files between directories or buckets . it will use GCP's copy 
    function then delete the blob from the old location.
    
    inputs
    -----
    bucket_name: name of bucket
    blob_name: str, name of file 
        ex. 'data/some_location/file_name'
    new_bucket_name: name of bucket (can be same as original if we're just moving around directories)
    new_blob_name: str, name of file in new directory in target bucket 
        ex. 'data/destination/file_name'
    """
    storage_client = storage.Client()
    source_bucket = storage_client.get_bucket(bucket_name)
    source_blob = source_bucket.blob(blob_name)
    destination_bucket = storage_client.get_bucket(new_bucket_name)
 
    # copy to new destination
    new_blob = source_bucket.copy_blob(
        source_blob, destination_bucket, new_blob_name)
    # delete in old destination
    source_blob.delete()
    
    print(f'File moved from {source_blob} to {new_blob_name}')

Python - Move files between GCP Buckets

2 Answers2