Copying and deleting files from google storage but getting python error as quote_from_bytes() expected bytes

Question

I am trying to implement a requirement where we are supposed to move the files from one particular directory in google cloud storage to another archive directory after the process is over so that the next day previous day files are not processed.

I looked at below post to implement a solution How to move files in Google Cloud Storage from one bucket to another bucket by Python

Google Cloud Storage - Move file from one folder to another - By using Python

I have written below python in cloud function

#!/usr/bin/python
# -*- coding: utf-8 -*-
from google.cloud import storage


def mv_blob(bucket_name,source_blobs):
    """Copies a blob from one bucket to another with a new name."""

    bucket_name = 'whr-asia-datalake-dev-standard'
    directory_name = 'outbound/Archive/'
    destination_bucket_name = 'whr-asia-datalake-dev-standard'

    storage_client = storage.Client()

    source_bucket = storage_client.bucket(bucket_name)
    source_blobs = source_bucket.list_blobs(prefix=directory_name)
    destination_bucket = storage_client.bucket(destination_bucket_name)

    for blob in source_blobs:
        blob_copy = source_bucket.copy_blob(
        blob, destination_bucket, blob
        )
        blob.delete()
        print(
        "Blob {} in bucket {} copied to blob {} in bucket {}.".format(
            source_blob.name,
            source_bucket.name,
            blob_copy.name,
            destination_bucket.name,

) )

However, im getting " quote_from_bytes() expected bytes" below error in python, please see execution details

File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 449, in run_background_function
    _function_handler.invoke_user_function(event_object)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 268, in invoke_user_function
    return call_user_function(request_or_event)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 265, in call_user_function
    event_context.Context(**request_or_event.context))
  File "/user_code/main.py", line 21, in mv_blob
    blob, destination_bucket, blob
  File "/env/local/lib/python3.7/site-packages/google/cloud/storage/bucket.py", line 487, in copy_blob
    api_path = blob.path + '/copyTo' + new_blob.path
  File "/env/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 171, in path
    return self.path_helper(self.bucket.path, self.name)
  File "/env/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 146, in path_helper
    return bucket_path + '/o/' + quote(blob_name, safe='')
  File "/opt/python3.7/lib/python3.7/urllib/parse.py", line 834, in quote
    return quote_from_bytes(string, safe)
  File "/opt/python3.7/lib/python3.7/urllib/parse.py", line 859, in quote_from_bytes
    raise TypeError("quote_from_bytes() expected bytes")
TypeError: quote_from_bytes() expected bytes

The delete part works fine but as soon as I add below in my function it gives this issue

blob_copy = source_bucket.copy_blob(
        blob, destination_bucket, blob
        )

Can someone please help to solve this? Appreciate your inputs and help on this,

score 2 · Accepted Answer · answered Feb 22 '21 at 08:26

Try to print the blob and you will understand quickly! In fact, in Cloud Storage, there isn't directory. You have only blobs; and their names in the fully path (path/to/file).

With this context, compile your code with your brain

blob_copy = source_bucket.copy_blob(
        blob, destination_bucket, blob
        )

From the source bucket whr-asia-datalake-dev-standard copy the blob outbound/Archive/XXX to the bucket destination whr-asia-datalake-dev-standard with the blob name outbound/Archive/XXX

You should feel the issue: the source and the destination are the same!

Thank you @guillaume blaquiere. I realized my mistake after posting this question and didn't post the answer as I got into another issue which took all my attention.:-) I was able to fix it and its all working fine now. I appreciate you looking into it. Thank you for your time and all help you provide. — radhika sharma, Feb 23 '21 at 04:30

Copying and deleting files from google storage but getting python error as quote_from_bytes() expected bytes

1 Answers1