How to upload & get custom metadata with gcloud-aio-storage?

Question

I am using gcloud-aio-storage to cache a file. There is an option to add metadata dict in upload function

async def upload(self, bucket: str, object_name: str, file_data: Any,
                 *, content_type: Optional[str] = None,
                 parameters: Optional[Dict[str, str]] = None,
                 headers: Optional[Dict[str, str]] = None,
                 metadata: Optional[Dict[str, Any]])

There is also a function to download metadata async def download_metadata(self, bucket: str, object_name: str,

One should be able to add metadata and retrieve it later using the above functions. What am I doing wrong?

Donnald Cucharo · Accepted Answer · 2021-07-06T04:20:22.067

From my understanding, you wanted to upload an object with custom metadata and then retrieve it once needed.

The custom metadata won't be added if you pass your key-value pairs into upload() like this:

metadata_dict = {
  "foo" : "bar",
  "color": "green"
}

The reason is because the library is looking for the metadata key within the dictionary. See code snippet here. I reviewed the library and in this case, the request body becomes:

{"foo": "bar", "color": "green", "name": "filename.txt"}

Where the correct one must be:

{"metadata": {"foo": "bar", "color": "green"}, "name": "filename.txt"}

The solution is to configure your dictionary like this:

meta_dict = {
    "metadata":{
      "foo" : "bar",
      "color": "green"
    }
}

Full code example (credits to the original answer):

import asyncio
import aiohttp
# pip install aiofile
from aiofile import AIOFile
# pip install gcloud-aio-storage
from gcloud.aio.storage import Storage 
import logging
import sys

BUCKET_NAME = 'BUCKET_NAME'
FILE_NAME  = 'FILE_NAME.txt'
meta_dict = {
    "metadata":{
      "foo" : "bar",
      "color": "green"
    }
}
async def async_upload_to_bucket(blob_name, file_obj, meta_dict):
    async with aiohttp.ClientSession() as session:
        storage = Storage(session=session) 
        status = await storage.upload(BUCKET_NAME, blob_name, file_obj, metadata=meta_dict)
        print(status['metadata'])
        return status['selfLink']

async def main():
    async with AIOFile(FILE_NAME, mode='r') as afp:
        f = await afp.read()
        url = await async_upload_to_bucket(FILE_NAME, f, meta_dict)
        print(url)


# Python 3.7+
asyncio.run(main()) 
print("success")

How to upload & get custom metadata with gcloud-aio-storage?

1 Answers1