Missing custom headers in Google Cloud Storage

Question

I've created Standard Storaget Type bucket in my GCS console with public access restriction
I've put a little file inside this bucket and add custom metadata ("name":"jack") using GUI
I've created simple Python script for downloading this file:

from pathlib import Path import requests

def url_retrieve(url: str, outfile: Path):
    R = requests.get(url, allow_redirects=True)
    if R.status_code != 200:
        raise ConnectionError('could not download {}\nerror code: {}'.format(url, R.status_code))

    print(R.history)
    print(R.headers)
    print(R.status_code)
    outfile.write_bytes(R.content)

url_retrieve(url, path)

I pass url from "Authenticated URL" field in object inside GCS Console GUI.

Problem: There is no my custom metadata inside headers. What am i doing wrong? :)

Example Response Headers:

Content-Type: text/html; charset=utf-8 X-Frame-Options: DENY Set-Cookie: __Host-GAPS=1:YbfLYYNRk-YoqA4WaishOZQsoWZyKg:Mgn4RZDSaVcAMUG4; Expires=Wed, 07-May-2025 08:51:52 GMT; Path=/; Secure; HttpOnly; Priority=HIGH x-auto-login: realm=com.google&args=service%3Dcds%26continue%3Dhttps://storage.cloud.google.com/kravtsov-test-bucket/tox_01.tox x-ua-compatible: IE=edge Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache Expires: Mon, 01 Jan 1990 00:00:00 GMT Date: Mon, 08 May 2023 08:51:52 GMT Strict-Transport-Security: max-age=31536000; includeSubDomains Cross-Origin-Opener-Policy-Report-Only: same-origin; report-to="AccountsSignInUi" Content-Security-Policy: script-src 'report-sample' 'nonce-4gxVLMlbGh5kkUnB0MDL4g' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /v3/signin//AccountsSignInUi/cspreport;worker-src 'self' Content-Security-Policy: require-trusted-types-for 'script';report-uri /v3/signin//AccountsSignInUi/cspreport Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version Cross-Origin-Resource-Policy: same-site Report-To: {"group":"AccountsSignInUi","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/AccountsSignInUi"}]} Permissions-Policy: ch-ua-arch=, ch-ua-bitness=, ch-ua-full-version=, ch-ua-full-version-list=, ch-ua-model=, ch-ua-wow64=, ch-ua-platform=, ch-ua-platform-version= Server: ESF X-XSS-Protection: 0 X-Content-Type-Options: nosniff Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Accept-Ranges: none Vary: Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site,Accept-Encoding Connection: close Transfer-Encoding: chunked

What is the header and what headers are being returned? Edit your post with those details. — John Hanley, May 08 '23 at 07:36
Please format the update correctly. One big line is hard to read. — John Hanley, May 08 '23 at 07:45
Do you mean that you set custom metadata on the object: https://cloud.google.com/storage/docs/metadata#custom-metadata — John Hanley, May 08 '23 at 07:45
Have a look at this stackoverflow [link](https://stackoverflow.com/questions/56366142/how-to-access-file-metadata-for-files-in-google-cloud-storage-from-a-python-go) & [blog](https://download.huihoo.com/google/gdgdevkit/DVD1/developers.google.com/storage/docs/gspythonlibrary.html) — Sathi Aiswarya, May 08 '23 at 10:21

score 0 · Accepted Answer · answered May 09 '23 at 05:14

Finally, I've figured it out - thanks to Sathi Aiswarya for pointing me right direction.

So, you won't get any object metadata headers using Authenticated URL from outside of Google Infrastructure. To get those headers you have 2 options:

Make your bucket/object public and use Public URL.
Generate signed URL for your object.

Script for generating such URL described here - https://cloud.google.com/storage/docs/access-control/signing-urls-manually. I've used it AS IS and it worked.

For this script you also have to generate private key for your service account. This key is passed as service_account_file parameter in script.

Process of private key creation is described here - https://cloud.google.com/iam/docs/keys-create-delete

Missing custom headers in Google Cloud Storage

1 Answers1