How to create text file in Google Cloud Storage without saving it locally?

Question

I know how to upload a string saved to a text file to Google Cloud Storage: using the upload_blob function below (source):

from google.cloud import storage

def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"
    # The path to your file to upload
    # source_file_name = "local/path/to/file"
    # The ID of your GCS object
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    blob.upload_from_filename(source_file_name)

I can create a file stored on local disk:

!touch localfile
!echo "contents of my file" > localfile
!cat localfile  # outputs: contents of my file

Upload this file to Google Cloud Storage:

upload_blob('my-project','localfile','gcsfile')

It is indeed uploaded:

How can I create gcsfile in Google Cloud Storage containing the string contents of my file, without saving it first?

I tried:

import io

output = io.BytesIO()
output.write(b'First line.\n')

upload_blob('adventdalen-003',output,'out')

Doesn't work, I get:

TypeError: expected str, bytes or os.PathLike object, not _io.BytesIO

Similar but different threads:

Neither of these are in Python.

The source says it wants a file. Why can't you create a local, temporary file? — OneCricketeer, Mar 06 '22 at 15:45
I would like to do this from Google Cloud Functions, and I prefer not messing about with files there. — zabop, Mar 06 '22 at 15:55
If you convert the BytesIO into an actual bytes object, like the error says it expects, then what error do you get? — OneCricketeer, Mar 06 '22 at 16:06
Also, you seem to be ignoring the `upload_blob_from_memory` function... Why? — OneCricketeer, Mar 06 '22 at 16:07
First question: if I replace `output` with `output.read()`, as suggested by [this](https://stackoverflow.com/a/54137810/8565438), I get `TypeError: cannot use a string pattern on a bytes-like object`. Second question: because I didn't know about it, but thanks! I think this will solve my problem. — zabop, Mar 06 '22 at 16:12
Consult the documentation to review the methods available. https://googleapis.dev/python/storage/latest/blobs.html For example **blob.upload_from_string()** Upload contents of this blob from the provided string. — John Hanley, Mar 06 '22 at 20:31

score 7 · Accepted Answer · edited Mar 07 '22 at 04:05

Using @johnhanley's suggestions this is the code to implement blob.upload_from_string():

from google.cloud import storage

def write_to_blob(bucket_name,file_name):
    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(file_name)
    blob.upload_from_string("written in python")

write_to_blob(bucket_name="test-bucket",file_name="from_string.txt")

Saved in Google Cloud Storage:

Inside from_string.txt:

How to create text file in Google Cloud Storage without saving it locally?

1 Answers1

Linked