Writing figure to Google Cloud Storage instead of local drive

Question

I want to upload the figure which is made with matplotlib to GCS.

Current code:

from tensorflow.gfile import MakeDirs, Open
import numpy as np
import matplotlib.pyplot as plt
import datetime

_LOGDIR = "{date:%Y%m%d-%H%M%S}".format(date=datetime.datetime.now())

_PATH_LOGDIR = 'gs://{0}/logs/{1}'.format('skin_cancer_mnist', _LOGDIR)
MakeDirs(_PATH_LOGDIR))


def saving_figure(path_logdir):
    data = np.arange(0, 21, 2)
    fig = plt.figure(figsize=(20, 10))
    plt.plot(data)
    fig.savefig("{0}/accuracy_loss_graph.png".format(path_logdir))
    plt.close()

saving_figure(_PATH_LOGDIR)

"/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 512, in print_png filename_or_obj = open(filename_or_obj, 'wb')

FileNotFoundError: [Errno 2] No such file or directory: 'gs://skin_cancer_mnist/logs/20190116-195604/accuracy_loss_graph.png'

(The directory exists, I checked)

I could change the source code of matplotlib to use the Open method of tf.Gfile.Open, but there should be a better option...

score 4 · Answer 1 · answered Jun 04 '21 at 16:36

Joans 2nd Option didn't work for me, I found a solution that worked for me:

from google.cloud import storage
import io

def saving_figure(path_logdir):
    data = np.arange(0, 21, 2)
    fig = plt.figure(figsize=(20, 10))
    plt.plot(data)
    fig_to_upload = plt.gcf()

    # Save figure image to a bytes buffer
    buf = io.BytesIO()
    fig_to_upload.savefig(buf, format='png')

    # init GCS client and upload buffer contents
    client = storage.Client()
    bucket = client.get_bucket('skin_cancer_mnist')
    blob = bucket.blob('logs/20190116-195604/accuracy_loss_graph.png')  
    blob.upload_from_file(buf, content_type='image/png', rewind=True)

You don't use path_logdir inside the function. Is it 'logs/20190116-195604/accuracy_loss_graph.png'? — maxatSOflow, Mar 02 '22 at 14:36

Joan Grau Noël · Answer 2 · 2019-01-17T10:48:41.843

You cannot directly upload a file to Google Cloud Storage using the python open function (which is the one that matplotlib.pyplot.savefig is using behind the curtains). Instead, you should use the Cloud Storage Client Library for Python. Check this documentation for details on how this library is used. This will allow you to manipulate files and upload/download them to GCS, among other things.

You will have to import this library in order to use it, you can install it by running pip install google-cloud-storage and import it as from google.cloud import storage.

As well, since the plt.figure is an object, and not the actual .png image that you want to upload, you cannot directly upload it to Google Cloud Storage either.

However you can do either one of the following:

Option 1: Save the image locally, and then upload it to Google Cloud Storage:

Using your code:

from google.cloud import storage

def saving_figure(path_logdir):
    data = np.arange(0, 21, 2)
    fig = plt.figure(figsize=(20, 10))
    plt.plot(data)
    fig.savefig("your_local_path/accuracy_loss_graph.png".format(path_logdir))
    plt.close()


    # init GCS client and upload file
    client = storage.Client()
    bucket = client.get_bucket('skin_cancer_mnist')
    blob = bucket.blob('logs/20190116-195604/accuracy_loss_graph.png')  # This defines the path where the file will be stored in the bucket
    your_file_contents = blob.upload_from_filename(filename="your_local_path/accuracy_loss_graph.png")

Option 2: Save the image result from the figure to a variable, then upload it to GCS as a string (of bytes):

I have found the following StackOverflow answer that seems to save the figure image into a .png byte string, however I haven't tried it myself.

Again, based in your code:

from google.cloud import storage
import io
import urllib, base64

def saving_figure(path_logdir):
    data = np.arange(0, 21, 2)
    fig = plt.figure(figsize=(20, 10))
    plt.plot(data)
    fig_to_upload = plt.gcf()

    # Save figure image to a bytes buffer
    buf = io.BytesIO()
    fig_to_upload.savefig(buf, format='png')
    buf.seek(0)
    image_as_a_string = base64.b64encode(buf.read())

    # init GCS client and upload buffer contents
    client = storage.Client()
    bucket = client.get_bucket('skin_cancer_mnist')
    blob = bucket.blob('logs/20190116-195604/accuracy_loss_graph.png')  # This defines the path where the file will be stored in the bucket
    your_file_contents = blob.upload_from_string(image_as_a_string, content_type='image/png')

Edit: Both options assume that the environment you are running the script from, has the Cloud SDK installed, and a Google Cloud authenticated account activated (if you haven't, you can check this documentation that explains how to do it).

I think the key point is saving a file in memory, see also: https://stackoverflow.com/questions/8598673/how-to-save-a-pylab-figure-into-in-memory-file-which-can-be-read-into-pil-image — information_interchange, Aug 13 '20 at 09:27

Writing figure to Google Cloud Storage instead of local drive

2 Answers2