23

This is my working script that generates a plot, saves it locally to disk, uploads to S3 and deletes the file:

plt.figure(figsize=(6,6))
plt.plot(x, y, 'bo')
plt.savefig('file_location')

conn = boto.s3.connect_to_region(
    region_name=AWS_REGION,
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    calling_format=boto.s3.connection.OrdinaryCallingFormat()
    )
bucket = conn.get_bucket('bucket_name')
k = Key(bucket)
k.key = 'file_name'
k.set_contents_from_filename('file_location')

os.remove(file_location)

What I want is to skip the disk writing and upload the plot directly from memory.

Any suggestions how to achieve that?

GuyB7
  • 606
  • 1
  • 6
  • 15

2 Answers2

33

Putting it all together:

img_data = io.BytesIO()
plt.savefig(img_data, format='png')
img_data.seek(0)

s3 = boto3.resource('s3')
bucket = s3.Bucket(BUCKET_NAME)
bucket.put_object(Body=img_data, ContentType='image/png', Key=KEY)

Thanks @padraic-cunningham and @guyb7 for the tips!

Aidan Feldman
  • 5,205
  • 36
  • 46
  • 7
    the uploaded image turns up empty for me :( but i'm trying to upload an actual image, not plt graph ... would that make a difference? – Raksha Apr 05 '19 at 18:28
  • How do I read the image back? I tried img=skio.imread(io.BytesIO(obj.get()['Body'].read())) but I get an error:"Could not find a format to read the specified file in mode 'i'" – Supamee May 23 '19 at 01:56
  • @Raksha try the following: replace plt.savefig(img_data, format='png') with plt.imsave(img_data, my_img) where my_img is your image data. – Selman Tunc Yilmaz Aug 03 '19 at 21:35
  • If the uploaded image turns out empty as @Raksha mentioned. Check this answer https://stackoverflow.com/questions/9012487/matplotlib-pyplot-savefig-outputs-blank-image – Miguel Trejo Jul 31 '20 at 17:51
  • Does anyone know why this solution crop the plot when I open it using AWS signedURL? – Niv Cohen Oct 26 '20 at 23:04
  • In reply to my own comment case anyone else need it... just use these 2 parameters bbox_inches, pad_inches plt.savefig(img_data, format='png', bbox_inches='tight', pad_inches=0.5) – Niv Cohen Oct 27 '20 at 05:55
  • Befer saving figure make sure img_data is filled with matplotlib object ! put plt.imshow(matplotlib_object_to_draw) or plt.plot(things to plot) before savefig else output will be empty white image – Gorkem Nov 06 '20 at 11:29
5
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.backends.backend_agg import FigureCanvasAgg
import boto3
import io

# some random plotting. We need the figure object later
fig, ax = plt.subplots(1,1,figsize=(6,6))
ax.plot(np.linspace(0,1,50),
        np.random.normal(0.5,0.5,50))


canvas = FigureCanvas(fig) # renders figure onto canvas
imdata = io.BytesIO() # prepares in-memory binary stream buffer (think of this as a txt file but purely in memory)
canvas.print_png(imdata) # writes canvas object as a png file to the buffer. You can also use print_jpg, alternatively

s3 = boto3.resource('s3',
                    aws_access_key_id='your access key id',
                    aws_secret_access_key='your secret access key',
                    region_name='us-east-1') # or whatever region your s3 is in

s3.Object('yourbucket','picture.png').put(Body=imdata.getvalue(),
                                          ContentType='image/png') 
# this makes a new object in the bucket and puts the file in the bucket
# ContentType parameter makes sure resulting object is of a 'image/png' type and not a downloadable 'binary/octet-stream'

s3.ObjectAcl('yourbucket','picture.png').put(ACL='public-read')
# include this last line if you find the url for the image to be inaccessible
Matthew K
  • 189
  • 2
  • 12