How to attach large files to an email using Python - Gmail API

Question

I'm trying to send an email with an attachment (ideally multiple attachments) that are larger than 10 MB and smaller than the limit of 25 MB in total. The reason I mention 10 MB is because it seems to be the lower bound for when the normal way of attaching files stops working and you get Error 10053.

I've read in the documentation that the best way to do this would be by using the resumable upload method but I haven't been able to get it to work nor have I been able to find any good examples in Python. Most of the SO questions on this simply link back to the documentation which doesn't have a Python example or their code resulted in other errors.

I'm looking for an explanation in Python because I want to make sure I understand it correctly.

Questions I've looked through:

Code:

import base64
import json
import os
from email import utils, encoders
from email.message import EmailMessage
from email.mime import application, multipart, text, base, image, audio
import mimetypes

from apiclient import errors
from googleapiclient import discovery, http
from google.oauth2 import service_account

def send_email(email_subject, email_body, email_sender='my_service_account@gmail.com', email_to='', email_cc='', email_bcc='', files=None):

    # Getting credentials
    with open(os.environ.get('SERVICE_KEY_PASSWORD')) as f:
        service_account_info = json.loads(f.read())

    # Define which scopes we're trying to access
    SCOPES = ['https://www.googleapis.com/auth/gmail.send']

    # Setting up credentials using the gmail api
    credentials = service_account.Credentials.from_service_account_info(service_account_info, scopes=SCOPES)

    # This allows us to assign an alias account to the message so that the messages aren't coming from 'ServiceDriod-8328balh blah blah'
    delegated_credentials = credentials.with_subject(email_sender)

    # 'Building' the service instance using the credentials we've passed
    service = discovery.build(serviceName='gmail', version='v1', credentials=delegated_credentials)

    # Building out the email 
    message = multipart.MIMEMultipart()
    message['to'] = email_to
    message['from'] = email_sender
    message['date'] = utils.formatdate(localtime=True)
    message['subject'] = email_subject
    message['cc'] = email_cc
    message['bcc'] = email_bcc
    message.attach(text.MIMEText(email_body, 'html'))


    for f in files or []:
        mimetype, encoding = mimetypes.guess_type(f)

        # If the extension is not recognized it will return: (None, None)
        # If it's an .mp3, it will return: (audio/mp3, None) (None is for the encoding)
        # For an unrecognized extension we set mimetype to 'application/octet-stream' so it won't return None again. 
        if mimetype is None or encoding is not None:
            mimetype = 'application/octet-stream'
        main_type, sub_type = mimetype.split('/', 1)

        # Creating the attachement:
        # This part is used to tell how the file should be read and stored (r, or rb, etc.)
        if main_type == 'text':
            print('text')
            with open(f, 'rb') as outfile:
                attachement = text.MIMEText(outfile.read(), _subtype=sub_type)
        elif main_type == 'image':
            print('image')
            with open(f, 'rb') as outfile:
                attachement = image.MIMEImage(outfile.read(), _subtype=sub_type)
        elif main_type == 'audio':
            print('audio')
            with open(f, 'rb') as outfile:
                attachement = audio.MIMEAudio(outfile.read(), _subtype=sub_type)          
        elif main_type == 'application' and sub_type == 'pdf':   
            with open(f, 'rb') as outfile:
                attachement = application.MIMEApplication(outfile.read(), _subtype=sub_type)
        else:                              
            attachement = base.MIMEBase(main_type, sub_type)
            with open(f, 'rb') as outfile:
                attachement.set_payload(outfile.read())

        encoders.encode_base64(attachement)
        attachement.add_header('Content-Disposition', 'attachment', filename=os.path.basename(f))
        message.attach(attachement)



    media_body = http.MediaFileUpload(files[0], chunksize=500, resumable=True)
    print('Uploading large file...')
    body = {'raw': base64.urlsafe_b64encode(message.as_bytes()).decode()}


    message = (service.users().messages().send(userId='me', body=body, media_body=media_body).execute())

Note: Right now, in the MediaFileUpload I'm using files[0] because I'm only using one file for testing and I just wanted to attach one file for now until it works.

Error:

Exception has occurred: ResumableUploadError
<HttpError 400 "Bad Request">
  File "C:\Users\CON01599\AppData\Local\Continuum\anaconda3\Lib\site-packages\googleapiclient\http.py", line 927, in next_chunk
    raise ResumableUploadError(resp, content)
  File "C:\Users\CON01599\AppData\Local\Continuum\anaconda3\Lib\site-packages\googleapiclient\_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "C:\Users\CON01599\AppData\Local\Continuum\anaconda3\Lib\site-packages\googleapiclient\http.py", line 822, in execute
    _, body = self.next_chunk(http=http, num_retries=num_retries)
  File "C:\Users\CON01599\AppData\Local\Continuum\anaconda3\Lib\site-packages\googleapiclient\_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "C:\Users\CON01599\Documents\GitHub\pipelines\components\email\send_email.py", line 105, in send_email
    message = (service.users().messages().send(userId='me', body=body, media_body=media_body).execute())

Answer:

import base64
import io
import json
import os
from email import utils, encoders
from email.message import EmailMessage
from email.mime import application, multipart, text, base, image, audio
import mimetypes

from apiclient import errors
from googleapiclient import discovery, http
from google.oauth2 import service_account


def get_environment_variables():
    """ Retrieves the environment variables and returns them in
        a dictionary object.
    """
    env_var_dict = {
        'to': os.environ.get('TO'),
        'subject': os.environ.get('SUBJECT'),
        'body': os.environ.get('BODY'),
        'file': os.environ.get('FILE')
    }

    return env_var_dict


def send_email(email_subject, email_body, email_sender='my_service_account@gmail.com', email_to='', email_cc='', email_bcc='', files=None):

    # Pulling in the string value of the service key from the parameter
    with open(os.environ.get('SERVICE_KEY_PASSWORD')) as f:
        service_account_info = json.loads(f.read())

    # Define which scopes we're trying to access
    SCOPES = ['https://www.googleapis.com/auth/gmail.send']

    # Setting up credentials using the gmail api
    credentials = service_account.Credentials.from_service_account_info(service_account_info, scopes=SCOPES)
    # This allows us to assign an alias account to the message so that the messages aren't coming from 'ServiceDriod-8328balh blah blah'
    delegated_credentials = credentials.with_subject(email_sender)
    # 'Building' the service instance using the credentials we've passed
    service = discovery.build(serviceName='gmail', version='v1', credentials=delegated_credentials)

    # Building out the email 
    message = multipart.MIMEMultipart()
    message['to'] = email_to
    message['from'] = email_sender
    message['date'] = utils.formatdate(localtime=True)
    message['subject'] = email_subject
    message['cc'] = email_cc
    message['bcc'] = email_bcc
    message.attach(text.MIMEText(email_body, 'html'))


    for f in files or []:
        f = f.strip(' ')
        mimetype, encoding = mimetypes.guess_type(f)

        # If the extension is not recognized it will return: (None, None)
        # If it's an .mp3, it will return: (audio/mp3, None) (None is for the encoding)
        # For an unrecognized extension we set mimetype to 'application/octet-stream' so it won't return None again. 
        if mimetype is None or encoding is not None:
            mimetype = 'application/octet-stream'
        main_type, sub_type = mimetype.split('/', 1)

        # Creating the attachement:
        # This part is used to tell how the file should be read and stored (r, or rb, etc.)
        if main_type == 'text':
            print('text')
            with open(f, 'rb') as outfile:
                attachement = text.MIMEText(outfile.read(), _subtype=sub_type)
        elif main_type == 'image':
            print('image')
            with open(f, 'rb') as outfile:
                attachement = image.MIMEImage(outfile.read(), _subtype=sub_type)
        elif main_type == 'audio':
            print('audio')
            with open(f, 'rb') as outfile:
                attachement = audio.MIMEAudio(outfile.read(), _subtype=sub_type)          
        elif main_type == 'application' and sub_type == 'pdf':   
            with open(f, 'rb') as outfile:
                attachement = application.MIMEApplication(outfile.read(), _subtype=sub_type)
        else:                              
            attachement = base.MIMEBase(main_type, sub_type)
            with open(f, 'rb') as outfile:
                attachement.set_payload(outfile.read())

        encoders.encode_base64(attachement)
        attachement.add_header('Content-Disposition', 'attachment', filename=os.path.basename(f))
        message.attach(attachement)

    media_body = http.MediaIoBaseUpload(io.BytesIO(message.as_bytes()), mimetype='message/rfc822', resumable=True)
    body_metadata = {} # no thread, no labels in this example

    try:
        print('Uploading file...')
        response = service.users().messages().send(userId='me', body=body_metadata, media_body=media_body).execute()
        print(response)
    except errors.HttpError as error:
        print('An error occurred when sending the email:\n{}'.format(error))


if __name__ == '__main__':

    env_var_dict = get_environment_variables()
    print("Sending email...")
    send_email(email_subject=env_var_dict['subject'], 
            email_body=env_var_dict['body'], 
            email_to=env_var_dict['to'],
            files=env_var_dict['file'].split(','))

    print("Email sent!")

I have to ask: Why do the files have to be sent through email? Perhaps an alternative approach could be to store the files on a server and grant the recipient access to where the files are stored on that server? There are so many ways to share large files (FTP, Cloud services like Dropbox, downloading from a web server through a browser, etc) and I would say that emails is not a good one, so I just want to check with you to make sure email is really the only way to go about this. — LetEpsilonBeLessThanZero, Apr 05 '19 at 19:42
@LetEpsilonBeLessThanZero this will be used mostly for sending reports. I don't think the reports will be larger than 35 MB but regardless I was under the impression that large files would get converted to google drive links like when you try to attach a large file through the GUI. — Jonathan Porter, Apr 05 '19 at 19:45

score 2 · Answer 1 · answered Apr 08 '19 at 13:47

2

You mention the attachment being larger than 10Mb, but you don't mention it being smaller than 25Mb: there's a limitation to gmail that attachments can't be larger than 25Mb, so if this is your case, there's simply no way to get this done, as it is beyond gmail limitations.

The explanation can be found here.

Can you confirm that your attachment is not too large?

answered Apr 08 '19 at 13:47

Dominique

16,450
15
56
112

1

Yes, the attachment would be below 25 MB – Jonathan Porter Apr 08 '19 at 14:01

tehhowch · Accepted Answer · 2019-04-08T21:18:55.133

The issue you're having here is that your MediaUpload is a single attachment.

Instead of uploading a single attachment as a resumable MediaUpload, you need to upload the entire RFC822 message as a resumable MediaUpload.

In other words:

import ...
...
from io import BytesIO
from googleapiclient.http import MediaIoBaseUpload

SCOPES = [ 'scopes' ]

creds = get_credentials_somehow()
gmail = get_authed_service_somehow()

msg = create_rfc822_message(headers, email_body)
to_attach = get_attachment_paths_from_dir('../reports/tps/memos/2019/04')
add_attachments(msg, to_attach)

media = MediaIoBaseUpload(BytesIO(msg.as_bytes()), mimetype='message/rfc822', resumable=True)
body_metadata = {} # no thread, no labels in this example
resp = gmail.users().messages().send(userId='me', body=body_metadata, media_body=media).execute()
print(resp)
# { "id": "some new id", "threadId": "some new thread id", "labelIds": ["SENT"]}

I pieced this together from your provided code, reviewing this GitHub issue and Google's Inbox-to-Gmail email importer, specificially this bit.

When sending replies to existing messages, you will almost certainly have some sort of metadata that you should provide to help Gmail keep track of your new response and the original conversation. Namely, instead of an empty body parameter, you would pass informative metadata such as

body_metadata = { 'labelIds': [
                    "your label id here",
                    "another label id" ],
                  'threadId': "some thread id you took from the message you're replying to"
                }

Other good refs:

How to attach large files to an email using Python - Gmail API

2 Answers2

Linked