boto3 s3 file upload using IAM role for authentication

Question

I made a code to upload the files to S3 using boto3. The code runs in docker using cron job. Initially I've set the aws credentials in the Dockerfile using ENV, and later switch to binding /home/$USER/.aws/ to the container to /root/.aws/.

FROM python:3.7-alpine

WORKDIR /scripts

RUN pip install boto3

# ENV AWS_ACCESS_KEY_ID=
# ENV AWS_SECRET_ACCESS_KEY=

COPY s3-file-upload-crontab /etc/crontabs/root
RUN chmod 644 /etc/crontabs/root

COPY s3_upload.py /scripts/s3_upload
RUN chmod a+x /scripts/s3_upload

RUN mkdir /root/info/
RUN touch /root/info/max_mod_time.json
RUN touch /root/info/error.log

RUN mkdir /root/.aws/
RUN touch /root/.aws/credentials
# RUN touch /root/.aws/config


ENTRYPOINT crond -f

version: '3.8'
services:
  s3-data-transfer:
    image: ap-aws-s3-file-upload 
    build:
      context: ./
    volumes:
      - ../data/features:/data
      - ./info:/root/info
      - ~/.aws/credentials:/root/.aws/credentials
      # - ~/.aws/config:/root/.aws/config

At this point the code is using my credentials (AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY) for authentication and works perfectly.

I'm trying to switch the authentication to IAM roles. I've created a role in AWS called Upload_Data_To_S3 with the AmazonS3FullAccess policy.

I'm reading the docs on how to set up boto3 for IAM roles. I've set my ~/.aws/config/ as follows

[default]
region=ca-central-1

[profile crossaccount]
role_arn=arn:aws:iam::#######:role/Upload_Data_To_S3
source_profile=

I don't have aws cli installed so no profile, besides my user on aws account. My python code contains no code to do with authentication.

#!/usr/local/bin/python3

import boto3
from botocore.errorfactory import ClientError
import os
import glob
import json
import time

# TODO: look into getting credentials from IAM role
s3_client = boto3.client('s3')
s3_bucket_name = 'ap-rewenables-feature-data'

max_mod_time = '0'
file_list = glob.glob('/data/*.json')  # get a list of feature files
file_mod_time = None

# get mod time for all files in data directory
file_info = [{'file': file, 'mod_time': time.strftime(
    '%Y-%m-%d %H:%M:%S', time.gmtime(os.path.getmtime(file)))} for file in file_list]

# sort files my mod time (min -> max)
timestamp_sorted_file_info = sorted(file_info, key=lambda f: f['mod_time'])
# print('File Info Sorted by Time Stamp:\n',timestamp_sorted_file_info)

# check if the file exists and not empty -> set max_mod_time from it
if os.path.exists('/root/info/max_mod_time.json') and os.stat('/root/info/max_mod_time.json').st_size != 0:
    with open('/root/info/max_mod_time.json', 'r') as mtime:
        max_mod_time = json.load(mtime)['max_mod_time']

# upload the files to s3
mod_time_last_upload = "0"
for file in timestamp_sorted_file_info:
    file_mod_time = file['mod_time']  # set mod time for the current file
    # file_mod_time = '2020-09-19 13:28:53' # for debugging
    file_name = os.path.basename(file['file'])  # get file name from file path

    if file_mod_time > max_mod_time:  # compare current file mod_time to max_mod_time from previous run
        with open(os.path.join('/data/', file_name), "rb") as f:
            s3_client.upload_fileobj(f, s3_bucket_name, file_name)

            # error check - https://stackoverflow.com/a/38376288/7582937
            # check if the file upload was successful
            try:
                s3_client.head_object(Bucket=s3_bucket_name, Key=file_name)
                mod_time_last_upload = file_mod_time
                print(file_name, ' is UPLOADED')
            except ClientError as error:
                # Not found
                if error.response['ResponseMetadata']['HTTPStatusCode'] == 404:
                    # save error to log file
                    open('/root/info/error.log', 'w').write(str(error))
                    print("error: ", error)
                break

        print('File Mod Time: ', file_mod_time)
        print('Mod Time Last Upload: ', mod_time_last_upload)


# save max mod time to file
# https://stackoverflow.com/a/5320889/7582937
# create JSON object to write to the file
object_to_write = json.dumps(
    {"max_mod_time": mod_time_last_upload})

# write max_mod_time to the file to be passed to the next run
if mod_time_last_upload is not "0":
    if object_to_write:
        open('/root/info/max_mod_time.json', 'w').write(str(object_to_write))

When I build and run the container I get the following error:

Traceback (most recent call last):
  File "/scripts/s3_upload", line 40, in <module>
    s3_client.upload_fileobj(f, s3_bucket_name, file_name)
  File "/usr/local/lib/python3.7/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.7/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/lib/python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 337, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 643, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 662, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

That's understandable since I don't have the credentials in the container. What do I need to add to the code or the ~/.aws/config file for it to use the IAM role I've set up? Unfortunately the docs aren't very clear in this regard.

Thanks in advance.

Answer 1

Try this:

import boto3

session = boto3.Session(profile_name="crossaccount")
s3 = session.client("s3")