Do I need to set aws config in docker compose as volume?

Question

In my project I need to configure a AWS bucket download as it always gets read time error or connection error when downloading a fairly large file in my deployment. I have a .aws/config in my root directory and in my dockerfile I use "ADD . ." which adds all the files in the project. To build the image I use Docker compose. However, for some reason it is not using the aws config values. Is there a way to pass these values to Docker so that they are actually used?

This is my "config" file which is in ".aws" in the root of the project.

[default]
read_timeout = 1200
connect_timeout = 1200
http_socket_timeout = 1200

s3 =
  max_concurrent_requests = 2
  multipart_threshold = 8MB
  multipart_chunksize = 8MB

My Dockerfile looks like this:

FROM python:3.7.7-stretch AS BASE

RUN apt-get update \
    && apt-get --assume-yes --no-install-recommends install \
        build-essential \
        curl \
        git \
        jq \
        libgomp1 \
        vim

WORKDIR /app

# upgrade pip version
RUN pip install --no-cache-dir --upgrade pip

RUN pip3 install boto3

ADD . .

I expected that through the "ADD . ." boto3 would use the config file. But that is unfortunately not the case.

kumar harshit · Accepted Answer · 2022-12-08T12:53:10.660

1

Perhaps this would answer your question on why the ADD command didn't work. Hidden file .env not copied using Docker COPY

Instead of relying on the local config setting of the machine where the docker image is built, you might want to put in the configuration as an explicit file in your repo, which is copied over to ~/.aws/config or anywhere in the container and referenced by setting its path to AWS_CONFIG_FILE; OR use any one of the the methods defined in the AWS documentation below:

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

wherein you can define your configuration as part of your python code or declare them as environment variables

edited Dec 08 '22 at 12:53

answered Dec 08 '22 at 12:40

kumar harshit

487
6
12

Thanks for your reply, so I do not have a dockerignore, so the copy problem may not be the issue. For point two, I created a folder "aws_new" with the config file inside. Do I then need to write a volume in dockercompose like so "- ./aws_new /app/aws_new" and then for AWS_CONFIG_FILE something like "/app/aws_new/config". It still did not work so I probably did it wrong – Bananachip Dec 08 '22 at 13:06
Dont really need a volume now. Since you are copying all files via the ADD . . , just reference its full path in the AWS_CONFIG_FILE envar. i guess it would be /app/aws_new/config but maybe exec into the container and double check. – kumar harshit Dec 08 '22 at 13:13

Do I need to set aws config in docker compose as volume?

1 Answers1