Download a file from s3 to local and then send in post request

Question

Suppose i have this file in s3 bucket:

s3://qq-dag/production/special-db/xx.gz

I want now that somebody would be able to send me this path and i would automatically use this file path in order to send the file itself in a post request i make.

The solution i thought about is first downloading it to local and then somehow using my local file path to send in post request. Could it work? Anyone can post an example of how i do it?

i tried something like this but i keep getting errors for every solution i try any variation of this code:

file = r'C:\Users\jhon.king\files\xx.gz'

bucket = 'qq-dag/production/special-db'
key='xx.gz'
     
s3client = boto3.client('s3')
s3client.download_file(bucket, key, file)









//now post request part
files=[
  ('file',('xx.gz',ocpen(''r'C:\Users\jhon.king\files\xx.gz,'rb'),'text/gz'))
]
headers = {
'X-Atlassian-Token': 'nocheck',
 'Authorization': 'Basic Ymyb3dpdHo6kkczEyMzQ=',

}
response = requests.request("POST", url, headers=headers, data=payload, files=files)

I am sure i somehow misplay with the bucket or file name parts . Any one can help me?

i tried this for download based on an answer bellow i keep on getting error

​import os
prefix = 'user/mynew-research'
file = "mymyjira.gz"
#s3.Bucket(bucket).download_file(f'{prefix}/{file}', file)
​
AWS_KEY_ID = "kUvRWN8xabRdg++lUP84A3g"
AWS_ACCESS_KEY = 'AKIAJPQ'
bucket = 'my-research'
​
s3=boto3.resource(
    service_name='s3',
    region_name='us-east-1',
    aws_access_key_id=AWS_KEY_ID,
    aws_secret_access_key=AWS_ACCESS_KEY
)
s3.Bucket(bucket).download_file(f'{prefix}/{file}', file)

ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

Answer 1

The download file users the bucket/path/file altogether.

>>> bucket = "s3_bucket"
>>> prefix = "path/to"
>>> file = "file.gz"
>>> s3.Bucket(bucket).download_file(f'{prefix}/{file}', file)

I see you are downloading the file s3client.download_file(bucket, key, file) but you are posting trying to upload from the s3 path with requests, but you are not going to be able to do that.

requests is only sending a url (in your example) not a file content, you are mixing requests with S3 locations but you need to use it separately. boto3 is going to grant you permissions to download you files or do something with S3, but requests can only send the data that you have already in memory or files or data, etc.

So I would download the file (or get the file from S3) and then open the file to be sent.

# Your download file code goes here
# And now the file "xx.gz" exists in the file system

response = requests.post(
    url=url,
    headers=headers,
    files={'file_or_name_needed': open('xx.gz', 'rb')},
    data=data,
)

Here's an answer for uploading files.

When instantiating the s3 client

AWS_KEY_ID = os.environ['AWS_KEY_ID']
AWS_ACCESS_KEY = os.environ['AWS_ACCESS_KEY']

boto3.resource(
    service_name='s3',
    region_name='xx-yyyy-i',
    aws_access_key_id=AWS_KEY_ID,
    aws_secret_access_key=AWS_ACCESS_KEY,
)

Edit:

You don't need to add the file into the prefix because the fstring will do the trick.

prefix = 'my-research/user/mynew-research'
file = "mymyjira.gz"

And the fstring results in my-research/user/mynew-research/mymyjira.gz

I've tested this myself and it's working fine:

import boto3
import os
import requests

config = dict(
    service_name="s3",
    region_name="eu-west-1",
    aws_access_key_id=os.environ["AWS_KEY"],
    aws_secret_access_key=os.environ["AWS_SECRET"],
)
s3_ = boto3.resource(**config)

bucket = "bucket"
prefix = "path/to"
file = "file"

s3_.Bucket(bucket).download_file(f"{prefix}/{file}", file)

response = requests.post(
    url="https://httpbin.org/post",
    files={
        "myfile": (open(file, "rb")),
    },
)

# or if you don't want to download the file and just read it
s3_response = s3_client.get_object(Bucket=bucket, Key=f"{prefix}/{file}")

response = requests.post(
    url="https://httpbin.org/post",
    files={
        "myfile": (s3_response["Body"]).read(),
    },
)

Pay special attention to bucket and path and file.
Let's say the bucket is called test-bucket and the file is in test_data/regression/evidence.gz
The code should be s3.Bucket("test-bucket").download_file("test_data/regression/evidence.gz", "evidence.gz")