0

I'm working on a legacy code at work. We are storing our files in AWS S3. What I need to do is to move files from one S3 bucket to another(the one that our new application uses). There is a lot of stuff going one when one uploads a file to our application, so I decided the easiest way would be to emulate user uploading files to a new endpoint.

Now, how do I actually get a file from URL as a file object and then proxy it to another endpoint? Also, is there a way to do this without downloading file first? Im concerned about bigger files like videos.

What I've done so far is this:

url= 'https://example.s3.com/image'
file =  urllib.request.urlopen(url, context=ssl._create_unverified_context())

payload = {'file': open(file, 'rb')}
data = {'title': 'Picture title'}
requests.post(final_url, files=payload, data=data, headers=headers, verify=False)

The error I get is this:

File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
worker_1   |     raise HTTPError(req.full_url, code, msg, hdrs, fp)
worker_1   | OSError

Now, I'm assuming that this is because urllib has nowhere to put file, even temporary.

How to fix this, and what would be a better way of doing this?

intelis
  • 7,829
  • 14
  • 58
  • 102

1 Answers1

1

Using Boto is likely your best option assuming you are able to directly access the buckets. This answer is an excellent reference. How to move files between two Amazon S3 Buckets using boto?

BryceH
  • 2,740
  • 2
  • 21
  • 24
  • thanks @BryceH but I cant just upload to S3. We're doing some custom stuff with files before uploading them, so we really need to proxy them through our app. – intelis Sep 25 '18 at 10:25