0

I want to be download a file from some public URL straight into a S3 object without downloading it first to my server and then to upload to S3.

I see here that it's possible to use File or InputStream:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html

I can use this https://stackoverflow.com/a/6932406/4279201 with a PutObjectRequest that accepts InputStream but I think it will download the file to my server first and then upload it to S3, so is there a way to prevent this and upstream the file directly to S3?

shinzou
  • 5,850
  • 10
  • 60
  • 124
  • What you are asking isn't possible. You need to download the file somewhere and then upload it to S3. It's like asking someone else to pay for the network and compute required to move one file from one location to another. – kgiannakakis Nov 01 '21 at 10:52
  • Are you simply trying to avoid the file being persisted to disk on your server, or are you literally trying to avoid the bytes flowing through your server (via streaming)? – jarmod Nov 01 '21 at 11:11
  • @jarmod I prefer the latter, I guess what I linked in the question is the former. – shinzou Nov 01 '21 at 11:26
  • S3 has no feature to pull files from a remote location so *some* process will have to fetch the file and upload/stream it to S3. If you simply want to avoid your server being that process then, as long as the files are not too large, you could perhaps create a Lambda function that you trigger from your server and the Lambda function would fetch the file and stream it to S3. – jarmod Nov 01 '21 at 11:30
  • @jarmod what if the files might be large? – shinzou Nov 01 '21 at 11:32
  • 1
    Lambda has a limited runtime (15 minutes). Obviously fetching the file and streaming it to S3 takes time (you can measure approximately how long). If that doesn't work, then you'd need some other compute solution. You could, for example, trigger the launch of an EC2 instance to which you pass details of the file to be fetched and the target in S3, then the EC2 instance does the work, then auto-terminates. – jarmod Nov 01 '21 at 11:36

0 Answers0