0

Is there is way (programmatically) to transfer file from s3 bucket to an external ftp server without downloading it to an ec2 instance ?

More details:

  1. I have a Django server running on EC2 which serves an angular web app.

  2. User uploads a file to S3 bucket using my web app and once the upload is complete the web app sends a POST request containing the file object s3 url.

  3. The Django server upon receiving the POST request may need to copy the file (uploaded to s3) to an external ftp server. The target ftp server may be different depending upon the user who uploaded the file (each user group may have her own ftp server).

  4. I understand that upon receiving POST request, Django server can download the file from s3 and then upload it to the appropriate target ftp server.

My question is: Can I reduce overhead on my EC2 instance in step 4 by somehow initiating a transfer from s3 to the target ftp server and get a callback/notification when that transfer completes (success or error).

Thanks.

user4848830
  • 779
  • 12
  • 22
  • 1
    You could download it to something that's not an ec2 instance, but that's probably not the kind of answer which you're looking for. You should [edit] your question to be clearer about how you want to automate this process and what platform you're running on now. What programming language are you expecting to use for this? Do you have any existing code that you plan to modify? – Kenster Jan 17 '18 at 19:50
  • Hi @Kenster - added more details. – user4848830 Jan 18 '18 at 00:49
  • *"reduce overhead on my EC2 instance"* - What *"overhead"* are you trying to reduce? – Martin Prikryl Jan 18 '18 at 05:12
  • @MartinPrikryl : 1. bandwidth in/out of ec2 instance; 2. local storage; – user4848830 Jan 18 '18 at 07:47
  • You can save storage by streaming S3 directly to FTP. But that won't save you the bandwidth obviously. – Martin Prikryl Jan 18 '18 at 08:02
  • @MartinPrikryl : How can I trigger such transfer from my ec2 instance ? Any pointers. Thanks. – user4848830 Jan 18 '18 at 08:21
  • I do not have any pointers. But it's definitely doable. Just use some S3 and FTP library (with streaming support) from your favourite language. If you need more help, you need to ask language-specific question, otherwise your question too broad. – Martin Prikryl Jan 18 '18 at 09:08
  • @MartinPrikryl : I have added more language specific details (Python, django). However, solution in any language will work for me. I just need some hint. Thanks. – user4848830 Jan 20 '18 at 06:25

1 Answers1

1

You can create a lambda function to do the same.

A complete reference of the implementation is discussed here.

https://pythonvibes.wordpress.com/2016/12/09/ftp-and-sftp-through-lambda/

Hope it helps.

Kannaiyan
  • 12,554
  • 3
  • 44
  • 83