25

I'm trying to upload a file from iOS to AWS API Gateway and pass it through to a Lambda function, How can I implement this scenario?

I can use multipart/form-data to upload to AWS API Gateway but how make input Model support binary data?

[Edit1] moved from answer by Spektre

Thanks For response, after a little of reading I figure out that's no way to upload file to lambda (and it's not logical because it's event based) and the only valid use case to upload to S3 and make S3 notify lambda.

Eloims
  • 5,106
  • 4
  • 25
  • 41
Henawey
  • 508
  • 1
  • 9
  • 18
  • **[Edit1]** response by @kixorz moved from invalid answer by Spektre: You need to configure your Integration Request and Content Types and then you can upload like you would to any other web service. – Spektre Aug 29 '17 at 08:30
  • Here I've answered this in detail if someone is still stuck: https://stackoverflow.com/a/75459125/17309297 – Rohan Taufique Feb 15 '23 at 14:35

3 Answers3

22

I'd highly recommend using direct S3 upload using one of the AWS SDKs. AWS Lambda is best suited for processing events, not content transfers like uploads. You can check its billing and limits to make a more informed decision on if it's really something you're looking for.

adamkonrad
  • 6,794
  • 1
  • 34
  • 41
  • 13
    That's not exactly true, especially now. There's a 5 minute timeout on file uploads -- if your file upload cannot finish within this time then the app needs an architecture review. Uploading directly to S3 has it's pluses and minuses -- security, data transformation limitations, etc. – Vladimir Dec 06 '16 at 15:44
  • 1
    What's not true? API Gateway now supports binary, but that doesn't change anything about the above recommendation. – adamkonrad Dec 06 '16 at 15:47
  • 10
    The above isn't a recommendation but a warning -- there's no real reason not to use Lambda for file uploads. Using S3 to upload from a website, although very useful in many situations, creates another layer of processing. By uploading directly to Lambda you can do your transformation at the time of the upload and post the result to S3 once. On the other hand, uploading to S3 first requires a Lambda function to be called to transform the file and then save it again to a different bucket (or path). Everything has its pluses and minuses -- there's no right answer, especially without more info. – Vladimir Dec 06 '16 at 15:53
  • 6
    Using S3 may create another layer of processing, but it works universally. Uploading via API Gateway to Lambda has problems, for tiny uploads it may be OK, but once you get to 1MB-10MB-100MB or GB levels where you need multi-part, you will have difficult problems using API Gateway/Lambda. – adamkonrad Dec 06 '16 at 18:46
  • 3
    I haven't heard of any issues with the uploads. Either way it depends on your requirements, such as the ability to throttle, pre-process, redirect to multiple destinations, cache, etc. – Vladimir Dec 06 '16 at 19:44
  • 3
    While your Lambda will run for 5 minutes, which is plenty of time for files in the MB order, it will surely be more expensive than uploading them to S3 and triggering the lambda with the upload. – Blueriver Apr 18 '17 at 22:48
  • Where and how to authenticate files from device? authentication is a must for uploading file to S3, because directly uploading to S3 means giving a writing permission to devices or clients. It doesn't make sense – Jason Jul 19 '17 at 09:31
  • @Jason You can generate so-called pre-signed links using Lambda and then use these links or other credentials for the upload. These links or credentials can be created just to upload one object. – adamkonrad Jul 19 '17 at 15:27
  • 1
    devices or clients (req) -> lambda, lambda (res)-> devices or clients (res : pre signed url), device or clients (upload)-> S3. Is it the same as what you mean? If S3 offer credential url, it could be one of authenticated process. But, how can I distinguish the file contents are proper or not? Because client get auth url, but we have no ideas the files from devices are real things or not when clients upload the file to S3 – Jason Jul 19 '17 at 23:55
  • 1
    @Jason The flow you're proposing is correct. You don't know if those contents are proper unless you validate it after upload. Same as you would do with any other backend service. – adamkonrad Jul 20 '17 at 02:27
  • @Jason Let your clients upload to temp location, run validation and ingestion pipeline and then move objects to permanent location. – adamkonrad Jul 20 '17 at 02:28
  • 1
    @kixorz Thanks for your reply. you are right. I have something to ask. Can I set the path of file which is saved in S3 when function in lambda generate pre signed url? – Jason Jul 20 '17 at 05:24
  • 1
    @kixorz If device choose the path in S3, it could be the problem – Jason Jul 20 '17 at 05:25
  • for example it could duplicate other files which are uploaded from other devices – Jason Jul 20 '17 at 05:55
  • @Jason That's really dependent on your implementation. You could generate filenames based on some content hash and this way you'd prevent duplicates. The pre-signed url is just a tool to auth specific operation. – adamkonrad Jul 20 '17 at 15:00
  • 7
    API Gateway has a max 29 second timeout, the Lambda timeout of 5 minutes is irrelevant, 29 seconds is pretty restrictive to handle a file upload, so about the only way to do it is to have API Gateway + Lambda return a signed S3 URL to POST the file to so you aren't using API Gateway at all. https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html – Jeremy Zerr Jun 07 '18 at 19:56
  • I think the answer - Violates standard architecture rules. - Sharing S3 credentials access at the client-side - Not be able to create backend validation - Beside the timeout issue at the above replies – Amr Feb 28 '22 at 13:46
  • The answer doesn't mention anything about credentials or backend validation. – adamkonrad Mar 01 '22 at 14:06
12

API Gateway has added support for an S3 Proxy. This allows you to expose file uploading directly to S3.

http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-s3.html

wesman16
  • 191
  • 2
  • 10
  • 1
    Does this have size limitations? Can I upload files >100MB using this? – Sujay DSa May 11 '18 at 07:18
  • 4
    You still run into the 29 second max API Gateway integration timeout: https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html – Jeremy Zerr Jun 07 '18 at 19:57
11

If want to upload file through lambda, one way is to open your AWS API Gateway console.

Go to

"API" -> {YourAPI} -> "Settings"

There you will find "Binary Media Types" section.

Add following media type:

multipart/form-data

Save your changes.

Then Go to "Resources" -> "proxy method"(eg. "ANY") -> "Method Request" -> "HTTP Request Headers" and add following headers "Content-Type", "Accept".

Finally deploy your api.

For more info visit: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-configure-with-console.html

varad_s
  • 764
  • 1
  • 12
  • 24