API Gateway - POST multipart/form-data

Question

It seems my question maybe a little similar to this one.

I have an API within my API Gateway and am doing a HTTP proxy through to an endpoint that POST's multipart/form-data files.

If I call the HTTP endpoint directly (not through the API gateway) - using postman, it works as expected, however, using the API gateway endpoint (through postman) fails.

I have compared both requests (through fiddler and CloudWatch logs) which seem to be identical:

Request for direct API call (working):

POST https://domainname/api/v1/documents HTTP/1.1
Host: api.service
Connection: keep-alive
Content-Length: 202
Authorization: AuthToken
Postman-Token: a75869d6-1d64-6b9f-513d-a80ac192c8e1
Cache-Control: no-cache
Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop
docMetaInfo: some extra data needed
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryB85rsPlMffA2fziS
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8

------WebKitFormBoundaryB85rsPlMffA2fziS
Content-Disposition: form-data; name=""; filename="Test.txt"
Content-Type: text/plain

This is a test Text File
------WebKitFormBoundaryB85rsPlMffA2fziS--

Request from the API Gateway (not working):

POST https://GATEWAY_domainname/api/v1/documents HTTP/1.1
Host: api-Gateway.service
Connection: keep-alive
Content-Length: 202
Authorization: AuthToken
Postman-Token: e25536fa-3dfa-ddcb-8ca6-3f3552d2bc40
Cache-Control: no-cache
Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop
docMetaInfo: some extra data needed
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarybX9MyWBsuLGm6QIC

x-api-key: *********************
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8

------WebKitFormBoundarybX9MyWBsuLGm6QIC
Content-Disposition: form-data; name=""; filename="Test.txt"
Content-Type: text/plain

This is a test Text File
------WebKitFormBoundarybX9MyWBsuLGm6QIC--

I have tried a few things from the gateway side, including changing the Integration Request to map a new body for the same content-type, no luck.

As far as I am aware, I should only need to passthrough this call, hence why it's becoming a little confusing - there should be no need for data manipulation / interception?

The error I get is 400 - bad request (complaining about the file not being found), but as you can see in the request, it's there.

Any ideas?

EDIT Logs from CloudWatch on the same APIGateway POST

Error is still 400 - no file found

Hello, I got exactly same issue. couldn't get solution yet. Did you find anything? I have used binary type in aws api gateway. still same issue. I am not using lambda, but directly api gateway to http proxy passthrough. — Mihir Shah, Feb 07 '19 at 12:36
@MihirShah - yes, as mentioned in the comments. I got around this by using S3 Signed URL's - they are simple and really useful in hindsight. Alternatively, you could try one of the other answers below, however, I think using S3 with this functionality fits well. — Hexie, Feb 10 '19 at 21:16

score 56 · Accepted Answer · edited Jun 12 '19 at 07:58

56

API Gateway does not currently support multipart form data. This is being considered for future development. In the meantime, you will need to modify your client to use multiple requests or a single one-part request.

Update: API Gateway now supports binary payloads. Simply define multipart/form-data as a binary media type for your API and proxy the payload directly to a Lambda function. From there you can parse the body to get your file content. There should be libraries available to help parse the multipart body (parse-multipart in Node.js for example).

edited Jun 12 '19 at 07:58

Yves M.

29,855
23
108
144

answered Jan 20 '17 at 19:03

RyanG

3,973
25
19

When you say "use multiple requests", what do you mean, multiple binary requests? Also - is there a thread I can watch and monitor this development, to know when this is supported? – Hexie Jan 20 '17 at 19:31
Possibly, depending on the nature of what you are sending. API Gateway does support binary request data. Please keep an eye on the forums (https://forums.aws.amazon.com/forum.jspa?forumID=199&start=0) for any announcements and we will try to update this post as well. – RyanG Jan 20 '17 at 20:59
7

@RyanG-AWS this doesn't make sense. If passthrough does not transparently pass the request through to the backend... then what does it do? (And why is it called "passthrough?") It's a simple `POST` with `Content-Type: multipart/form-data` -- calling it a "multipart upload" doesn't really sound like the right term for what this is, either. – Michael - sqlbot Jan 20 '17 at 23:34
2

I used the term upload because multipart form data typically contains file content as one of the content parts. I've edited the post for clarity. Keep in mind that API Gateway is not a pure reverse proxy - the API Gateway data plane performs operations on the request data even in "passthrough" mode, and specific HTTP features (such as multipart/form-data) are implemented on a case-by-case basis. That being said, I understand the confusion and I would consider this a limitation. We will look at getting this work prioritized. – RyanG Jan 22 '17 at 18:37
@RyanG-AWS Do we have any updated timelines or estimations on this as yet? – Hexie May 10 '17 at 22:51
@Hexie, have you found any solution to this problem? What did you end up going with? At the current time, I'm stuck with doing my image uploads with base64 encoded images and I can't stand the time the requests take. – Matt Clevenger Aug 01 '17 at 14:41
3

Hi @Matt, yeah - the simplest route to follow for this limitation would be to use the AWS SDK and S3 signed URL's. See these links: http://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURLDotNetSDK.html and http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjectPreSignedURLDotNetSDK.html – Hexie Aug 01 '17 at 21:53
@Hexie , I also analysed the pre-signed urls path, but the problem with that solution is that you would have to generate a new pre-signed url for each object you put on S3, or generate a list of pre-signed urls with extended expiration, what is actually inconvenient. Pre-signed urls are a solution mainly for anonymous user uploads, users that dont need to know anything from aws, load any SDK, or ask for credentials. – WilliamX Aug 16 '17 at 21:41
This is an interesting demo I found recently. I haven't tried yet, but looks promising, and would (in case it works) contradict what @RyanG-AWS originally stated at the beginning of this post. https://www.youtube.com/watch?v=BrYJlR0yRnw – WilliamX Aug 16 '17 at 21:44
@WilliamX Not sure I follow, each pre signed URL can be assigned a policy to it, hence, allowing secure access (access you want the user to have). We use our Signed URL's with policies and set the temp bucket (that it is writing to) with a TTL policy, ensuring all files get moved. We are also using this for a document upload / download solution, hence, a URL per call is exactly what we need. – Hexie Aug 17 '17 at 21:55
@Hexie Thats awesome! Its hard to tell the exact requirement, because every project is different. For my project, pre-signed URLs is an overkilling process. In my case I have a list of files and I want to upload them in one call with its formData (i.e. import a list of products img+data). – WilliamX Aug 17 '17 at 22:20
@RyanG-AWS Thanks for the update, this doesn't really answer the question, unless, it also applies to proxying direct calls to already existing API endpoints expecting `multipart/form-data` objects, meaning there is no need for a Lambda? – Hexie Aug 18 '17 at 00:28
Yes you can proxy a binary "multipart-form-data" payload to an HTTP integration using the same approach. – RyanG Aug 25 '17 at 00:04
7

@RyanG-AWS Does including `multipart/form-data` as a binary media type also match `multipart/form-data; boundary=90a8997f`? – bckygldstn Sep 12 '17 at 14:14
I don't believe so unfortunately. Last I checked API Gateway does strict matching of the entire Content-Type header value. – RyanG Sep 12 '17 at 18:50
8

@RyanG-AWS "API Gateway now supports binary payloads. Simply define "multipart/form-data" as a binary media type". Is this documented somewhere? – Hexie Apr 06 '18 at 01:43
1

less 1, answer outdated. OP, please accept the more relevant answer from @Dre. – Cory Kleiser Oct 12 '20 at 17:55

score 51 · Answer 2 · edited Mar 13 '20 at 10:03

51

For those who still need some help, this is now officially documented:

https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-configure-with-console.html

To summarize, the steps are as follows:

Go to the API Gateway settings tab for your API and add multipart/form-data to the binary media types section.
Add Content-Type and Accept to the request headers for your proxy method
Add those same headers to the integration request headers
Re-deploy the API

edited Mar 13 '20 at 10:03

Yuri

4,254
1
29
46

answered May 14 '19 at 13:46

Dre

1,985
16
13

1

This was very helpful, thanks! To add, if you are using Lambda Proxy integration ({proxy+}, just steps 1 and 4 are enough. – lakshminb7 Oct 16 '20 at 14:56
2

It's worth noting that once adding the binary media types the contents in the "body" of the lambda will be base64 encoded. – Danilo Souza Morães May 27 '21 at 17:56
Thank you soo much, spent last 4 hours trying to figure out the cause. – Sahil Yadav Apr 19 '22 at 11:05
1

If you're using an AWS Template to define your lambdas, then look at `anton-khodak`'s answer in this link https://github.com/aws/aws-sam-cli/issues/1216. @Dre's answer is the correct one -- however, the link provided only describes how to set up the API Gateway from the AWS console. – Drew H Apr 26 '22 at 18:00

Swalih · Answer 3 · 2018-11-29T10:00:41.657

I had same problem to integrate with my tomcat server, I found below changes needed to fix it.

Add Content-Type in your api's HTTP Request Headers in api gateway by console or add it in open api documentation like

{
    "/yourApi":{
        "post":{
            "operationId":"uploadImageUsingPOST",
            "produces":[
                "application/json"
            ],
            "parameters":[
            {
                "name":"Content-Type",
                "in":"header",
                "required":false,
                "type":"string"
            },
            {
                //Other headers
            }]   
        }
    }

Above step also add Content-Type in your api's HTTP Headers of integration request, If not add it there also and add one more header Accept ='/' in api gateway by console or or add it in open api documentation like
```
"requestParameters":{
    "integration.request.header.Accept":"'*/*'",
    "integration.request.header.Content-Type":"method.request.header.Content-Type",
    //Other headers
}
```
Set Content Handling as Passthrough in your api's integration request.
Add multipart/form-data as Binary Media Types in your api's settings through console or by open api documentation
```
"x-amazon-apigateway-binary-media-types": [
    "multipart/form-data"
]
```
Deploy above changes to desired stage where you are going to upload image as multipart.

Api gateway will pass your multipart file as binary array and you can still use @RequestBody MultipartFile multipartFile in your controller and spring will parse this binary to multipart for you.

your json in point 1 is invalid.. some closing brackets are incorrect — Suraj Rao, Nov 28 '18 at 08:32

score 5 · Answer 4 · answered Jun 08 '20 at 04:55

5

Solved: https://github.com/mscdex/busboy/issues/199#issuecomment-505239005

I am using express-fileupload for multipart-form-data in node.js

Then only configure setting on AWS API Gateway

Select API => setting => Binary Media Types =>

Now not corrupting any file in formdata and everything is working fine.

answered Jun 08 '20 at 04:55

yash

253
7
17

You'll still have the 10MB payload size issue. S3 is the way to go - I learnt that the hard way. – Hexie Jun 09 '20 at 05:02
And redeploy your lambda. Just in case someone also forget it as I did. :D – Gaurav Jeswani Nov 26 '21 at 04:02

score 2 · Answer 5 · answered Apr 27 '18 at 07:12

There seems to have been a change and API Gateway does no longer do a strict matching of the entire Content-Type header value so now everything for "binary" support works as expected.

Set your API to POST (or PUT) and set the Lambda integration to "proxy". Go to Settings for your API and add the media types you want to use as "binary". I've added multipart/signed. The received media type is actually: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg="sha256"; boundary="----54645645645664564563424768"

The API GW still picks this up as "binary" and delivers it as base64 to my Lambda.

In your Lambda you will then catch this:

Context:
{
  "callbackWaitsForEmptyEventLoop": true,
  "logGroupName": "/aws/lambda/api-invoice",
  "logStreamName": "2018/04/27/[$LATEST]3454",
  "functionName": "api-invoice",
  "memoryLimitInMB": "128",
  "functionVersion": "$LATEST",
  "invokeid": "345-49e2-11e8-34-345",
  "awsRequestId": "345-49e2-11e8-34-345",
  "invokedFunctionArn": "arn:aws:lambda:eu-west-1:12345:function:api-invoice"
}
-------
Event:
{
  "resource": "/peppol/as2",
  "path": "/peppol/as2",
  "httpMethod": "POST",
  "headers": {
    "Accept": "*/*",
    "AS2-From": "PEPPOL_AP",
    "AS2-To": "234567890",
    "AS2-Version": "1.1",
    "cache-control": "no-cache",
    "Content-Type": "multipart/signed; protocol=\"application/pkcs7-signature\"; micalg=\"sha256\"; boundary=\"----54645645645664564563424768\"",
    "Date": "Fri, 27 Apr 2018 06:17:10 GMT",
    "Disposition-Notification-Options": "signed-receipt-protocol=optional, pkcs7-signature; signed-receipt-micalg=optional, sha1,md5",
    "Disposition-Notification-To": "ignored@example.com",
    "Host": "123.execute-api.eu-west-1.amazonaws.com",
    "Message-ID": "<456-9d44-4c61-456-456546@172.17.0.3>",
    "MIME-Version": "1.0",
    "Postman-Token": "ert-59c1-45656-94d1-456546",
    "Recipient-Address": "as2s://123.execute-api.eu-west-1.amazonaws.com/dev/peppol/as2",
    "Subject": "234567890;PEPPOL_AP",
    "User-Agent": "PostmanRuntime/7.1.1",
    "Via": "1.1 ert-",
    "X-Amzn-Trace-Id": "Root=1-4556-ertfd6554",
    "X-CLIENT-IP": "172.17.0.1",
    "X-Forwarded-For": "xx.xxx.xx.80",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "queryStringParameters": null,
  "pathParameters": null,
  "stageVariables": null,
  "requestContext": {
    "resourceId": "80r6gp",
    "resourcePath": "/peppol/as2",
    "httpMethod": "POST",
    "extendedRequestId": "sdsdd343434=",
    "requestTime": "27/Apr/2018:06:17:11 +0000",
    "path": "/dev/peppol/as2",
    "accountId": "123",
    "protocol": "HTTP/1.1",
    "stage": "dev",
    "requestTimeEpoch": 1524809831262,
    "requestId": "354-49e2-3445-b2ba-535345",
    "identity": {
      "cognitoIdentityPoolId": null,
      "accountId": null,
      "cognitoIdentityId": null,
      "caller": null,
      "sourceIp": "xx.xxx.xx.80",
      "accessKey": null,
      "cognitoAuthenticationType": null,
      "cognitoAuthenticationProvider": null,
      "userArn": null,
      "userAgent": "PostmanRuntime/7.1.1",
      "user": null
    },
    "apiId": "123"
  },
  "body": "VGhpcyBpcyBhbiBTL01/ [snip] /S0NCg==",
  "isBase64Encoded": true
}

Correct, there is now support for binary from the API Gateway, the problem however, is that you are still limited with payload / request length limitations - which for binary files, is a problem. Although this will now work with small files, I still feel S3 Signed URL's is the best way forward / best practice scenario for this. — Hexie, May 03 '18 at 23:29
Yes, 10MB for any request to API GW. There has been a max. 6MB for Lambda but I think it is removed/increased as we have processed larger files than 6MB through Lambda... — Anders, May 04 '18 at 04:05
Correct - the lambda timeouts are now increased to 15 min (FYI) — Hexie, Dec 02 '18 at 21:21
@Hexie is pre-signed URL a two step process for client of the API? Do the caller need invoke API to get pre-signed URL and then use that URL to upload a file? — LP13, May 04 '20 at 22:25
@LP13 Yes, thats exactly how it works. They GET a pre-signed URL, then use that for POSTing / work requried. It's a very quick process and avoids any limitations you'll hit using other methods. — Hexie, May 05 '20 at 23:05

score 2 · Answer 6 · edited May 11 '21 at 08:42

2

Image Upload and Retrieval from Server Using AWS API Gateway Proxy+ and Lambda code example for uploading and retriveing images and manage binary data through AWS Gateway and lambda proxy+

Check here

edited May 11 '21 at 08:42

Prabhat Mishra

951
2
12
33

answered May 09 '21 at 07:16

Preetham Shreyas

27
2

API Gateway - POST multipart/form-data

6 Answers6

Linked