1

I need to process MailGun webhooks. I did implement a solution directly on our web servers to process the webhooks, but MailGun generates so many calls from a large campaign that it effectively becomes a DOS attack.

One solution I've been looking at is using AWS API Gateway to a Lambda function to then push onto an SQS queue. We can then poll the queue at a rate we can manage. Unfortunately we can't get this to work as AWS API Gateway does not support multipart/form-data content types (which some of the webhooks are). This means that our SQS messages are not well formatted / structured. The best we can do is use the $util.escapeJavaScript($input.body) function in the mapping template to create an SQS message that contains the raw string of the webhook content (with escaped javascript chars) that is effectively unparsable i.e. we can't get data out of it.

I've had a go at using Zapier to process the webhook and push directly on the SQS queue. This can parse the various content types effectively and create a nicely structured message for us, but the cost of the service is not viable.

Has anybody managed this problem in another way? Are there solutions to API Gateway not parsing the content properly? I've deliberately stayed away from MailGuns event polling API as it involves significant delays before the polled data can be 'trusted' (according to MailGun).

Basically, is there another way of getting a nicely parsed message from content types multipart/form-data and application/x-www-form-urlencoded onto the queue?

Any ideas would be much appreciated!

To add, this link higlights issues with APS Gateway and multipart\form-data content: API Gateway - Post multipart\form-data

dmulter
  • 2,608
  • 3
  • 15
  • 24
InTooDeep
  • 578
  • 2
  • 7
  • 19
  • OK, just thinking about this and we could use $util.base64Encode($input.body) in order to create a 64 bit representation of the raw webhook message. This can be stored without escape chars on the queue and decoded when we pick it up. My question is how can we parse this in the same way a web server does in order to get the content? One thought is to send this to our webserver and let it parse at as it would any other request - thing is, how can we send it as a raw request? Or perhaps is there a way to parse it directly in PHP? – InTooDeep Jul 06 '17 at 21:12

3 Answers3

1

As you've mentioned you can base64 encode in api gateway and call base64decode in the lambda function to retrieve the original payload (There are standard libraries in every language).

Also, note you can that you can use multipart form data for non file bodies. Get non file body from multipart/form-data using AWS API Gateway and Lambda

Abhigna Nagaraja
  • 1,874
  • 15
  • 17
  • Thanks, tried. We fell down on parsing the decoded base64 string. Without writing our own parser I can't see anyway of getting the key : value parameters out of it. Any ideas? – InTooDeep Jul 09 '17 at 13:16
  • Did you try using a standard library like https://docs.python.org/2/library/base64.html and then parsing the output? – Abhigna Nagaraja Jul 09 '17 at 13:36
  • We have no problem decoding from base64, the issue is with parsing the result of that. Not sure if you've ever looked at 'raw' data for multipart/form-data and application/x-www-form-urlencoded? It is very unstructured and would probably require a good few days of looking at the specs just to get a rudimentary parsing solution in place. Perhaps I'm not understanding you? Do you know of an 'out of the box' way to parse this? – InTooDeep Jul 09 '17 at 15:22
1

I had the same challenge when building Suet. I ended up switching to Google Cloud functions which I really recommend. Don't waste time on Amazon API Gateway. Use Google Cloud Functions and use a middleware like multer. (You can see the source of Suet's webhook handler here).

kehers
  • 4,076
  • 3
  • 30
  • 31
0

Not sure if you ever came to a solution, but I have this working with the following settings.

  1. Setup your API Gateway method to use "Use Lambda Proxy integration"
  2. In your lambda (I use node.js) use busboy to work through the multi-part submission from the mailgun webhook. (use this post for help with busboy Busboy help)
  3. Make sure that any code you are going to execute after all busboy is complete is executed in the 'finish' portion of the busboy code.
Greg Fennell
  • 176
  • 1
  • 12