4

I have implemented an API which adhere with the Snowflake's Asynchronous External Function.

In our developed system, we are using AWS API gateway, Lambda function and a Third Party API( TPA). In our scenarios, we store certain information in Snowflake's table and try to enrich this table using Snowflake's External User Defined Function.

We are able to enrich the table if the number of records are less. If we try to enrich the 3 millions of records, then after certain time, our TAPI starts sending HTTP 429. This is a indicator which tells our lambda function to slow the number of Snowflake's requests.

We understand this and the moment Lambda function gets the HTTP 429, then it sends the HTTP 429 back to Snowflake in any polling/post requests. It is expected that Snowflake will slow down the request rather than throwing an error and stopped processing further.

Below response to Snowflake

{
   "statusCode" : 429
}

And it is a fixed situation which looks like Snowflake is not respecting HTTP 429 in the Request-Reply Pattern.

Praveen Kumar
  • 190
  • 5
  • 15

1 Answers1

1

Snowflake does handle HTTP 4xx responses when working with external functions. Have you engaged support? I have worked with customers having this issue, and snowflake team is able to review.

AWS API gateway has a default limit of 10000 rps.

Please review Designing High Performance External Functions

Remote services should return HTTP response code 429 when overloaded. If Snowflake sees HTTP 429, Snowflake scales back the rate at which it sends rows, and retries sending batches of rows that were not processed successfully.

Your options for resolution are:

Work with AWS to increase your API Gateway rate limit.

However, some proxy services, including Amazon API Gateway and Azure API Management, have default usage limits. When the request rate exceeds the limit, these proxy services throttle requests. If necessary, you might need to ask AWS or Azure to increase your quota on your proxy service.

or

Try using a smaller warehouse, so that snowflake sends less volume to API gateway per second. This has obvious drawback of you running slower.

john.da.costa
  • 4,682
  • 4
  • 29
  • 30