I'm running function on AWS lambda, using python 3.7.6 I have a timeout of 1 second on my function and I started to get sometime request timeout for my service. After some investigation(Added runtime logs), I found that the boto3 KMS client decrypts method is too slow. The weird thing is that it not consistently. Most of the time the decryption is taking about 0.04 seconds. But once in a while, the decrypt method is taking 0.5 seconds on the same values exactly!
What can cause that performance gap?