7

The trick to use requests from botocore.vendored no longer works with Python 3.8 on AWS Lambda which I assume includes the latest botocore-1.13. This version no longer embeds requests. Importing it works but the package is actually empty.

[ERROR] AttributeError: module 'botocore.vendored.requests' has no attribute 'post'

Is there a way to import requests on native lambda? It's a basic and it would be a pain to go through the process of packaging the code for very simple Lambda.

Alternatively, anyone has another trick to call API-Gateway without requests?

Vincent Alvo
  • 155
  • 1
  • 2
  • 9

5 Answers5

1

Oh wow, I guess this is a 'breaking-change' on AWS Lambda that we rarely see.

AWS have warned about this before, there's a few options:

  • Use Serverless Framework and that serverless-python-requirements plugin
  • See if the Python3.7 Lambda Layer for requests from this repo works for Python3.8
  • Package it yourself using lambci for python3.8
keithRozario
  • 376
  • 2
  • 4
  • For now I'm more or less decided to either not use 3.8 or send SNS messages rather than API calls... I guess it's the way AWS intended it to be. – Vincent Alvo Nov 22 '19 at 16:00
0

Install "requests" locally in your project:

pip install requests -t ./

and then package and deploy it - sample here using the serverless framework:

serverless deploy -s test
Jaffadog
  • 664
  • 8
  • 16
  • This increases the size of the package drastically. Any other workaround? – Nikhil Wagh Mar 18 '20 at 10:00
  • And this has significant startup latency, which requires a keep-warm mecanism. – Vincent Alvo Jul 15 '20 at 08:18
  • humm... we're talking about just a couple MB - so not a drastic increase in size - at least not anything you need to fret over. And I dont understand why this leads to any startup latency - again nothing appreciable. Have you measured it? – Jaffadog May 24 '23 at 21:18
0

You can pip install packages at an EFS endpoint. pip install requests -t efs_endpoint

Mount that EFS endpoint from your Lambda function. Then import sys in your lambda function and add your module installation path: sys.path.insert(0, '/efs_mnt/path/to/module')

Then you can import the module into your python environment in Lambda.

It's not too difficult to set up EFS and figure out how to mount it- you may need to change ownership of the mount point to allow writes.

You can also import external modules into Lambda in a zip file but you are limited (substantially) in the size.

Connor Dibble
  • 517
  • 3
  • 13
  • Interesting proposition ! Do you know how efficient it is? How long it takes to mount and do the Lambda need to stay warm ? – Vincent Alvo Jul 10 '20 at 16:45
  • I'm working on an implementation of this and will get back to you with any salient details, but so far it looks easy and fast. You configure Lambda with the EFS endpoint; I have not benchmarked the latency to deploy a lambda instance with and without an EFS mount, but it sure looks negligible for speed and performance from what I can tell. Your keep-warm question is a good one and I will investigate in the context of my current project. – Connor Dibble Jul 13 '20 at 22:39
  • Brillant! Looking forward to hearing from you. – Vincent Alvo Jul 15 '20 at 08:17
  • Hey @VincentAlvo - Just wanted to circle back on this. Pulling in dependencies into a Python runtime in Lambda is working nicely for me. It is super fast- I haven't bench-marked but it makes no discernible difference in my context. Your Lambda does not need to be kept warm, though the usual spinup still hast to occur for cold lambdas. The biggest caveat is that EFS comes with limited burst throughput as well as an AWS-typical burst credit system. So you need to be careful that the total EFS I/O per lambda * number of lambda invocations is not greater than burst capacity or credit rate. – Connor Dibble Aug 12 '20 at 21:43
  • Hi @connor, thanks for the feedback! Much appreciated. Mine are significantly slowed down, often going more than the 30sec timeout on Gateway. This might be just my code or library I use. How big approximately is your package ? I have between 10-50Mo depending on the api. – Vincent Alvo Aug 14 '20 at 08:28
  • I'm having a lot of path issues when importing modules, so I'm not a fan of efs anymore... – coderboi Jan 23 '21 at 22:58
0

I had the same problem. The way to solve it is to use a custom layer in AWS lambda that contains the relevant site-packages you require.

It worked for me.

One example that show you how to create a layer (via a quick google) is here: https://dev.to/razcodes/how-to-create-a-lambda-layer-in-aws-106m

Dharman
  • 30,962
  • 25
  • 85
  • 135
rafalg
  • 86
  • 3
-2

Install the requests dependency separately.

$ pip install requests

Use the below import.

import requests
response = requests.get('https://...')

Removing the vendored verion and alternate solution

Associated PR which introduced this change

notionquest
  • 37,595
  • 6
  • 111
  • 105