Cannot use Requests-Module on AWS Lambda

Question

I need to do a rest-call within a python script, that runs once per day. I can't pack the "requests" package into my python-package using the AWS Lambdas. I get the error: "Unable to import module 'lambda_function': No module named lambda_function"

I broke it down to the hello_world predefined script. I can pack it into a zip and upload it. Everything works. As soon as I put "import requests" into the file, I get this error.

Here is what I already did:

The permissions of the zip and the project folder (including subfolders) are set to `chmod 777`. So permissions shouldn't be a problem.
The script itself is within the root folder. When you open the zip file, you directly see it.
I installed the requests package into the root-folder of the project using `sudo pip install requests -t PATH_TO_ROOT_FOLDER`

The naming of everything looks like this:

zip-file: lambda_function.zip
py-file: lambda_function.py
handler method: lambda_handler(event, context)
handler-definition in the "webconfig: lambda_function.lambda_handler

The file I want to run in the end looks like this:

import requests
import json


def lambda_handler(event, context):
    url = 'xxx.elasticbeanstalk.com/users/login'
    headers = {"content-type": "application/json", "Authorization": "Basic Zxxxxxxxxx3NjxxZxxxxzcw==" }
    response = requests.put(url, headers=headers, verify=False)
    return 'hello lambda_handler'

I'm glad for ANY kind of help. I already used multiple hours on this issue.

In your question, the position of the `-t` option in the `pip install` command is incorrect - it must be `pip install requests -t PATH_TO_ROOT_FOLDER` . Did you just mistype it, or this is how you really ran it? — Leon, Nov 22 '16 at 12:09
@Leon: Right. Just checked the history and I did it like you mention it here. :) So that didn't cause the problem. — codepleb, Nov 22 '16 at 12:10
Have you seen the guide at http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html ? — Klaus D., Nov 22 '16 at 12:19
Does the contents of your zip file suggest that the `requests` package was correctly added to it? — Leon, Nov 22 '16 at 12:25
@Leon I just saw that the ZIP file did not contain the right structure. The files weren't in the "root" like I wrote in my question. Wtf man, such a dumb mistake. Thank you all! :) — codepleb, Nov 22 '16 at 12:32

score 232 · Answer 1 · edited Nov 22 '22 at 14:09

232

EDIT: On Oct-21-2019 Botocore removed the vendored version of requests: https://github.com/boto/botocore/pull/1829.

EDIT 2: (March 10, 2020): The deprecation date for the Lambda service to bundle the requests module in the AWS SDK is now January 30, 2021. https://aws.amazon.com/blogs/compute/upcoming-changes-to-the-python-sdk-in-aws-lambda/

EDIT 3: (Nov 22, 2022): AWS cancelled the deprecation so you can continue to use requests as described below. AWS Blog

To use requests module, you can simply import requests from botocore.vendored. For example:

from botocore.vendored import requests

def lambda_handler(event, context):
   response = requests.get("https://httpbin.org/get", timeout=10)
   print(response.json())

you can see this gist to know more modules that can be imported directly in AWS lambda.

edited Nov 22 '22 at 14:09

Red

140
1
7

answered Jan 29 '18 at 06:46

Sining Liu

2,575
2
9
9

Thank you. I've been driving myself insane trying to use `urllib2` (not as user friendly as `requests`) – The Unknown Dev Apr 18 '18 at 22:13
24

Is there a concern that these might go away in later versions? " While these vendored dependencies are still in the botocore package they should not be used as they will be removed in the future" https://botocore.amazonaws.com/v1/documentation/api/latest/index.html#upgrading-to-1-11-0 – Joshka Nov 26 '18 at 20:27
Wow. saved me from increasing my bundle size and keeping my lambda to few lines – Neeraj Gupta Apr 02 '19 at 10:24
1

This should be the accepted answer. Helped people like me to save a lot of time. – SMDC May 24 '19 at 01:55
1

FYI: I get a deprecation warning if i use this solution. – BoomShaka Aug 19 '19 at 11:49
5

I would not use this method anymore. The official repo for botocore has a warning saying that botocore.vendored requests will be removed in the future. – cvb Sep 18 '19 at 20:45
So if this is being deprecated, what do we use now? – Michael Brant Sep 24 '19 at 02:31
2

@MichaelBrant, you'll need to upload the requests library as part of your function. https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html – Sathed Oct 10 '19 at 00:09
37

A note - this no longer seems to work. ```AttributeError: module 'botocore.vendored.requests' has no attribute 'get'``` – Penumbra Dec 12 '19 at 01:07
1

@Penumbra this is because you are likely using Python 3.8. – davegallant Jun 27 '20 at 01:26
3

Right, the blog says "The Lambda runtimes for Python 3.8 and later do not include the ‘requests’ module." – Molossus Dec 12 '22 at 12:34

score 62 · Answer 2 · answered Feb 19 '20 at 02:18

62

If you're working with Python on AWS Lambda, and need to use requests, you better use urllib3, it is currently supported on AWS Lambda and you can import it directly, check the example on urllib3 site.

import urllib3

http = urllib3.PoolManager()
r = http.request('GET', 'http://httpbin.org/robots.txt')

r.data
# b'User-agent: *\nDisallow: /deny\n'
r.status
# 200

answered Feb 19 '20 at 02:18

Diego Tejada

963
8
10

2

This is the best answer. – CLAbeel Jun 16 '21 at 16:41

score 25 · Accepted Answer · answered Nov 22 '16 at 12:30

25

I finally solved the problem: The structure in my zip file was broken. It is important that the python script and the packed dependencies (as folders) are in the root of the zip file. This solved my problem.

It's a bit depressing if you find such easy errors after hours of try and failure.

answered Nov 22 '16 at 12:30

codepleb

10,086
14
69
111

Does your script return json then? I'm trying to do similar, but keep getting 'not json serialisable'. I don't have 'import json' however... code works fine locally without 'import json' so not sure why it doesn't in Lambda. – kafka Jan 20 '17 at 12:38
@kafka You may need to import the json library. Not sure though, but AWS uses python 2.7 and not 3.x. My script doesn't return json. – codepleb Jan 20 '17 at 12:41
yeah python 2.7 here. Going to attempt with explicit reference to json library. Currently re-packaging. – kafka Jan 20 '17 at 12:42
OK no joy here, in your code you simply return 'hello lambda_helper', I want to return the entire JSON response from the API Get though.... no worries if you don't know I might do a new question – kafka Jan 20 '17 at 12:46
I return status codes. But yeah, I'm not much into python. It was barely enough to write that script. – codepleb Jan 20 '17 at 12:53
May I suggest that the answer goes one step further to clarify a crucial point? What does packed dependances mean? In this case, when you are zipping the packed contents you should do something like: zip package.zip -r * whilst in the directory containing the handler script and the dependances – John Paul Hayes Oct 22 '17 at 18:55
Little more details please – softmarshmallow Sep 06 '18 at 05:31
Sorry guys, this question is so old, I cannot give a reliable answer anymore. That project I was working on is long gone. Packed dependencies meant dependencies as folders, that usually are in the root folder, but I can't give you any more on this sadly. – codepleb Mar 22 '19 at 07:52
Is there any way without zipping and without using vendor modules? – Tula May 22 '20 at 06:17

score 24 · Answer 4 · edited Dec 20 '17 at 10:20

I believe you have lambda_function.py on the Lambda console. You need to first create the Lambda function deployment package, and then use the console to upload the package.

You create a directory, for example project-dir on your system (locally)
create lambda_function.py in project-dir, copy the content of lambda_function.py from lambda console and paste it in project-dir/lambda_function.py
pip install requests -t /path/to/project-dir
Zip the content of the project-dir directory, which is your deployment package (Zip the directory content, not the directory)

Go to the Lambda console, select upload zip file in code entry type and upload your deployment package. Import requests should work without any error.

you can use `pip install requests -t .` (that's a dot after the -t) from within the directory containing your lambda_function.py rather than typing out the full project dir path. (form qarly_blue's answer). Bonus: Also works on windows — Hansang, Jan 13 '20 at 11:33

Ramis · Answer 5 · 2022-03-15T06:23:06.927

Most of the comments somehow correct, but not enough informative for AWS beginners. Here is my long resume what needs to be done for accessing requests functionality:

1. Creates root folder for AWS Lambda function

% mkdir lambda-function

2. Go inside crated root folder

% cd lambda-function

3. Create entry point Python file for AWS Lambda.

% vi lambda_function.py

4. Paste a code into lambda_function.py

import requests
def lambda_handler(event, context):   
    response = requests.get("https://www.test.com/")
    print(response.text)
    return response.text

5. Install requests library. Note:package folder created

% pip install --target ./package requests

6. Go inside package

% cd package

7. Zip package

zip -r ../deployment-package.zip .

8. Go into parent folder

% cd ..

9. Zip deployment packge and lambda function file

% zip -g deployment-package.zip lambda_function.py

In the AWS Lambda functions tap "Upload from" and pick ".zip file". Navigate to your zip package zip file: deployment-package.zip.
After upload all files will be inside AWS Lambda function.

score 9 · Answer 6 · edited May 28 '19 at 20:20

9

With this command download the folder package

pip install requests -t .

Run this command on your local machine, then zip your working directory, then upload to aws.

edited May 28 '19 at 20:20

mbunch

560
7
21

answered Mar 21 '19 at 17:17

qarly_blue

380
3
7

3

don't think this applies to AWS Lambda – Jason Wheeler Apr 01 '19 at 21:51
this answer comined with the one below by pramod munemanik together is currently the best answer – Hansang Jan 13 '20 at 11:53

score 8 · Answer 7 · answered Jan 26 '21 at 05:22

python 3.8 windows 10

lambda is looking for a specific folder structure and we are going to recreate in this manner in the steps below (https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-create):

make a folder on your desktop called "python," open a cmd terminal: cd desktop
pip install --target python requests
right click your python folder and zip it and rename the zip to 'requests.zip' - now if you look inside the zip you should see the python folder.
aws console > lambda > layers > create layer => name layer/upload requests.zip
aws console > functions > create function => in the "designer" box select layers and then "add layers." Choose custom layers and select your layer.
Go back to the function screen by clicking on the lambda symbol in the designer box. Now you can see "function code" again. Click lambda_function.py

Now you can import requests like this:

import json
import requests

def lambda_handler(event, context):
    # TODO implement
    response = requests.get('your_URL')
    return {
        'statusCode': 200,
        'body': json.dumps(response.json())
    }

score 1 · Answer 8 · answered Dec 07 '20 at 01:38

Copy whatever you have in the lambda_function fron AWS lambda console and paste it in a new python script and save it as lambda_function.py.
Make a new folder (I name it as package) and save requests module in it by running the following code in terminal: pip install -t package requests
Move lambda_function.py into the folder (package).
Go to the folder and select all content and zip them.
Go back to the AWS Lambda console. select the function and under the Function code section, click on 'Action' (on the right side) and select Upload a .zip file.
Upload the folder. lambda_function should be uploaded automatically.
Run and Enjoy.

This is just [this 2017 answer](https://stackoverflow.com/a/47903297/5067311) rephrased. — Andras Deak -- Слава Україні, Nov 08 '21 at 11:07

user15998420 · Answer 9 · 2022-12-19T15:29:21.317

0

Add a layer to your lambda function by specifying this arn (ap-south-1)

 arn:aws:lambda:ap-south-1:770693421928:layer:Klayers-p38-requests-html:10

edited Dec 19 '22 at 15:29

answered Dec 19 '22 at 15:28

user15998420

1
1

2

Where does this ARN come from? Adding non-official code to your lambda seems like a dangerous action – Ryan Amos May 10 '23 at 17:44