How can I install psycopg2 (Python Postgres driver) on AWS Lambda?

Question

Should this work? Nothing seems to happen when I try to run code using psycopg2.

I follow these instructions:

http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

so on an Amazon Linux instance, I create a Python 2.7 virtualenv

Then I do "pip install --upgrade pip"

Then I do pip install psycopg2 - I have see now these files:

(venv27)[ec2-user@ip-172-30-0-194 applyreplyPythonTest]$ ls /home/ec2-user/venv27/lib64/python2.7/site-packages/psycopg2
errorcodes.py   extensions.py   extras.py   __init__.py   _ipaddress.py   _json.py   pool.py   psycopg1.py   _psycopg.so  _range.pyc  sql.pyc  tz.py
errorcodes.pyc  extensions.pyc  extras.pyc  __init__.pyc  _ipaddress.pyc  _json.pyc  pool.pyc  psycopg1.pyc  _range.py    sql.py      tests    tz.pyc
(venv27)[ec2-user@ip-172-30-0-194 applyreplyPythonTest]$

I copy psycopg2 to the root of my Lambda code directory, where I have a lambda_function.py

#!/usr/bin/python
from __future__ import print_function
import psycopg2
import sys
import pprint
import json
import urllib
import boto3


def getdata():

    conn_string = "host='some address' dbname='DBNAME' user='XXXXXXX' password='XXXXXXX'"
    # print the connection string we will use to connect
    print("Connecting to database\n ->%s" % (conn_string))

    # get a connection, if a connect cannot be made an exception will be raised here
    print('floob')
    conn = psycopg2.connect(conn_string)
    print('conn.status', conn.status)
    print('conn.server_version', conn.server_version)

    # conn.cursor will return a cursor object, you can use this cursor to perform queries
    cursor = conn.cursor()

    # execute our Query
    cursor.execute("SELECT * FROM cognitouser")

    # retrieve the records from the database

    results = []
    for row in cursor.fetchall():
        print(row)
        #results.append(row)

    # print out the records using pretty print
    # note that the NAMES of the columns are not shown, instead just indexes.
    # for most people this isn't very useful so we'll show you how to return
    # columns as a dictionary (hash) in the next example.
    #pprint.pprint(records)




def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    getdata()
    return json.dumps(event)

I then upload my function to AWS as a zipfile via S3.

It does run, however, there is no output on the Lambda Cloudwatch log after it prints "floob".

I have independently checked that the database server is accessible.

Can anyone suggest what I might be doing wrong?

thanks

You need to statically link the shared library. Please see [this answer](https://stackoverflow.com/a/36608956/1316252). — skunkwerk, Jul 01 '17 at 17:41
You can use vanilla psycopg2-binary [distribution](https://stackoverflow.com/a/70983576/7991462) for your architecture — Constantine, Apr 26 '22 at 10:24

Rahul Patni · Answer 1 · 2021-08-08T19:29:26.930

3

I added a lamdba layer to my lambda functions that included psycopg2. Here is a list of available Lambda layers: https://github.com/jetbridge/psycopg2-lambda-layer

I'm using the serverless framework and this is what my Lambda function looks like:

functions:
  example:
    handler: handler.example
    layers:
      - arn:aws:lambda:us-east-1:898466741470:layer:psycopg2-py37:3
    events:
      - http:
          path: example
          method: post
          authorizer: aws_iam
          cors: true

edited Aug 08 '21 at 19:29

answered Aug 08 '21 at 19:19

Rahul Patni

81
1
4

To clarify this dependency doesnt need to be included in the requirements.txt since its a layer correct? – Leonardo Wildt Jan 11 '22 at 22:43

score 0 · Answer 2 · answered Mar 22 '20 at 07:51

3 years later, python 2.7 is deprecated on AWS Lambda.

With Python 3.x you may want some steps to add this dependency.

You can check [1] for more information about performing a non-standard build of psycopg2.

These are the bash commands that I used to build a layer zip on an EC2 instance using the Amazon Linux 1 AMI from [2]:

$ sudo yum update
$ sudo yum install python36
$ curl -O https://bootstrap.pypa.io/get-pip.py
$ python3 get-pip.py --user
$ sudo yum groupinstall "Development Tools"
$ sudo yum install python36-devel
$ sudo yum install postgresql-devel
$ mkdir -p package/python/lib/python3.6/site-packages/
$ pip3 install psycopg2 -t package/python/lib/python3.6/site-packages/
$ mkdir package/lib
$ sudo find / -name libpq.so*
$ cp /usr/lib64/libpq.so package/lib
$ cp /usr/lib64/libpq.so.5 package/lib
$ cp /usr/lib64/libpq.so.5.5 package/lib
$ cd package
$ zip -r psycopg2.zip *

You can then use the zip file created when creating a layer on the Lambda console.

[1] http://initd.org/psycopg/docs/install.html

[2] https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

How can I install psycopg2 (Python Postgres driver) on AWS Lambda?

2 Answers2