Organizing external dependencies in AWS lambda

Question

I am relatively new to python and even newer to Lambda. I have created a lambda function that requires some external dependencies (elasticsearch and elasticsearch-curator).

My root folder is called index_curator, in which I have one single python file main.py. I installed the dependencies via pip as per Amazon's instructions e.g.

pip install elasticsearch elasticsearch-curator -t /path/to/index_curator

There are now many other files in this root directory and many child directories, which is not a surprise as these dependencies are quite large. For someone else looking at this package it would be difficult to differentiate between files I wrote and external dependencies. For example:

index_curator/
    dateutil/
    click/
    idna/
    main.py    <-- the only file I wrote
    README
    LICENSE
    six.py
    ...

Is there any way of shifting all these external dependencies into a sub-folder, e.g.

index_curator/
    /external/
        dateutil/
        click/
        idna/
        README
        LICENSE
        six.py
    main.py    <-- the only file I wrote

For completeness the imports in main.py are:

from elasticsearch import Elasticsearch, RequestsHttpConnection
import curator

Any pointers would be appreciated.

score 2 · Accepted Answer · answered Aug 02 '18 at 13:56

Separating external dependencies from your code is definitely best practice. There are a number of ways you can achieve this in python.

By default, python searches for modules in the locations specified here. To specify an additional location (i.e your external dependencies folder), this new location must be added to the python path. You can do this in your main.py as follows:

import sys, os
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), 'external')))

Then you will be able to import all your dependencies as usual, since the interpreter will check the additional folder during runtime:

from elasticsearch import Elasticsearch, RequestsHttpConnection
import curator

For more detail, check the answers here

Since your code will be a lambda function, you will always have one handler. But for a more generalised case, or if you start to write multiple files, and need to manage external dependencies in those as well - you can choose to maintain a context.py file, which sets the dependency path and all imports like so:

import sys, os
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), 'external')))
import elasticsearch
import context

Then in your files, you can call them with:

from context.elasticsearch import Elasticsearch, RequestsHttpConnection
import context.curator

Organizing external dependencies in AWS lambda

1 Answers1