3

I was in the need to move files with a aws-lambda from a SFTP server to my AWS account, then I've found this article:

https://aws.amazon.com/blogs/compute/scheduling-ssh-jobs-using-aws-lambda/

Talking about paramiko as a SSHclient candidate to move files over ssh.

Then I've written this calss wrapper in python to be used from my serverless handler file:

import paramiko
import sys

class FTPClient(object):

    def __init__(self, hostname, username, password):
        """
        creates ftp connection
            Args:
                hostname (string): endpoint of the ftp server
                username (string): username for logging in on the ftp server
                password (string): password for logging in on the ftp server
        """

        try:
            self._host = hostname
            self._port = 22

            #lets you save results of the download into a log file.
            #paramiko.util.log_to_file("path/to/log/file.txt")
            self._sftpTransport = paramiko.Transport((self._host, self._port))
            self._sftpTransport.connect(username=username, password=password)
            self._sftp          = paramiko.SFTPClient.from_transport(self._sftpTransport)
        except:
            print ("Unexpected error" , sys.exc_info())
            raise

    def get(self, sftpPath):
        """
        creates ftp connection
            Args:               
                sftpPath = "path/to/file/on/sftp/to/be/downloaded"
        """

        localPath="/tmp/temp-download.txt"

        self._sftp.get(sftpPath, localPath)
        self._sftp.close()

        tmpfile = open(localPath, 'r')

        return tmpfile.read()


    def close(self):
        self._sftpTransport.close()

On my local machine it works as expected (test.py):

import ftp_client


sftp = ftp_client.FTPClient(
    "host", 
    "myuser",
    "password")

file = sftp.get('/testFile.txt')


print(file)

But when I deploy it with serverless and run the handler.py function (same as the test.py above) I get back the error:

 Unable to import module 'handler': No module named 'paramiko'

Looks like the deploy is unable to import paramiko (by the article above it seems like it should be available for lambda python 3 on AWS) isn't it?

If not what's the best practice for this case? Should I include the library into my local project and package/deploy it to aws?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
koalaok
  • 5,075
  • 11
  • 47
  • 91
  • 1
    Last I saw paramiko is not included in Lambda. I refer this link for the list of available packages: gist.github.com/gene1wood/4a052f39490fae00e0c3. Also, step # 5 indicates that you need to create a virtualenv, package all the dependencies (inluding paramiko) and upload it to the lambda function. – krishna_mee2004 Aug 22 '18 at 18:36

6 Answers6

3

A comprehensive guide tutorial exists at :

https://serverless.com/blog/serverless-python-packaging/

Using the serverless-python-requirements package as serverless node plugin.

Creating a virtual env and Docker Deamon will be required to packup your serverless project before deploying on AWS lambda

koalaok
  • 5,075
  • 11
  • 47
  • 91
2

In the case you use 

custom:
  pythonRequirements:
    zip: true

in your serverless.yml, you have to use this code snippet at the start of your handler

try:
  import unzip_requirements
except ImportError:
  pass

all details possible to find in Serverless Python Requirements documentation

y0j0
  • 3,369
  • 5
  • 31
  • 52
1

You have to create a virtualenv, install your dependencies and then zip all files under sites-packages/

sudo pip install virtualenv 
virtualenv -p python3 myvirtualenv
source myvirtualenv/bin/activate
pip install paramiko
cp handler.py myvirtualenv/lib/python
zip -r myvirtualenv/lib/python3.6/site-packages/ -O package.zip

then upload package.zip to lambda

Leandro Mantovani
  • 21
  • 2
  • I like this example, it just needs some more info for the use-case and machine. For me, I use macOS Catalina 10.15.7, and the command works like this: `zip -r package.zip /Users/me/.local/share/virtualenvs/lambda-dGk8CRLc/lib/python3.8/site-packages/` – nate Apr 22 '21 at 15:25
0

You have to provide all dependencies that are not installed in AWS' Python runtime.

Take a look at Step 7 in the tutorial. Looks like he is adding the dependencies from the virtual environment to the zip file. So I'd assume your ZIP file to contain the following:

  • your worker_function.py on top level
  • a folder paramico with the files installed in virtual env

Please let me know if this helps.

Konstantin A. Magg
  • 1,106
  • 9
  • 19
0

I tried various blogs and guides like:

  1. web scraping with lambda
  2. AWS Layers for Pandas

spending hours of trying out things. Facing SIZE issues like that or being unable to import modules etc.

.. and I nearly reached the end (that is to invoke LOCALLY my handler function), but then my function even though it was fully deployed correctly and even invoked LOCALLY with no problems, then it was impossible to invoke it on AWS.

The most comprehensive and best by far guide or example that is ACTUALLY working is the above mentioned by @koalaok ! Thanks buddy!

actual link

stelios_ioannides
  • 41
  • 5
  • This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://stackoverflow.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://stackoverflow.com/help/whats-reputation), you can also [add a bounty](https://stackoverflow.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/29994067) – Jan Wilamowski Oct 05 '21 at 07:33
0

The same happened to me as well. Paramiko is not working for higher versions of Python. This is only working for version 3.7. Try to edit your lambda run-time configuration to 3.7 and it will work.

Sravanthi Gudidevini
  • 19
  • 2