1

I am trying to run a code on AWS Lambda but it is returning me the error: "Unable to import module 'main': No module named 'email.FeedParser'".

My code does not use email feedparser module or function. It just connect to one Google API and download a CSV report.

I've checked my code scope and the reference for this module is being done by httplib2 library and on the email/parser.py from the python standard library.

All required libraries are fully updated in requirements.txt file. The code is also configured by a samTemplate.yaml file to execute in a python 3.7 environment at aws.

Do you guys had experienced this problem before? How can I solve it?

Thank you!

import httplib2
from googleapiclient import discovery
from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage
from oauth2client.tools import run_flow
from urllib.parse import urlencode
import requests
import json
import time as t
import pandas as pd
from datetime import datetime, timedelta
from calendar import monthrange
from dateutil.relativedelta import relativedelta
Victor H.
  • 89
  • 1
  • 3
  • 8
  • 1
    Do you use python2 locally by any chance? How did you create the deployment package (the .zip archive) that you upload to Lambda? There is no `email.FeedParser` in py3 version of httplib2, but there is one in its py2 version (see https://github.com/httplib2/httplib2/search?q=Feedparser&unscoped_q=Feedparser) I think that's the cause of the issue - you're uploading a py2 compatible code to a py3 container. – Milan Cermak Feb 04 '19 at 16:33
  • Hi Milan, thank you so much for your answer. I do not use python2 in my projects. There is a virtual environment of python 3.7.1 for this project. I've checked and there is a "import email.feedparser" line at httplib2. Also in the python standard lib (in email/parser.py) it is called as "from email.feedparser import FeedParser". – Victor H. Feb 06 '19 at 12:50
  • You mention `email.FeedParser` (note the CamelCase name) in the error message. That would suggest py2 version of httplib. Is the error message correct then? – Milan Cermak Feb 06 '19 at 12:52
  • Yes! The error message is: no module named "email.FeedParser". I do not know why AWS could be calling the py2 version of httplib2. You're totally right at this point, however AWS can execute it in Python 2.7 and Python 3.6/3.7. The requirements.txt file has the newest version of all these libs. – Victor H. Feb 06 '19 at 12:59
  • Just to make sure - your Lambda is configured to use py 3.6/3.7 and not 2.7, right? – Milan Cermak Feb 06 '19 at 13:00
  • Yes it is! I set this information in the samTemplate.yaml file. Also Lambda shows as the default version for this project. We are deploying the package in Lambda with Jenkins, I do not know if he can change something in the execution. We converted the code to Python 2.7 and worked fine in Lambda. However we would like to have it working in 3.7 version, as python 2.7 will be deprecated from 2020. – Victor H. Feb 06 '19 at 13:07
  • Does Jenkins build the package (i.e. install the dependencies) for you? If so, it's probably using py2 to do so and that's the culprit. – Milan Cermak Feb 06 '19 at 13:41
  • Yes, Jenkins install the dependencies. After many hours of work in Jenkinsfile to make the libs installed using pip3, we got a different error with numpy (during Jenkins build): https://pastebin.com/w9Ev7iMq This error is in the pandas installation. It really does not make sense, I've tried to google this error but I had not found anything that could help. – Victor H. Feb 06 '19 at 13:59

2 Answers2

1

I had to make sure pip3 was actually installing to python3 and not python2. Instead of:

pip3 install <package>

I had to do:

python3.8 -m pip install <package>

See answer here: Why pip3 install in python2 sitepackages

bruscantini
  • 71
  • 4
0

I had exactly the same issue...

I did this:

  1. vim ~/.pydistutils.cfg

    [install]

    prefix=

see here: 24257803 for more info on this step

  1. rm -rf [dependencies_dir]

  2. pip3 install -r requirements.txt --target [dependencies_dir]

Pip3 will install this dependencies to Python3, where pip was installing to Python2 (where some of these packages don't exist). So when deploying to Lambda via serverless the packages weren't included.

amcleod83
  • 107
  • 1
  • 4