ImportError: No module named boto3 on AWS EC2 instance

Question

I am running my AWS EC2 instance and when I try to run my web app, MyCoolApp.py I get the following error:

Traceback (most recent call last):   
File "MyCoolApp.py", line 9, in <module>
    import boto3
ImportError: No module named boto3

This is after I try to start my application which I do, and used to do before I made this latest change with the following command:

[ec2-user@ip-XXX-XXX-XXX-XXX CoolApp]$ sudo python MyCoolApp.py

As you can guess I am trying to import Boto3 for use and I do that in the my python application with the following code at the top of my program:

from werkzeug.security import check_password_hash, generate_password_hash
from datetime import datetime
import time, os, random, json
import urllib
import boto3

Lastly, because I know you are going to ask this, YES I already installed Boto3 and when I run pip freeze I can see it. This is the output below:

[ec2-user@ip-XXX-XXX-XXX-XXX CoolApp]$ pip freeze
alembic==0.9.9
aws-cfn-bootstrap==1.4
awscli==1.15.83
Babel==0.9.4
backports.ssl-match-hostname==3.4.0.2
blinker==1.4
boto==2.48.0
boto3==1.9.87
botocore==1.12.87
chardet==2.0.1
click==6.7
cloud-init==0.7.6
colorama==0.2.5
configobj==4.7.2
docutils==0.14
ecdsa==0.11
Flask==1.0.2
Flask-Login==0.4.1
Flask-Mail==0.9.1
Flask-Migrate==2.1.1
Flask-Mobility==0.1.1
Flask-SQLAlchemy==2.3.2
futures==3.2.0
hibagent==1.0.0
iniparse==0.3.1
itsdangerous==0.24
Jinja2==2.10
jmespath==0.9.3
jsonpatch==1.2
jsonpointer==1.0
kitchen==1.1.1
lockfile==0.8
Mako==1.0.7
MarkupSafe==1.0
mysql-connector-python==8.0.11
paramiko==1.15.1
PIL==1.1.6
pip-tools==3.3.2
ply==3.4
protobuf==3.5.2.post1
pyasn1==0.1.7
pycrypto==2.6.1
pycurl==7.19.0
pygpgme==0.3
pyliblzma==0.5.3
pystache==0.5.3
python-daemon==1.5.2
python-dateutil==2.7.3
python-editor==1.0.3
pyxattr==0.5.0
PyYAML==3.10
requests==1.2.3
rsa==3.4.1
s3transfer==0.1.13
simplejson==3.6.5
six==1.11.0
SQLAlchemy==1.2.7
urlgrabber==3.10
urllib3==1.24.1
virtualenv==15.1.0
Werkzeug==0.14.1
yum-metadata-parser==1.1.4

Now because I suspect that someone will be suggesting this I am just going to head this off now. I have these packages all installed globally to my --user. I am NOT using a virtual environment, yes I know I should be but I didn't think that I was going to need one because everything runs on the one server. Yes I acknowledge that this might be a mistake and I use them everywhere else but thats not the point here if I could get some help with this error that would be great. It starts on my local machine and works just fine it is just when I try to run it on my server it fails.

Lastly, if this is due to a conflict with packages on my server it would be great if someone could help me figure out how to remove them, instructions would be appreciated as I am on AWS AMI Linux.

Answer 1

So with the details as provide in the question, we have the scenario where a Python script and the environment it will be executed at has:

1. Dependency on some packages;
2. No virtualenv is used, but the Python user install directory (as the ec2-user user) is used instead for simplicity for the installation of the script's dependencies;
3. The script however requires root privileges to run (due to the usage of low ports, with the elevated permission achieved via sudo (as the root user).

Naturally, running a script as root while the dependencies were installed local to a user other than root will mean that none of the dependencies will actually be accessible by that script. As the dependencies are installed under the site.USER_BASE for the ec2-user, in order for the script to be able to import its dependency while being executed under the root user, the location may be defined using the PYTHONUSERBASE environment variable.

To achieve that while using sudo, try:

$ sudo PYTHONUSERBASE=/home/ec2-user/.local python MyCoolApp.py

If that works, great, we can move onto a more cleaner approach if desired, as this approach can potentially result in situations where the ec2-user user cannot remove files from their .local directory, as files like .pyc files will be written into there as the root.

A number of approaches can be done:

1. Laziest manner: sudo pip, install packages globally; comes with all the problems of having globally available python packages.
2. Use sudo pip --user instead - only local to the root user.
3. A number of approaches at the OS level that is available on Linux may be attempted
4. Use virtualenv, and start the Python script (with sudo as root) using the env/bin/python created by virtualenv. Example (uses /var/tmp/env as the base for demonstration):

$ virtualenv /var/tmp/env
Using base prefix '/usr'
New python executable in /var/tmp/env/bin/python3.6
Also creating executable in /var/tmp/env/bin/python
Installing setuptools, pip, wheel...done.
$ . /var/tmp/env/bin/activate
(env) $ pip install -r requirements.txt
...
(env) $ deactivate
$ sudo /var/tmp/env/bin/python MyCoolApp.py

Finally, it may be beneficial to drop root privileges back to the original user (or another limited user) after the port has been opened, to improve the security situation of the runtime environment.