Ansible aws_s3 module fails says Boto3 is missing when it is not

Question

This Play installs python3, pip3, boto3 and botocore, and tries to use aws_s3 module to download a file:

TASK [run yum update -y using yum module] 
**********************************************************************
ok: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Install python3 and pip3] *************************************************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Install boto3 and botocore with pip3 module] ******************************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Create a directory if it does not exist using file module] ****************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [downlod file from s3 with aws_s3 module] **********************************************************************************
fatal: [ip-10-200-2-137.us-west-2.compute.internal]: FAILED! => 
{"changed": false, "msg": "Python modules \"botocore\" or \"boto3\" 
are missing, please install both"}

It fails because it says boto3 is missing, but it actually it is not:

From the Target host you can see that boto3 was installed:

[ec2-user@ip-10-200-2-137 ~]$ pip3 freeze
boto3==1.9.120
botocore==1.12.120
docutils==0.14
jmespath==0.9.4
python-dateutil==2.8.0
s3transfer==0.2.0
six==1.12.0
urllib3==1.24.1
[ec2-user@ip-10-200-2-137 ~]

This is the task that installed boto3:

- name: Install boto3 and botocore with pip3 module
    pip:
      name: 
      - boto3
      - botocore
      executable: pip-3.7

This is the task that fails:

- name: downlod file from s3 with aws_s3 module 
    aws_s3:
      bucket: mybucket
      object: mybucket/jre-8u201-linux-x64.tar.gz
      dest: /home/ec2-user/updater/jre-8u201-linux-x64.tar.gz
      mode: get

The target host does have two versions of Python installed:

[ec2-user@ip-10-200-2-157 ~]$ which python
/usr/bin/python
[ec2-user@ip-10-200-2-157 ~]$ which python3
/usr/bin/python3

My config file looks like this:

[defaults]
private_key_file=/home/ec2-user/manual-builds/key.pem
ansible_python_interpreter=/usr/bin/python3

Is this a bug? I see some similar questions have been asked going back almost a year, but I see no solutions - thanks much for any help.

score 4 · Answer 1 · edited Feb 01 '21 at 09:41

4

Ansible use /usr/bin/python as default python interpreter. And you install AWS libraries for python3 only:

- name: Install boto3 and botocore with pip3 module
    pip:
      name: 
      - boto3
      - botocore
      executable: pip-3.7

You can install AWS libraries to python2 by using pip or install for both (python3 and python2) or you can define: ansible_python_interpreter=/usr/bin/python3 in your inventory file then you limit ansible execution to python3 only.

edited Feb 01 '21 at 09:41

Yuri

4,254
1
29
46

answered Mar 25 '19 at 13:17

ozlevka

1,988
16
28

Thanks - Python 2 is pre-installed on amamzon Linux. If I limit ansible execution to python3 only - will all ansible modules still work? or will some fail because they need python 2? – kavise Mar 25 '19 at 16:58
I added this to my cfg, but got the exact same result. My cfg contents are added to the original post – kavise Mar 25 '19 at 18:03
@kavise are you sure that pip-3.7 is the default for your system? – ozlevka Mar 26 '19 at 07:17
I'm not sure how to tell, but python2 comes pre-installed on the awslinux2 image that I use, so I would guess that Python2 is default – kavise Mar 26 '19 at 17:52
@kavise it's not on ansible.cfg, but on the inventory (hosts) file. Also, why is your answer the accepted one, when ozlevka provided the answer earlier? – Pedro Gordo May 15 '19 at 20:25
Thanks @Pedro Gordo, so ansible_python_interpreter=/usr/bin/python3 would go in Inventory not cfg. That’s helpful to know. I missed that in ozlevka's post, but In my situation I actually needed BOTH python versions to work with Ansible, so limiting all Tasks to a single version would not be good for me – this is why the solution of specifying version at the Task level was chosen. – kavise May 17 '19 at 18:21

score 4 · Accepted Answer · answered Mar 25 '19 at 21:40

The problem was that my playbook had two tasks and Ansible was using the python2 interpreter for the first one AND the second one. The second task needed the python3 interpreter to work so I had to specifiy it at the task level:

- name: downlod file from s3 with aws_s3 module
  vars:
      ansible_python_interpreter: /usr/bin/python3    
  aws_s3:
      bucket: launch-data
      object: jre-8u201-linux-x64.tar.gz
      dest: /home/ec2-user/updater/jre-8u201-linux-x64.tar.gz
      mode: get

this didn't work for me, I got the same error on Amazon Linux 2 — openCivilisation, May 01 '21 at 10:32

score 1 · Answer 3 · edited Feb 01 '21 at 09:40

Ansible might not be referring to the incorrect python version. Simply run ansible version command to see where ansible is pointing to.

ansible --version
ansible 2.9.9
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.17 (default, Apr 15 2020, 17:20:14) [GCC 7.5.0]

As you see Ansibe is pointing to the Python module located at /usr/lib/python2.7/ (ansible python module location) you can see if this python version has boto3 and botocore packages are installed in /usr/lib/python2.7/dist-packages/ path.

If it's missing just install boto3 and botocore there.

score 0 · Answer 4 · edited Feb 01 '21 at 09:37

When the ansible says

Failed to import the required Python library (botocore or boto3) Please read the module documentation and install it in the appropriate location.

Modules might already be installed. Confirm by python -m pip list installed command. Then open python in question directly in a terminal by typing python and try to import boto3 or botocore by import boto3. if it throws any error try to work out the error.

This answer might be useful.

score 0 · Answer 5 · answered Mar 20 '21 at 00:05

0

https://github.com/aws/aws-cli/issues/3092#issuecomment-550281243

Uninstalling python3-botocore then installing with pip3 seems to work.

answered Mar 20 '21 at 00:05

transient_loop

5,984
15
58
117

Ansible aws_s3 module fails says Boto3 is missing when it is not

5 Answers5