Debugging ansible trace for possible bug in python cloud stack module

Question

Consider the following problem for an Ansible task for uploading ssh key:

  File "/home/foo/.ansible/tmp/ansible-tmp-1539175437.69-201636907966313/AnsiballZ_cs_sshkeypair.py", line 113, in <module>
    _ansiballz_main()
  File "/home/foo/.ansible/tmp/ansible-tmp-1539175437.69-201636907966313/AnsiballZ_cs_sshkeypair.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/foo/.ansible/tmp/ansible-tmp-1539175437.69-201636907966313/AnsiballZ_cs_sshkeypair.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py", line 267, in <module>
  File "/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py", line 258, in main
  File "/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py", line 133, in register_ssh_key
  File "/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py", line 211, in get_ssh_key
  File "/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py", line 227, in _get_ssh_fingerprint
  File "/usr/local/lib/python2.7/dist-packages/sshpubkeys/keys.py", line 157, in hash_md5
    fp_plain = hashlib.md5(self._decoded_key).hexdigest()
TypeError: md5() argument 1 must be string or buffer, not None

fatal: [localhost]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/foo/.ansible/tmp/ansible-tmp-1539175437.69-201636907966313/AnsiballZ_cs_sshkeypair.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/home/foo/.ansible/tmp/ansible-tmp-1539175437.69-201636907966313/AnsiballZ_cs_sshkeypair.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/foo/.ansible/tmp/ansible-tmp-1539175437.69-201636907966313/AnsiballZ_cs_sshkeypair.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py\", line 267, in <module>\n  File \"/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py\", line 258, in main\n  File \"/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py\", line 133, in register_ssh_key\n  File \"/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py\", line 211, in get_ssh_key\n  File \"/tmp/ansible_cs_sshkeypair_payload_Jb1ZG5/__main__.py\", line 227, in _get_ssh_fingerprint\n  File \"/usr/local/lib/python2.7/dist-packages/sshpubkeys/keys.py\", line 157, in hash_md5\n    fp_plain = hashlib.md5(self._decoded_key).hexdigest()\nTypeError: md5() argument 1 must be string or buffer, not None\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", 
    "rc": 1
}

While the ssh key is present, this seems to be some bug as the md5() arguments possible gets lost.

Can't identify it exactly in this form to be a known problem.

Tthis issue comes up for both Python 2.x and 3.x. (Dockerfiles for Ubuntu 16.04/18.04 LTS below).

Ubuntu 16.04 LTS, Python 2

FROM ubuntu:16.04

ENV WORKDIR /work
WORKDIR $WORKDIR
COPY . $WORKDIR/example/


RUN mv example/.ssh/ ~/ && ls -alh ~/.ssh/ && \
    apt-get -y update && apt-get -y install software-properties-common && apt-add-repository ppa:ansible/ansible && \
    apt-get -y update && \
    apt-get -y install python-pip && pip install ansible cs sshpubkeys && \
    echo "List installed Python packages:" && pip list && python --version && ansible --version

Ubuntu 18.04 LTS, Python 3

FROM ubuntu:18.04

ENV WORKDIR /work
WORKDIR $WORKDIR
COPY . $WORKDIR/example/


RUN mv example/.ssh/ ~/ && ls -alh ~/.ssh/ && \
    apt-get -y update && apt-get -y install software-properties-common && apt-add-repository ppa:ansible/ansible && \
    apt-get -y update && \
    apt-get -y install python3-pip && pip3 install ansible cs sshpubkeys && \
    echo "List installed Python packages:" && pip3 list && python3 --version && ansible --version

It seems like the system deletes the temporary generated Python scripts. Is there an option to keep them for debugging?

Side note: example/.ssh/ contains a fresh SSH piblic key not known to target system.

An assumption: while final dynamic file is named AnsiballZ_cs_sshkeypair.py, does it come from the Python cs (Apache Cloud Stack bindings) module?

Answer 1

The temporary scripts will by default be deleted because they are basically only copies of the module file (wrapped into Ansiballz) transferred from the Ansible host. In that case I guess the remote host is the same as the Ansible host but it does not matter as the concept is the same. The naming scheme for ansiballz-wrapped modules is mostly like you assumed.

But you can in fact overwrite the automatic deletion by prepending ANSIBLE_KEEP_REMOTE_FILES=1 to your playbook call like so:

ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook <playbookname>.yml -vvv

That way the ansiballz*.py is kept but the module itself is passed as some kind of base64 string. I don't think it will help much.

For further debugging I would recommend you to clone the Ansible repository from github and then do (preferrably in an extra venv)

source /path/to/dev/Ansible/hacking/env-setup

to setup the dev environment. You can then call the module directly with the Python Interpreter and use tools like pdb. I would recommend you to prepare a json file in the following format.

{"ANSIBLE_MODULE_ARGS": { "key1": "value1", "key2"... }}

You should then be able to call the module like so:

python <module-name-plus-path> <json-file-plus-path>

I hope it helps!

EDIT: I forgot one thing. Check out the Ansible dev documentation. It will help you a lot if you really want to dig deep into the matter.