Is subprocess using the wrong version of Python to run Ansible?

Question

I'm trying to analyze the output of Ansible plays that are run as part of a larger Python3.7 application. I'm running these plays with

Popen(command, shell=False, cwd=[directory], stdout=PIPE, stderr=STDOUT)

where command is something like ['ansible-playbook', '-i', 'hosts.yml', 'playbook.yml']. The pipe goes to a file where I use regex to look at the results and output.

My regexes are breaking because on the machine I've deployed this, there seems to be a Unicode problem: for tasks where I iterate over a list of objects and use properties of the current object in the play, I get a printout like this:

changed: [0.0.0.0] => (item={uprocess: upython, unumber: u4})

I expect what I get on my development machine:

changed: [0.0.0.0] => (item={process: python, number: 4})

This isn't a problem for plays where I iterate over a flat list.

I've tried:

• adding ansible_python_interpreter: /usr/bin/python3 as an "all" variable in the hostfile

• all manner of shenanigans with tojson, to_json, safe, etc. in the Jinja portion of the loop. {{ obj_list | list }}} is the original loop that works perfectly on one machine and not the other; I've put those filters both before and after list and the results have either been the same or an entirely new error because the output isn't a valid list.

• setting Python3 to be the default version of Python on the production machine. The prod machine is running Ubuntu 18.04 and the default Python version is 3.6.8. Compare the dev machine, which is running Ubuntu 16.04.6 and where the default Python version is 3.7.3. The project runs in a Python 3.7 virtualenv anyway.

Since the "u" prefix remains but not the quotes, I'm inclined to believe this is an issue with subprocess. However, running a test script containing only

import subprocess, sys

subprocess.run(['python'])

produces a Python3 console. I'm unsure how to use related solutions because I'm not running a Python script. Why is the Unicode prefix appearing and how do I make it stop? I know how to get around it with regexes but this may be a recurring problem.

---

This example covers each element of what I'm doing, and produces the problem on the prod machine and not the dev machine:

hosts.yml:

all:
  hosts:
    127.0.0.1:
      objects:
      - number: '1'
        name: one
        type: 1
      - number: '2'
        name: two
        type: 2
  vars:
    ansible_python_interpreter: /usr/bin/python3

test.yml:

---
- hosts: all
  tasks:
  - name: print objects
    ignore_errors: true
    shell: echo {{ item.number }} && echo {{ item.name }} && echo {{ item.type }}
    loop: "{{ objects | list }}"
    register: output

  - name: show output
    debug:
      msg: "{{ item.stdout_lines | length > 0 }}"
    loop: "{{ output.results }}"
    loop_control:
      label: "{{ item.item }}"

test.py

from asynchronousfilereader import AsynchronousFileReader
from pathlib import Path
from subprocess import Popen, PIPE, STDOUT

bash = ['ansible-playbook', '-i', 'hosts.yml', 'test.yml']
# Required to connect to localhost for the purposes of the example
bash += ['--connection=local']

proc = Popen(bash, shell=False, cwd=Path.cwd(), stdout=PIPE, stderr=STDOUT)
stdout_reader = AsynchronousFileReader(proc.stdout, autostart=True)

with open('output.txt', 'a') as f:
    while not stdout_reader.eof():
        for line in stdout_reader.readlines():
            line = str(repr(line.decode("utf-8"))).replace("\'", '').replace('\\n', '\n')
            f.write(line)
stdout_reader.join()
proc.stdout.close()

Good output:

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [127.0.0.1]

TASK [print objects] ***********************************************************
"changed: [127.0.0.1] => (item={number: 1, name: one, type: 1})
""changed: [127.0.0.1] => (item={number: 2, name: two, type: 2})
"
TASK [show output] *************************************************************
"ok: [127.0.0.1] => (item={number: 1, name: one, type: 1}) => {
"    "msg": true
}
"ok: [127.0.0.1] => (item={number: 2, name: two, type: 2}) => {
"    "msg": true
}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Bad output:

PLAY [all] **********************************************************************

TASK [Gathering Facts] **********************************************************
ok: [127.0.0.1]

TASK [print objects] ************************************************************
"changed: [127.0.0.1] => (item={utype: 1, unumber: u1, uname: uone})
""changed: [127.0.0.1] => (item={utype: 2, unumber: u2, uname: utwo})
"
TASK [show output] **************************************************************
"ok: [127.0.0.1] => (item={utype: 1, unumber: u1, uname: uone}) => {
"    "msg": true
}
"ok: [127.0.0.1] => (item={utype: 2, unumber: u2, uname: utwo}) => {
"    "msg": true
}

PLAY RECAP **********************************************************************
127.0.0.1                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Note also that the order of the arguments in item has changed.