python fabric command stdout/stderr contains unencoded (garbage) symbols

Question

I'm running some commands over an AWS ec2 instance using fabric. The output (in some parts) contains characters such as:

[K    81% |â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ      | 142.9MB 161.0MB/s eta 0:00:01
[K    81% |â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ      | 142.9MB 165.6MB/s eta 0:00:01
[K    81% |â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ      | 142.9MB 167.0MB/s eta 0:00:01
[K    81% |â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ      | 142.9MB 167.2MB/s eta 0:00:01
[K    81% |â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ      | 143.0MB 166.0MB/s eta 0:00:01
[K    81% |â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ      | 143.0MB 162.4MB/s eta 0:00:01

These are the progress bars that would be printed properly on a terminal shell should the command be run locally.

How do we capture this correctly in stdout of a fabric connection run() command? I've tried setting the expected environment variables and other fabric settings for encoding (if that's the issue here).

This is how I set the fabric connection

def get_ec2_fabric_connection(instance_id, instance_pem_file, region):
    """
    establish connection with EC2 instance if necessary
    :param instance_id: ec2_instance id
    :param instance_pem_file: instance key name
    :param region: Region where ec2 instance is launched
    :return: Fabric connection object
    """
    user = get_instance_user(instance_id, region=region)
    conn = Connection(
        user=user,
        host=get_public_ip(instance_id, region),
        inline_ssh_env=True,
        connect_kwargs={"key_filename": [instance_pem_file]},
    )
    return conn

This is how I run a command over fabric:

            for command in commands_list:
                LOGGER.info(f"*** Executing command on ec2 instance: {command}")
                ret_obj = ec2_connection.run(
                    command,
                    echo=True,
                    warn=True,
                    pty=True,
                    shell="/bin/bash",
                    env={
                        "LC_CTYPE": "en_US.utf8",
                        "JAVA_HOME": "/usr/lib/jvm/java-11-openjdk-amd64",
                        "PYTHONIOENCODING": "utf8",
                    },
                    encoding="utf8"
                )

Thank you.

Edit:

My resolution was the following, based on this link: I was using 'pty=True' for no good reason. Removing its usage meant that progress bars were no longer printed by most processes. The output is now proper.

Two questions: 1) Which command exactly outputs these progress bars? 2) Do you need those progress bars to print correctly? Or will these outputs be redirected to a file? — Itagyba Abondanza Kuhlmann, Nov 20 '21 at 01:03
1.There are several commands that output such progress bars in the list, this one I believe is a pip install. 2. Yes, it would really help to have those printed correctly, else they sometimes get in the way of important errors. The outputs are not redirected to a file, as of now. — Cruizo, Nov 20 '21 at 01:12
Have you tried to change `--progress-bar` argument on the pip command? [here are the options](https://pip.pypa.io/en/stable/cli/pip_wheel/?highlight=--progress-bar#cmdoption-progress-bar) — Itagyba Abondanza Kuhlmann, Nov 20 '21 at 01:25
I think you can't get these chars as in console because there are special codes for console which are used to move cursor, change color, etc. - and normally console know that it shouldn't display them. Normally you should run programs without progress bars when you want to save output in file. Now you may only use string functions to remove chars or replace them with other chars. — furas, Nov 20 '21 at 02:41
Thanks @MartinPrikryl and others, I guess I was using 'pty=True' for no good reason. Removing its usage meant that progress bars were no longer printed by most processes. The output is now proper. — Cruizo, Nov 24 '21 at 00:28

python fabric command stdout/stderr contains unencoded (garbage) symbols

0 Answers0