Format logger to ignore ANSII code - Python logging

Question

I've created a python script that uses parallel-ssh module (AKA pssh) in order to run various commands on remote nodes. One of the commands is puppet agent -t, which outputs using ANSII color codes. When the script is running and outputs to the terminal, the coloring works as expected. However, in the script log I see that it doesn't parse the ANSII codes and instead I get an ugly wrapper for each line, as below:

2016-01-12 20:23:30,748  INFO: [ubuntu01]       ESC[1;31mWarning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
2016-01-12 20:23:30,748  INFO: [ubuntu01]       (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')ESC[0m
2016-01-12 20:23:30,749  INFO: [ubuntu01]       ESC[0;32mInfo: Retrieving pluginESC[0m
2016-01-12 20:23:31,984  INFO: [ubuntu01]       ESC[0;32mInfo: Caching catalog for ubuntu01.puppetlabESC[0m
2016-01-12 20:23:32,014  INFO: [ubuntu01]       ESC[0;32mInfo: Applying configuration version '1452623010'ESC[0m
2016-01-12 20:23:32,083  INFO: [ubuntu01]       ESC[mNotice: Finished catalog run in 0.08 secondsESC[0m
2016-01-12 20:23:32,351  INFO: [ubuntu01]       * agent is running
2016-01-12 20:23:32,353  INFO: [centos01]       ESC[0;32mInfo: Retrieving pluginfactsESC[0m
2016-01-12 20:23:32,353  INFO: [centos01]       ESC[0;32mInfo: Retrieving pluginESC[0m
2016-01-12 20:23:33,712  INFO: [centos01]       ESC[0;32mInfo: Caching catalog for centos01.puppetlabESC[0m
2016-01-12 20:23:33,838  INFO: [centos01]       ESC[0;32mInfo: Applying configuration version '1452623010'ESC[0m
2016-01-12 20:23:34,101  INFO: [centos01]       ESC[mNotice: Finished catalog run in 0.27 secondsESC[0m
2016-01-12 20:23:34,421  INFO: [centos01]       puppet (pid  2069) is running...

This is very frustrating, since it makes the log less readable. I've tried to modify the logger config with re.compile(r'\x1b[^m]*m') method which I've found in this thread, like this:

import logging
import re

ansi_escape = re.compile(r'\x1b[^m]*m')
message = ansi_escape.sub('', '%(message)s')


def set_logger(log_file):
    """Define the logger.."""
    try:
        logging.basicConfig(filename=log_file, level=logging.INFO,
                        format='%(asctime)s  %(levelname)s: ' + message)
        logger = logging.getLogger(__name__)
        return logger
    except IOError:
        print "ERROR: No permissions to access the log file! Please run the script as root user (sudo will also do the trick)..\n"
        exit(2)

The script runs properly, however no changes and the log still looks messy with all these ANSII codes. I assume that there might be another place in which I can set a separate handler for pssh logger, but I wasn't able to find it.

Any help would be very much appreciated!

Answer 1

Did you try to disable puppet colors?

puppet agent -t --color=false

Answer 2

Your current code only escapes the actual logging template. Hopefully someone can come along and tell you how to properly do this, but you could use a logging adapter, which is like a logger except it allows you to modify the message being logged.

class myAdapter(logging.LoggerAdapter):

    def process(self, msg, kwargs):
        msg = re.sub(r'\x1b[^m]*m', '', msg)
        return '%s' % (msg), kwargs

def set_logger(log_file):
    """Define the logger.."""
    try:
        logging.basicConfig(filename=log_file, level=logging.INFO,
                    format='%(asctime)s  %(levelname)s: ' + message)
        logger = logging.getLogger(__name__)
        return myAdapter(logger, {})
    except IOError:
        print "ERROR: No permissions to access the log file! Please run the script as root user (sudo will also do the trick)..\n"
        exit(2)

More information can be found in the logging cookbook : https://docs.python.org/2/howto/logging-cookbook.html

Answer 3

This version keeps the original output and adds 'stdout.noansi' so you'll have both the original output and the one without ANSI formatting

import re

client = pssh.ParallelSSHClient(nodes, pool_size=args.batch, timeout=10, num_retries=1)

output = client.run_command(command, sudo=True)

for node in output:
    # let's remove ansi formatting and put it into 'stdout.noansi'...
    output[node]['stdout.noansi'] = re.sub(r'\x1b\[[^m]*?m', '', output[node]['stdout'], re.I | re.S | re.M)
    print '--------------- <<<<< NOANSI OUTPUT >>>>> ---------------'
    [print '[{0}] {1}'.format(node, line) for line in output[node]['stdout.noansi'] ]
    print '--------------- <<<<< ORIGINAL OUTPUT >>>>> ---------------'
    [print '[{0}] {1}'.format(node, line) for line in output[node]['stdout'] ]