How to print output when using pytest with xdist

Question

I'm using py.test to run tests. I'm using it with pytest-xdist to run the tests in parallel. I want to see the output of print statements in my tests.

I have: Ubuntu 15.10, Python 2.7.10, pytest-2.9.1, pluggy-0.3.1.

Here's my test file:

def test_a():
    print 'test_a'


def test_b():
    print 'test_b'

When I run py.test, nothing is printed. That's expected: by default, py.test captures output.

When I run py.test -s, it prints test_a and test_b, as it should.

When I run py.test -s -n2, again nothing is printed. How can I get the print statements to work while using -n2?

I've already read pytest + xdist without capturing output and this bug report.

I found a workaround, although not a full solution. By redirecting stdout to stderr, the output of print statements is displayed. This can be accomplished with a single line of Python code: sys.stdout = sys.stderr. If placed in conftest.py, it applies to all tests. — Steve Saporta, May 13 '16 at 13:08
More complete answer in http://stackoverflow.com/q/27006884/151641, also check the GitHub issue linked there (https://github.com/pytest-dev/pytest/issues/680) — mloskot, Jun 03 '16 at 17:16

score 11 · Answer 1 · edited Jun 11 '21 at 08:39

I just see the explain about this issue in github https://github.com/pytest-dev/pytest/issues/1693

pytest-xdist use sys stdin/stdout to control execution, the only way to show print out should be std.err.

import sys
def test_a():
    print >> sys.stderr, 'test_a'


def test_b():
    print >> sys.stderr, 'test_b'

Not a good idea, but work.

Also you should note that the arg -s has no use if you use xdist plugin.

In python 3, I think logging.warning is a better choice, since that it is set up to write to stderr by default.

import logging
logging.basicConfig(format='%(message)s')

def test_a():
    logging.warning('test_a')


def test_b():
    logging.warning('test_b')

to see the log file, create a file called pytest.ini and add the below to it

log_cli = 1
log_cli_level = WARN
log_file = ./TestResults/test_log.log
log_file_level = WARN

I tried this way but have several issues: 1. Messages are not shown in terminal output. 2. Different workers overwrite the log file and only several last messages are written in the log. — algot, Jan 24 '22 at 17:31

score 0 · Answer 2 · answered May 16 '23 at 08:52

I have written the following into conftest.py to have console output plus dedicated worker log files:

def pytest_configure():
    """
    Configures pytest logging to output each worker's log messages
    to its own worker log file and to the console.
    """
    # Determine worker id
    # Also see: https://pytest-xdist.readthedocs.io/en/latest/how-to.html#creating-one-log-file-for-each-worker
    worker_id = os.environ.get("PYTEST_XDIST_WORKER", default="gw0")

    # Create logs folder
    logs_folder = os.environ.get("LOGS_FOLDER", default="logs_folder")
    os.makedirs(logs_folder, exist_ok=True)

    # Create file handler to output logs into corresponding worker file
    file_handler = logging.FileHandler(f"{logs_folder}/logs_worker_{worker_id}.log", mode="w")
    file_handler.setFormatter(
        logging.Formatter(
            fmt="{asctime} {levelname}:{name}:{lineno}:{message}",
            style="{",
        )
    )

    # Create stream handler to output logs on console
    # This is a workaround for a known limitation:
    # https://pytest-xdist.readthedocs.io/en/latest/known-limitations.html
    console_handler = logging.StreamHandler(sys.stderr)  # pytest only prints error logs
    console_handler.setFormatter(
        logging.Formatter(
            # Include worker id in log messages, \r is needed to separate lines in console
            fmt="\r{asctime} " + worker_id + ":{levelname}:{name}:{lineno}:{message}",
            style="{",
        )
    )

    # Configure logging
    logging.basicConfig(level=logging.INFO, handlers=[console_handler, file_handler])


@pytest.fixture(scope="function", autouse=True)
def log_test_name_at_start(request):
    """
    Before starting a test, log its name.
    This makes it easier to retrieve the logs for a specific test.
    """
    logging.info("=" * 20 + request.node.nodeid + "=" * 20)

I am starting the tests with pytest -v -s test_folder.

How to print output when using pytest with xdist

2 Answers2

Linked