Disabling output when compiling with distutils

Question

I have a setup.py script which needs to probe the compiler for certain things like the support for TR1, the presence of windows.h (to add NOMINMAX define), etc. I do these checks by creating a simple program and trying to compile it with Distutils' Compiler class. The presence/lack of errors is my answer.

This works well, but it means that the compiler's ugly error messages get printed to the console. Is there a way to suppress error messages for when the compile function is called manually?

Here is my function which tries to compile the program, which now DOES eliminate the error messages by piping the error stream to a file (answered my own question):

def see_if_compiles(program, include_dirs, define_macros):
    """ Try to compile the passed in program and report if it compiles successfully or not. """
    from distutils.ccompiler import new_compiler, CompileError
    from shutil import rmtree
    import tempfile
    import os

    try:
        tmpdir = tempfile.mkdtemp()
    except AttributeError:
        # Python 2.2 doesn't have mkdtemp().
        tmpdir = "compile_check_tempdir"
        try:
            os.mkdir(tmpdir)
        except OSError:
            print "Can't create temporary directory. Aborting."
            sys.exit()

    old = os.getcwd()

    os.chdir(tmpdir)

    # Write the program
    f = open('compiletest.cpp', 'w')
    f.write(program)
    f.close()

    # redirect the error stream to keep ugly compiler error messages off the command line
    devnull = open('errors.txt', 'w')
    oldstderr = os.dup(sys.stderr.fileno())
    os.dup2(devnull.fileno(), sys.stderr.fileno())
    #
    try:
        c = new_compiler()
        for macro in define_macros:
            c.define_macro(name=macro[0], value=macro[1])
        c.compile([f.name], include_dirs=include_dirs)
        success = True
    except CompileError:
        success = False
    # undo the error stream redirect
    os.dup2(oldstderr, sys.stderr.fileno())
    devnull.close()

    os.chdir(old)
    rmtree(tmpdir)
    return success

Here is a function which uses the above to check for the presence of a header.

def check_for_header(header, include_dirs, define_macros):
    """Check for the existence of a header file by creating a small program which includes it and see if it compiles."""
    program = "#include <%s>\n" % header
    sys.stdout.write("Checking for <%s>... " % header)
    success = see_if_compiles(program, include_dirs, define_macros)
    if (success):
        sys.stdout.write("OK\n");
    else:
        sys.stdout.write("Not found\n");
    return success

Answer 1

Zac's comment spurred me to look a bit more and I found that Mercurial's setup.py script has a working method for his approach. You can't just assign the stream because the change won't get inherited by the compiler process, but apparently Python has our good friend dup2() in the form of os.dup2(). That allows the same OS-level stream shenanigans that we all know and love, which do get inherited to child processes.

Mercurial's function redirects to /dev/null, but to keep Windows compatibility I just redirect to a file then delete it.

Quoth Mercurial:

# simplified version of distutils.ccompiler.CCompiler.has_function
# that actually removes its temporary files.
def hasfunction(cc, funcname):
    tmpdir = tempfile.mkdtemp(prefix='hg-install-')
    devnull = oldstderr = None
    try:
        try:
            fname = os.path.join(tmpdir, 'funcname.c')
            f = open(fname, 'w')
            f.write('int main(void) {\n')
            f.write('    %s();\n' % funcname)
            f.write('}\n')
            f.close()
            # Redirect stderr to /dev/null to hide any error messages
            # from the compiler.
            # This will have to be changed if we ever have to check
            # for a function on Windows.
            devnull = open('/dev/null', 'w')
            oldstderr = os.dup(sys.stderr.fileno())
            os.dup2(devnull.fileno(), sys.stderr.fileno())
            objects = cc.compile([fname], output_dir=tmpdir)
            cc.link_executable(objects, os.path.join(tmpdir, "a.out"))
        except:
            return False
        return True
    finally:
        if oldstderr is not None:
            os.dup2(oldstderr, sys.stderr.fileno())
        if devnull is not None:
            devnull.close()
        shutil.rmtree(tmpdir)

Answer 2

Here's a context manager that I recently wrote and found useful, because I was having the same problem with distutils.ccompiler.CCompiler.has_function while working on pymssql. I was going to use your approach (nice, thanks for sharing!) but then I thought that it could be done with less code and would be more general and flexible if I used a context manager. Here's what I came up with:

import contextlib


@contextlib.contextmanager
def stdchannel_redirected(stdchannel, dest_filename):
    """
    A context manager to temporarily redirect stdout or stderr

    e.g.:

    with stdchannel_redirected(sys.stderr, os.devnull):
        if compiler.has_function('clock_gettime', libraries=['rt']):
            libraries.append('rt')
    """

    try:
        oldstdchannel = os.dup(stdchannel.fileno())
        dest_file = open(dest_filename, 'w')
        os.dup2(dest_file.fileno(), stdchannel.fileno())

        yield
    finally:
        if oldstdchannel is not None:
            os.dup2(oldstdchannel, stdchannel.fileno())
        if dest_file is not None:
            dest_file.close()

The context for why I created this is at this blog post. Pretty much the same as yours I think.

This uses code that I borrowed from you to do the redirection (e.g.: os.dup2, etc.), but I wrapped it in a context manager so it's more general and reusable.

I use it like this in a setup.py:

with stdchannel_redirected(sys.stderr, os.devnull):
    if compiler.has_function('clock_gettime', libraries=['rt']):
        libraries.append('rt')

Answer 3

@Adam I just want to point out that there is /dev/null equivalent on Windows. It's 'NUL' but good practice is to get it from os.devnull

Answer 4

I'm pretty new to programming and python, so disregard this if it's a stupid suggestion, but can't you just reroute the error messages to a text file instead of the screen/interactive window/whatever?

I'm pretty sure I read somewhere you can do something like

error = open('yourerrorlog.txt','w')
sys.stderr = error

Again, sorry I'm probably repeating something you already know, but if the problem is you WANT the errors when it's called by another function (automated) and no errors when it's ran manual, can't you just add a keyword argument like compile(arg1, arg2, manual=True ) and then under your "except:" you add

if manual == False: print errors to console/interactive window
else: print to error  

Then when it's called by the program and not manually you just call it with compile(arg1,arg2, manual=False) so that it redirects to the file.

Answer 5

Does running in quiet mode help at all? setup.py -q build

Not a direct answer to your question, but related to your use case: there is a config command in distutils that’s designed to be subclassed and used to check for C features. It’s not documented yet, you have to read the source.