Run python program from another python program (with certain requirements)

Question

Let's say I have two python scripts A.py and B.py. I'm looking for a way to run B from within A in such a way that:

1. B believes it is __main__ (so that code in an if __name__=="__main__" block in B will run)
2. B is not actually __main__ (so that it does not, e.g., overwrite the "__main__" entry in sys.modules)
3. Exceptions raised within B propagate to A (i.e., could be caught with an except clause in A).
4. Those exceptions, if not caught, generate a correct traceback referencing line numbers within B.

I've tried various techniques, but none seem to satisfy all my requirements.

• using tools from the subprocess module means exceptions in B do not propagate to A.
• execfile("B.py", {}) runs B, but it doesn't think it's main.
• execfile("B.py", {'__name__': '__main__'}) makes B.py think it's main, but it also seems to screw up the exception traceback printing, so that the tracebacks refer to lines within A (i.e., the real __main__).
• using imp.load_source with __main__ as the name almost works, except that it actually modifies sys.modules, thus stomping on the existing value of __main__

Is there any way to get what I want?

(The reason I'm doing this is because I'm doing some cleanup on an existing library. This library has no real test suite, just a set of "example" scripts that produce certain output. I'm trying to leverage these as tests to ensure that my cleanup doesn't affect the library's ability to execute these examples, so I want to run each example script from within my test suite. I'd like to be able to see exceptions from these scripts within the test script so the test script can report the type of failure, instead of just reporting a generic SubprocessError whenever an example script raises some exception.)

Answer 1

Your use case makes sense, but I still think you'd be better off refactoring the tests such that they can be run externally.

Do you test scripts have something like this?

def test():
    pass

if __name__ == '__main__':
    test()

If not, perhaps you should convert your tests to calling a function such as test. Then, from your main test script, you can just:

import test1
test1.test()
import test2
test2.test()

Provide a common interface to running tests, that the tests themselves use. Having a big block of code in a __main__ check is Not A Good Thing.

Sorry that I didn't answer the question you asked, but I feel this is the correct solution without deviating too far from the original test code.

Answer 2

Answering my own question because the result is kind of interesting and might be useful to others:

It turns out I was wrong: execfile("B.py", {'__name__': '__main__'} is the way to go after all. It does correctly produce the tracebacks. What I was seeing with incorrect line numbers weren't exceptions but warnings. These warnings were produced using warnings.warn("blah", stacklevel=2). The stacklevel=2 argument is supposed to allow for things like deprecation warnings to be raised where the deprecated thing is used, rather than at the warning call (see the documentation).

However, it seems that the execfile-d file doesn't count as a "stack level" for this purpose, and is invisible for stacklevel purposes. So if code at the top level of an execfile-d module causes a warning with stacklevel 2, the warning is not raised at the right line number in the execfile-d source file; instead it is raised at the corresponding line number of the file which is running the execfile.

This is unfortunate, but I can live with it, since these are only warnings, so they won't impact the actual performance of the tests. (I didn't notice at first that it was only the warnings that were affected by the line-number mismatches, because there were lots of warnings and exceptions intermixed in the test output.)