Testing script output

Question

I have a Python script that accepts one input (a text file): ./myprog.py file.txt. The script outputs a string based on the given input.

I have a set of test files that I would like to test my program with. I know the expected output for each file and want to make sure my script produces the correct output for each file.

What is the generally accepted way to do this type of testing?

I was thinking of using Python's unittest module as the testing framework, and then running my script through subprocess.check_output(stderr=subprocess.STDOUT), capturing stdout and stderr, and then doing a unittest assertEqual to compare the actual and expected strings. I want to make sure I'm not missing some nicer solution.

Unit testing is usually done on individual functions or classes (and doesn't usually involve writing output to files or running things with subprocess). I can't recommend anything else but to look at a python unit testing tutorial, and that would probably render this question off-topic. — byxor, Jan 23 '17 at 00:18
Its not technically a unit test because it changes things on the hard drive, but its okay to do it anyway. Consider refactoring your code by moving the functionality inside a function that takes a file-like object for input and output. The regular code would pass in an open file handle and `sys.stdout`. But your unit test code would use StringIO buffers. There are differences when writing to stringio instead of a pipe attached to a terminal so this is not always the best approach. You'll have to decide. — tdelaney, Jan 23 '17 at 00:23
you can always run `./myprog.py file.txt > result.txt` and later use tools to compare files - ie. `diff result.txt expected.txt` - but it is not so usefull as test - even if you put all in bash/batch script. — furas, Jan 23 '17 at 00:30

score 1 · Accepted Answer · edited May 23 '17 at 11:54

There's two problems here. Testing a program, as opposed to a library of functions, and testing something that prints, as opposed to values returned from a function. Both make testing more difficult, so it's best to side step these problems as much as possible.

The usual technique is to create a library of functions and then have your program be a thin wrapper around that. These functions return their results, and only the program does the printing. This means you can use normal unit testing techniques for most of the code.

You can have a single file which is both a library and a program. Here's a simple example as hello.py.

def hello(greeting, place):
    return greeting + ", " + place + "!"

def main():
    print(hello("Hello", "World"))

if __name__ == '__main__':
    main()

That last bit is how a file can tell if it was run as a program or if it was imported as a library. It allows access to the individual functions with import hello, and it also allows the file to run as a program. See this answer for more information.

Then you can write a mostly normal unit test.

import hello
import unittest
import sys
from StringIO import StringIO
import subprocess

class TestHello(unittest.TestCase):
    def test_hello(self):
        self.assertEqual(
             hello.hello("foo", "bar"),
            "foo, bar!"
        )

    def test_main(self):
        saved_stdout = sys.stdout
        try:
            out = StringIO()
            sys.stdout = out
            hello.main()
            output = out.getvalue()
            self.assertEqual(output, "Hello, World!\n")
        finally:
            sys.stdout = saved_stdout

    def test_as_program(self):
        self.assertEqual(
            subprocess.check_output(["python", "hello.py"]),
            "Hello, World!\n"
        )

if __name__ == '__main__':
    unittest.main()

Here test_hello is unit testing hello directly as a function; and in a more complicated program there would be more functions to test. We also have test_main to unit test main using StringIO to capture its output. Finally, we ensure the program will run as a program with test_as_program.

The important thing is to test as much of the functionality as functions returning data, and to test as little as possible as printed and formatted strings, and almost nothing via running the program itself. By the time we're actually testing the program, all we need to do is check that it calls main.

Great explanation. I think the module should be named `example.py` as we are importing it as `import example` — Kshitij Saraogi, Jan 23 '17 at 03:24
@KshitijSaraogi Just an oversight. I renamed it from `example.py` to `hello.py` while I was working on the code and didn't recopy the tests. — Schwern, Jan 23 '17 at 03:26
That's good to know. I learnt some new things about unit testing through your example. — Kshitij Saraogi, Jan 23 '17 at 06:16
Thanks for taking the time to illustrate different ways to test. — user7455238, Jan 24 '17 at 02:10

Testing script output

1 Answers1