Is this a proper way to test stdout with Python 3 unittest?

Question

Suppose I have a submission file, fileFromStudent.py, and the only thing in it is:

print("hello world")

I would like to test the stdout to see if the student has properly written out the print statement. Based on what I've read, I've been able to create the following code:

from io import StringIO
from unittest.mock import patch
import unittest, importlib, sys

class TestStringMethods(unittest.TestCase):
    def setUp(self):
        studentSubmission = 'fileFromStudent'

        ## Stores output from print() in fakeOutput
        with patch('sys.stdout', new=StringIO()) as self.fakeOutput:
            ## Loads submission on first test, reloads on subsequent tests
            if studentSubmission in sys.modules:
                importlib.reload(sys.modules[ studentSubmission ] )
            else:
                importlib.import_module( studentSubmission )

    ## Test Cases
    def test_print_passes(self):
        test_case = "Checking Output Statement - Will Pass"
        self.output = self.fakeOutput.getvalue().strip()
        self.assertEqual(self.output, 'hello world', msg=test_case)

    def test_print_fails(self):
        test_case = "Checking Output Statement - Will Fail"
        self.output = self.fakeOutput.getvalue().strip()
        self.assertEqual(self.output, 'hell world', msg=test_case)

if __name__ == '__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(TestStringMethods)
    testResult = unittest.TextTestRunner(verbosity=2).run(suite)

The above works, but am I going about things the correct way? One of the things I added in was the import.reload() call to make the student's program reload. This is because for the first few weeks, I will have the students using print() as their final output (until we get into functions).

I know it seems ambiguous, or why I should bother since it works, but is the above code a proper way to have this built or am I completely missing something that makes all this simple?

Maybe it would be simpler to do black box testing -- run the student's scripts as subprocesses, rather than importing. — Ismail Badawi, Jan 09 '16 at 05:09
Would this allow for back and forth communication? For example, run their code, provide input via stdin, retrieve output, provide more input, and repeat until done? — tsumnia, Jan 09 '16 at 05:21
Sure. You could look at `pexpect`: https://pexpect.readthedocs.org/en/stable/ which is helpful for this sort of thing. — Ismail Badawi, Jan 09 '16 at 05:23
executing a module and importing it are different things (consider `runpy` module). If you want [to catch exceptions and interact with the script; you could use `execfile()`](http://stackoverflow.com/q/12508752/4279) (also, check how `pdb` is implemented -- you could use a simplified version for your auto-grading system). — jfs, Jan 09 '16 at 05:46

tsumnia · Accepted Answer · 2016-01-24T01:10:14.833

I've spent a couple weeks working on this with moderate success, headaches, and Google. One of the reasons I did not go the Popen route was that I wanted to capture if students submitted bad code that instantly crashes. Believe or not, the first few weeks of an Intro course are like that. Since everything I've found was from 2011-2012, I figured I'd post this so future Google'ers can find it.

Expanding out what I wrote above, let's assume the next assignment was to get an input and say "Hi"

name = input("What's your name? ")
print("Hi " + name)

Now, I want to automate the test to see if I can type in "Adam" and get back "Hi Adam". To do this, I chose to use StringIO as my stdin (sys.stdin = StringIO("Adam")). This allows me to have control of where text streams are coming from and going. In addition, I don't want to see all the errors a student might have happen (sys.stderr = StringIO()).

As I mentioned, I chose to use importlib instead of Popen. I wanted to ensure that if the Student submitted bogus code, instead of breaking everything, just fail whatever test I was running. I experimented with subprocess and py.test and while they might be a better, cleaner fit, I couldn't find anything that made sense to me on how to get it moving properly.

Below is a copy of my latest version of the test:

from io import StringIO
from unittest.mock import patch
import unittest, importlib, sys, os
from time import sleep

# setup the environment
backup = sys.stderr

class TestTypingExercise(unittest.TestCase):
    def __init__(self, test_name, filename, inputs):
        super(TestTypingExercise, self).__init__(test_name)
        self.library = filename.split('.')[0]
        self.inputs = inputs

    def setUp(self):
        sys.stdin = StringIO(self.inputs[0])
        try:
            ## Stores output from print() in fakeOutput
            with patch('sys.stdout', new=StringIO()) as self.fakeOutput:
                ## Loads submission on first test, reloads on subsequent tests
                if self.library in sys.modules:
                    importlib.reload(sys.modules[ self.library ] )
                else:
                    importlib.import_module( self.library )
        except Exception as e:
            self.fail("Failed to Load - {0}".format(str(e)))

    ## Test Cases
    def test_code_runs(self):
        test_case = "Checking to See if code can run"
        self.assertTrue(True, msg=test_case)

    def test_says_hello(self):
        test_case = "Checking to See if code said 'Hi Adam'"
        # Regex might be cleaner, but this typically solves most cases
        self.output = self.fakeOutput.getvalue().strip().lower()
        self.assertTrue('hi adam' in self.output, msg=test_case)

if __name__ == '__main__':
    ignore_list = ["grader.py"]

    # Run Through Each Submitted File
    directory = os.listdir('.')
    for filename in sorted(directory):
        if (filename.split('.')[-1] != 'py') or (filename in ignore_list):
            continue
        #print("*"*15, filename, "*"*15)

        # 'Disables' stderr, so I don't have to see all their errors
        sys.stderr = StringIO()     # capture output

        # Run Tests Across Student's Submission
        suite = unittest.TestSuite()
        suite.addTest(TestTypingExercise('test_code_runs', filename, 'Adam'))
        suite.addTest(TestTypingExercise('test_says_hello', filename, 'Adam'))
        results = unittest.TextTestRunner().run(suite)

        # Reset stderr
        out = sys.stderr.getvalue() # release output
        sys.stderr.close()  # close the stream 
        sys.stderr = backup # restore original stderr

        # Display Test Results
        print(filename,"Test Results - ", end='')
        if not results.wasSuccessful():
            print("Failed (test cases that failed):")
            for error in results.failures:
                print('\t',error[1].split('\n')[-2])
        else:
            print("Pass!")
        sleep(0.05)

Here is the end result:

StudentSubmission01.py Test Results - Failed (test cases that failed):
     AssertionError: Failed to Load - EOL while scanning string literal (StudentSubmission01.py, line 23)
     AssertionError: Failed to Load - EOL while scanning string literal (StudentSubmission01.py, line 23)
StudentSubmission02.py Test Results - Pass!
StudentSubmission03.py Test Results - Pass!
StudentSubmission04.py Test Results - Pass!
StudentSubmission05.py Test Results - Pass!
StudentSubmission06.py Test Results - Pass!
StudentSubmission07.py Test Results - Pass!
StudentSubmission08.py Test Results - Pass!
StudentSubmission09.py Test Results - Pass!
StudentSubmission10.py Test Results - Pass!
StudentSubmission11.py Test Results - Pass!
StudentSubmission12.py Test Results - Pass!
StudentSubmission13.py Test Results - Pass!
[Finished in 0.9s]

I might need to move things around if I want to test multiple different inputs, but for now this works.

Is this a proper way to test stdout with Python 3 unittest?

1 Answers1

Linked