unicode_literals and doctest in Python 2.7 AND Python 3.5

Question

Consider the following demo script:

# -*- coding: utf-8 -*-
from __future__ import division
from __future__ import unicode_literals

def myDivi():
    """
    This is a small demo that just returns the output of a divison.
    >>> myDivi()
    0.5
    """
    return 1/2

def myUnic():
    """
    This is a small demo that just returns a string.
    >>> myUnic()
    'abc'
    """
    return 'abc'

if __name__ == "__main__":
    import doctest
    extraglobs = {}
    doctest.testmod(extraglobs=extraglobs)

The doctest passes on Python 3.5, but fails on Python 2.7.9.
The strange thing is, the divison test works, but the unicode test fails.

I have seen various questions, including the following

but they are all somewhat different (e.g. they are outdated (referring to Py 2.6 or Py 3.0), import statement is within the doctest instead of globally, use pytest instead of standard doctest, switch to different assert etc)
Still, I tried various alternatives based on these questions, including e.g.

if __name__ == "__main__":
    import doctest
    import __future__
    extraglobs = {'unicode_literals': __future__.unicode_literals}
    doctest.testmod(extraglobs=extraglobs)

or

def myUnic():
    """
    This is a small demo that just returns a string.
    >>> myUnic()
    u'abc' # doctest: +ALLOW_UNICODE
    """
    return 'abc'

but it still does not work, either on Python 2 or 3 or gives other errors.
Is there a way to make it pass on both 3.5+ AND 2.7.9+, without ugly hacks? I am also using these docstrings for generating documentation, so I would prefer to keep them more or less as they are.

pytest has a `ALLOW_UNICODE` option for doctests. See https://docs.pytest.org/en/latest/doctest.html#using-doctest-options (thanks, https://stackoverflow.com/a/55937187/2747370 !) — Chris Chudzicki, Jun 28 '19 at 02:52
I already tried that, see last code snippet in the question and it did not work as expected. — matth, Jun 29 '19 at 06:05
Are you running your doctests through `pytest`? Or directly with `doctest`? I believe `ALLOW_UNICODE` is only available through `pytest`. — Chris Chudzicki, Jun 29 '19 at 14:41

Rune Kaagaard · Answer 1 · 2019-06-08T23:19:03.237

This does the job with pure doctest:

if __name__ == "__main__":
    import doctest, sys, logging, re
    from doctest import DocTestFinder, DocTestRunner
    # Support print in doctests.
    L_ = logging.getLogger(":")
    logging.basicConfig(level=logging.DEBUG)
    pr = print = lambda *xs: L_.debug(" ".join(repr(x) for x in xs))

    # Make doctest think u"" and "" is the same.
    class Py23DocChecker(doctest.OutputChecker, object):
        RE = re.compile(r"(\W|^)[uU]([rR]?[\'\"])", re.UNICODE)

        def remove_u(self, want, got):
            if sys.version_info[0] < 3:
                return (re.sub(self.RE, r'\1\2', want), re.sub(
                    self.RE, r'\1\2', got))
            else:
                return want, got

        def check_output(self, want, got, optionflags):
            want, got = self.remove_u(want, got)
            return super(Py23DocChecker, self).check_output(
                want, got, optionflags)

        def output_difference(self, example, got, optionflags):
            example.want, got = self.remove_u(example.want, got)
            return super(Py23DocChecker, self).output_difference(
                example, got, optionflags)

    finder = DocTestFinder()
    runner = DocTestRunner(checker=Py23DocChecker())
    for test in finder.find(sys.modules.get('__main__')):
        runner.run(test)
    runner.summarize()

Treats u"foo" the same as "foo"
Use print("foo") or pr("foo") to debug when running doctests. This only works if you are only using print for debugging purposes.

I've stolen most of it from places I don't remember. Thanks to these unsung heroes of the internet.

score -2 · Accepted Answer · edited May 23 '17 at 10:31

In agreement with Martijn Pieters comments in Multi version support for Python doctests, I suggest to rely on testing using some real unit test framework.

You may still use the doctest strings, because they may be nice for documentation. Think for the future, and write them for Python 3. The same time, write unit tests for another unit-testing framework. Do not rely on doctest for Python 2 version of your application/module.

unicode_literals and doctest in Python 2.7 AND Python 3.5

2 Answers2

Linked