13

We have a large collection of python code that takes some input and produces some output.

We would like to guarantee that, given the identical input, we produce identical output regardless of python version or local environment. (e.g. whether the code is run on Windows, Mac, or Linux, in 32-bit or 64-bit)

We have been enforcing this in an automated test suite by running our program both with and without the -R option to python and comparing the output, assuming that would shake out any spots where our output accidentally wound up dependent on iteration over a dict. (The most common source of non-determinism in our code)

However, as we recently adjusted our code to also support python 3, we discovered a place where our output depended in part on iteration over a dict that used ints as keys. This iteration order changed in python3 as compared to python2, and was making our output different. Our existing tests (all on python 2.7) didn't notice this. (Because -R doesn't affect the hash of ints) Once found, it was easy to fix, but we would like to have found it earlier.

Is there any way to further stress-test our code and give us confidence that we've ferreted out all places where we end up implicitly depending on something that will possibly be different across python versions/environments? I think that something like -R or PYTHONHASHSEED that applied to numbers as well as to str, bytes, and datetime objects could work, but I'm open to other approaches. I would however like our automated test machine to need only a single python version installed, if possible.

Another acceptable alternative would be some way to run our code with pypy tweaked so as to use a different order when iterating items out of a dict; I think our code runs on pypy, though it's not something we've ever explicitly supported. However, if some pypy expert gives us a way to tweak dictionary iteration order on different runs, it's something we'll work towards.

Daniel Martin
  • 23,083
  • 6
  • 50
  • 70
  • 2
    FWIW, python 3 doesn't _have an `-R` option, you have to use the `PYTHONHASHSEED` environment variable. On a related note, if your code uses the `random` module, the standard random number generator doesn't produce the same sequence from the same seed in Python 2 vs Python 3. The good news is that in Python 3.6+ plain `dict`s now retain insertion order, although that's currently a CPython implementation detail, and shouldn't be relied on. – PM 2Ring Jun 02 '17 at 08:52
  • Interesting question, and one I'm not qualified to answer, but note that `int`s often hash to themselves and this can have some marginal performance benefits, so altering this might have unintended consequences I guess – Chris_Rands Jun 02 '17 at 09:53
  • @pm-2ring the current implementation of `dict` remains insertion order indeed, but this might not stay the same. – Maarten Fabré Jun 02 '17 at 10:17
  • One way to tackle this then would be to bear the (small?) performance penalty of using `OrderedDict` everywhere or `sorted` when dicts have to be iterated over – Maarten Fabré Jun 02 '17 at 10:18
  • @MaartenFabré Sure. I _did_ say that it's currently an implementation detail. OTOH, it is planned to make it an official feature in a few version's time, depending on how the Python community responds. – PM 2Ring Jun 02 '17 at 10:25
  • @MaartenFabré `OrderedDict` adds a sizable memory burden over `dict` (see: https://stackoverflow.com/questions/18951143/are-there-any-reasons-not-to-use-an-ordered-dictionary) Anyway, presumably, other structures like `set`s need to be considered also – Chris_Rands Jun 02 '17 at 11:46
  • There is no option to affect the hashing of integers, no. The solution is simple: **Test on all Python versions you plan to support**. There are legion reasons code execution can differ between Python versions, do **not** attempt to get away with testing on just one release. – Martijn Pieters Jun 21 '17 at 11:50
  • @MartijnPieters maybe you'd like to expand into an answer? – Chris_Rands Jun 21 '17 at 16:02

1 Answers1

6

Using PyPy is not the best choice here, given that it always retain the insertion order in its dicts (with a method that makes dicts use less memory). We can of course make it change the order dicts are enumerated, but it defeats the point.

Instead, I'd suggest to hack at the CPython source code to change the way the hash is used inside dictobject.c. For example, after each hash = PyObject_Hash(key); if (hash == -1) { ..error.. }; you could add hash ^= HASH_TWEAK; and compile different versions of CPython with different values for HASH_TWEAK. (I did such a thing at one point, but I can't find it any more. You need to be a bit careful about where the hash values are the original ones or the modified ones.)

Armin Rigo
  • 12,048
  • 37
  • 48
  • Sure, that's what PyPy does normally, but one of the advantages of PyPy is that it's a python interpreter that can be messed with comparatively easily and adjusting it so that iteration order came out as directed by an env. var would meet our requirements here. – Daniel Martin Jun 02 '17 at 16:52
  • (At least, I've been told that it can be tweaked to behave in odd ways. It isn't something I've actually done) – Daniel Martin Jun 02 '17 at 16:53
  • 1
    (Possibly relevant: Armin Rigo is one of the founders of the PyPy project, and he really knows what he's talking about.) – user2357112 Jun 02 '17 at 16:59
  • 1
    Sure, hacking the CPython source code is a *great idea*.. That's a rather drastic approach when the OP should just bite the bullet and test on all Python versions they plan to support, no? – Martijn Pieters Jun 21 '17 at 11:51