0

I like using hypothesis for my unit tests. I also like using pdb for debugging when things go wrong. But trying to use these two together can be very annoying. If I set a breakpoint in a file that is run by hypothesis using pytest <PATH-TO-FILE> -s, it will stop at the breakpoint as expected, and I can do my analysis. But after I am done, I want to be able to exit out of the test. However, if I do ctrl+c from inside the breakpoint, the test doesn't quit, it simply goes to the next hypothesis test case. And I have to keep doing this until hypothesis is done generating all it's test cases.

I usually end up opening system monitor and killing the pytest process everytime I want to be able to quit the test.

I'm hoping there is a better way.

The issue can be reproduced by the following snippet -

import hypothesis
from hypothesis import strategies as st

@hypothesis.given(st.integers())
def test_some_function(some_arg):
    breakpoint()
    print(some_arg)

test_some_function()

I am using python 3.8 with hypothesis 5.37.0

Ananda
  • 2,925
  • 5
  • 22
  • 45
  • do you want to enter the test case once and after that skip or what? – Azat Ibrakov Mar 20 '21 at 20:48
  • @AzatIbrakov Yes, essentially. – Ananda Mar 22 '21 at 06:26
  • Can you share an exact reproducing script? When I try reproducing with the `breakpoint()` builtin, I get the expected behaviour of `n` stepping through, `c` continuing to the next iteration, and ctrl-c breaking out of the debugger and pytest sessions entirely. – Zac Hatfield-Dodds Mar 22 '21 at 08:02
  • That's very surprising. I was assuming this was standard behavior. Could you please confirm if you do not have the issue with the code I updated the question with? @ZacHatfield-Dodds – Ananda Mar 22 '21 at 09:20
  • Yep, with your code `c` takes me to the next iteration and `ctrl-c` quits as expected. I've checked that on Python 3.8 and 3.9, Hypothesis 5.37.0 and 6.8.1 (worth the upgrade!), and via both python and pytest... same results in every case, so I'm totally stumped. – Zac Hatfield-Dodds Mar 22 '21 at 11:33
  • 1
    Ah, but! I *can* reproduce your problem under Ubuntu, just not Windows. This is bizzare. – Zac Hatfield-Dodds Mar 22 '21 at 11:33

1 Answers1

3

This happens under Linux but not under Windows, and it's unclear whether or not that's a bug in Hypothesis, or in Pdb, or 'just' undesirable behaviour from a combination of features.

As a workaround, you can import os; os._exit(0) to skip all cleanup logic and exit instantly.

A better, albeit somewhat more involved, solution is to disable multi-bug reporting and the shrinking phase when you use a debugger, so that Hypothesis stops generating examples immediately after the first failure. You can create a settings profile for use with the debugger, and then activate it via the --hypothesis-profile= argument to pytest.

Zac Hatfield-Dodds
  • 2,455
  • 6
  • 19