mocking external library return value throws access violation exception

Question

I'm writing TC for my method using external library sklearn.neighbors.KDTree.

My test target method is below,

# target.py
from sklearn.neighbors import KDTree

@staticmethod
def mymethod(a, b):
    ...
    dist, index = KDTree(a).query(b, k=3)

    # manipulate the return value from KDTree.query
    ...

and, the code I tried as TC is this.

# mytest.py
from unittest import mock

@mock.patch('sklearn.neighbors.KDTree')
def test_mymethod(mock_kdtree):
    # make test data and set mock
    a = ...
    b = ...

    mock_kdtree.return_value.query.return_value = ...

    # execute test target
    mymethod(a, b)
   
    assert mock_kdtree.called

When running TC it throws exception, Windows fatal exception: access violation on the line calling dist, index = KDTree(a).query(b, k=3).

Is there something wrong to mock KDTree return value?

From this link https://stackoverflow.com/questions/57523762/pytest-windows-fatal-exception-code-0x8001010d, I also tried disabling the faulthandler before test, and it does not disappear. I'm using python 3.7 and pytest 6.2.4... — cointreau, Mar 29 '22 at 07:20
Have you found out the exact cause of the 'access violation' error? If so, please share the findings as I'm having the same error, though not using the libraries that you mentioned. — MMM, Nov 01 '22 at 13:24
@MMM Do you mean you have same error on a test? Unfortunately I have no idea about the exact cause... but found one easier way. Just mock `@mock.patch('test_package.test_module.KDTree')`. With this I didn't need to use `monkeypatch`. — cointreau, Nov 01 '22 at 23:44

jossefaz · Answer 1 · 2022-03-29T06:25:35.030

0

It is not mocking the method correctly. You typically skip the step where you set the mock instance of the class.

from unittest import mock
import sklearn

@patch('sklearn.neighbors.KDTree')
def test_mymethod(mock_kdtree):
    # make test data and set mock
    a = ...
    b = ...
    # First create a mock instance of the kdtree class
    mock_kdtree_instance = mock.MagicMock()                                        
    mock_kdtree_instance.query.return_value = ...
    # assign this instance to the class mock
    mock_kdtree.return_value = mock_kdtree_instance
    # execute test target
    mymethod(a, b)
   
    mock_kdtree_query.assert_called()

edited Mar 29 '22 at 06:25

answered Mar 29 '22 at 05:59

jossefaz

3,312
4
17
40

When I changed my code according to your answer, I got an error `TypeError: can't set attributes of built-in/extension type 'sklearn.neighbors._kd_tree.KDTree'`. I heard `sklearn.neighbors.KDTree` is written in cython, does this make the problem? – cointreau Mar 29 '22 at 06:15
It may be changed to `mock_kdtree_instance.query.called` for assertion? I tried with edited code, and it now makes the `Windows fatal exception: access violation` again... – cointreau Mar 29 '22 at 06:34
`Windows fatal exception: access violation` IMHO means that the mock is not active – jossefaz Mar 29 '22 at 07:37
1

If you are using pytest, you can try the monckeypatch fixture...did you tried it ? I can try to show you how to use it – jossefaz Mar 29 '22 at 07:38
No I've never used monkeypatch fixture. I should try it – cointreau Mar 29 '22 at 08:02
finally I could mock `KDTree().query()` with `monkeypatch`. I still don't know how it works differently from `mock`... anyway, I needed to make mockKDTree to mock KDTree(..) and then `query` to return fake value. Thank you for advice :) – cointreau Mar 30 '22 at 01:18
Great : happy to know. Feel free to post you own answer... ;) – jossefaz Mar 30 '22 at 04:26

score 0 · Answer 2 · answered Apr 13 '22 at 01:14

For someone who might be struggling with same problem, I share how I made my test.

The key is using monkeypatch, from the advice @jossefaz. Thank you!

# mytest.py
def test_mymethod(monkeypatch):

    class MockKDTree(object):
        def __init__(self, *fake_args):
            pass

        def query(self, *fake_args, **fake_kwargs):
            return fake_kdtree_dists, None    # This is what I wanted to return. You need to prepare.

    monkeypatch.setattr("mypackage.mymodule.KDTree", MockKDTree)

    # execute test target
    mymethod(a, b)
   
    # assertion

mocking external library return value throws access violation exception

2 Answers2