8

Context

I have started using pytest-vcr which is a pytest plugin wrapping VCR.py which I have documented in this blog post on Advanced Python Testing.

It records all HTTP traffic to cassettes/*.yml files on the first test run to save snapshots. Similar to Jest snapshot testing for web components.

On subsequent test runs, if a request is malformed, it won't find a match and throws an exception saying that recording new requests is forbidden and it did not find an existing recording.

Question

VCR.py raises a CannotOverwriteExistingCassetteException which is not particularly informative as to why it didn't match.

How do I leverage pytest pytest_exception_interact hooks to replace this exception with a more informative one leveraging fixture information?

I dove into my site-packages where VCR.py is pip installed and rewrote how I want it to handle the exception. I just need to know how to get this pytest_exception_interact hook to work correctly to access the fixtures from that test node (before it gets cleaned up) and raise a different exception.

Example

Lets get the dependencies.

$ pip install pytest pytest-vcr requests

test_example.py:

import pytest
import requests

@pytest.mark.vcr
def test_example():
    r = requests.get("https://www.stackoverflow.com")
    assert r.status_code == 200

$ pytest test_example.py --vcr-record=once
...
test_example.py::test_example PASSED 
...
$ ls cassettes/
cassettes/test_example.yml
$ head cassettes/test_example.yml
interactions:
- request:
    uri: https://wwwstackoverflow.com
    body: null
    headers:
        Accept:
        - '*/*'
$ pytest test_example.py --vcr-record=none
...
test_example.py::test_example PASSED 
...

Now change the URI in the test to "https://www.google.com":

test_example.py:

import pytest
import requests

@pytest.mark.vcr
def test_example():
    r = requests.get("https://www.google.com")
    assert r.status_code == 200

And run the test again to detect the regression:

$ pytest test_example.py --vcr-record=none
E               vcr.errors.CannotOverwriteExistingCassetteException: No match for the request (<Request (GET) https://www.google.com/>)
...

I can add a conftest.py file to the root of my test structure to create a local plugin, and I can verify that I can intercept the exception and inject my own using:

conftest.py

import pytest
from vcr.errors import CannotOverwriteExistingCassetteException
from vcr.config import VCR
from vcr.cassette import Cassette

class RequestNotFoundCassetteException(CannotOverwriteExistingCassetteException):
    ...

@pytest.fixture(autouse=True)
def _vcr_marker(request):
    marker = request.node.get_closest_marker("vcr")
    if marker:
        cassette = request.getfixturevalue("vcr_cassette")
        vcr = request.getfixturevalue("vcr")
        request.node.__vcr_fixtures = dict(vcr_cassette=cassette, vcr=vcr)
    yield


@pytest.hookimpl(hookwrapper=True)
def pytest_exception_interact(node, call, report):
    excinfo = call.excinfo
    if report.when == "call" and isinstance(excinfo.value, CannotOverwriteExistingCassetteException):
        # Safely check for fixture pass through on this node
        cassette = None
        vcr = None
        if hasattr(node, "__vcr_fixtures"):
            for fixture_name, fx in node.__vcr_fixtures.items():
                vcr = fx if isinstance(fx, VCR)
                cassette = fx if isinstance(fx, Cassette)

        # If we have the extra fixture context available...
        if cassette and vcr:
            match_properties = [f.__name__ for f in cassette._match_on]
            cassette_reqs = cassette.requests
            #  filtered_req = cassette.filter_request(vcr._vcr_request)
            #  this_req, req_str = __format_near_match(filtered_req, cassette_reqs, match_properties)

            # Raise and catch a new excpetion FROM existing one to keep the traceback
            # https://stackoverflow.com/a/24752607/622276
            # https://docs.python.org/3/library/exceptions.html#built-in-exceptions
            try:
                raise RequestNotFoundCassetteException(
                    f"\nMatching Properties: {match_properties}\n" f"Cassette Requests: {cassette_reqs}\n"
                ) from excinfo.value
            except RequestNotFoundCassetteException as e:
                excinfo._excinfo = (type(e), e)
                report.longrepr = node.repr_failure(excinfo)

This is the part where the documentation on the internet gets pretty thin.

How do I access the vcr_cassette fixture and return a different exception?

What I want to do is get the filtered_request that was attempting to be requested and the list of cassette_requests and using the Python difflib standard library produce deltas against the information that diverged.

PyTest Code Spelunking

The internals of running a single test with pytest triggers pytest_runtest_protocol which effectively runs the following three call_and_report calls to get a collection of reports.

src/_pytest/runner.py:L77-L94

def runtestprotocol(item, log=True, nextitem=None):
    # Abbreviated
    reports = []
    reports.append(call_and_report(item, "setup", log))
    reports.append(call_and_report(item, "call", log))
    reports.append(call_and_report(item, "teardown", log))
    return reports

So I'm after modifying the report at the call stage... but still no clue how I get access to the fixture information.

src/_pytest/runner.py:L166-L174

def call_and_report(item, when, log=True, **kwds):
    call = call_runtest_hook(item, when, **kwds)
    hook = item.ihook
    report = hook.pytest_runtest_makereport(item=item, call=call)
    if log:
        hook.pytest_runtest_logreport(report=report)
    if check_interactive_exception(call, report):
        hook.pytest_exception_interact(node=item, call=call, report=report)
    return report

It looks like there are some helper methods for generating a new ExceptionRepresentation so I updated the conftest.py example.

src/_pytest/reports.py:L361

longrepr = item.repr_failure(excinfo)

UPDATE #1 2019-06-26: Thanks to some pointers from @hoefling in the comments I updated my conftest.py.

  • Correctly re-raising the exception using the raise ... from ... form.
  • Override the _vcr_marker to attach the vcr and vcr_cassette fixtures to the request.node which represent that individual test item.
  • Remaining: Get access to the intercepted request from the patched VCRConnection...

UPDATE #2 2019-06-26

It would seem impossible to get at the VCRHTTPConnections that were patched in creating the cassette context manager. I have opened up the following pull request to pass as arguments when the exception is thrown, to then catch and handle arbitrarily down stream.

https://github.com/kevin1024/vcrpy/pull/445

Related

Related questions that are informative but still don't answer this question.

Josh Peak
  • 5,898
  • 4
  • 40
  • 52
  • 2
    I wish all questions on SO were like this. – hoefling Jun 25 '19 at 11:18
  • 1
    Regarding your actual question: you can passthrough the fixture values by attaching them to a global object, usually I use the `config` for this. E.g. redefine `vcr` fixture with the body `request.config._vcr = vcr; return vcr` and access it via `node.session.config._vcr` in the hook. I can describe it in an answer with a working example if you need it. – hoefling Jun 25 '19 at 11:19
  • 1
    Regarding the exception modification: you are on the right track with the `pytest_exception_interact` hook; however, I would not modify the existing `ExceptionInfo` object and create a new one instead. What I don't like in your approach is that the exchanged exception won't match the traceback anymore; I would probably raise a new exception of your own type from the current one in the hook (`raise MyEx('text') from call.excinfo.value`), catch it and inject new excinfo via e.g. `call.excinfo = ExceptionInfo.from_current()`, so at least you preserve the original traceback. – hoefling Jun 25 '19 at 11:28
  • Thanks @hoefling! I'll give that a try tomorrow and circle back. I've managed to intercept the exception and replace it with a new one. This might be the trick to getting the fixture context I need for the new detailed exception. Oh and good tip on raising a new Exception. I didn't like the replacement anyhow. – Josh Peak Jun 25 '19 at 11:31
  • 1
    I'm not sure whether the raising and immediate catch is a good approach either, I can imagine it will look ugly in code. I'll think about something better and report back. – hoefling Jun 25 '19 at 11:34
  • @hoefling the re-raising and appending the fixtures to the node worked thank you! I updated the conftest.py example in the question. Unfortunately I still need access to the patched VCRConnection Request that triggered it all. But progress is appreciated. More spelunking! – Josh Peak Jun 26 '19 at 03:41
  • Hmm, the access to `VCRConnection` looks tricky - have you thought about monkeypatching it yourself? Subclass, override the methods you need that will store `self.cassette` in some global dict besides calling the super method (again under e.g. `session.config._cassettes`, maybe even make it a mapping of test nodeids to cassette objects?). – hoefling Jun 26 '19 at 15:37
  • Besides of that, I can imagine using a `requests.Session` (probably via a `pytest` fixture) and accessing the currently cached connections via available connection pool from session adapters, but am not sure whether it is a viable solution at all and its implementation will look clunky for sure. – hoefling Jun 26 '19 at 15:39

1 Answers1

3

Thanks to comments and guidance in the comments from @hoefling.

I could attach the cassette fixture to the request.node in a conftest.py local plugin overriding the pytest-vcr marker...

@pytest.fixture(autouse=True)
def _vcr_marker(request):
    marker = request.node.get_closest_marker("vcr")
    if marker:
        cassette = request.getfixturevalue("vcr_cassette")
        vcr = request.getfixturevalue("vcr")
        request.node.__vcr_fixtures = dict(vcr_cassette=cassette, vcr=vcr)
    yield

But I needed more than the cassette to get to my solution.

Ingredients

Recipe

Get latest VCRpy

These patches were released in vcrpy v2.1.0

pip install vcrpy==2.1.0

Override the pytest_exception_interact hook

In the root of your test directory create a conftest.py to create a local plugin that overrides the pytest_exception_interact hook.

@pytest.hookimpl(hookwrapper=True)
def pytest_exception_interact(node, call, report):
    """Intercept specific exceptions from tests."""
    if report.when == "call" and isinstance(call.excinfo.value, CannotOverwriteExistingCassetteException):
        __handle_cassette_exception(node, call, report)

    yield

Extract the Cassette and the Request from the exception.

# Define new exception to throw
class RequestNotFoundCassetteException(Exception):
   ...

def __handle_cassette_exception(node, call, report):
    # Safely check for attributes attached to exception
    vcr_request = None
    cassette = None
    if hasattr(call.excinfo.value, "cassette"):
        cassette = call.excinfo.value.cassette
    if hasattr(call.excinfo.value, "failed_request"):
        vcr_request = call.excinfo.value.failed_request

    # If we have the extra context available...
    if cassette and vcr_request:

        match_properties = [f.__name__ for f in cassette._match_on]
        this_req, req_str = __format_near_match(cassette.requests, vcr_request, match_properties)

        try:
            raise RequestNotFoundCassetteException(f"{this_req}\n\n{req_str}\n") from call.excinfo.value
        except RequestNotFoundCassetteException as e:
            call.excinfo._excinfo = (type(e), e)
            report.longrepr = node.repr_failure(call.excinfo)
Josh Peak
  • 5,898
  • 4
  • 40
  • 52