requests-mock: how can I match POSTed payload in a mocked endpoint

Question

What I've Done

I've written an authentication class for obtaining an application's bearer token from Twitter using the application's API Key and its API key secret as demonstrated in the Twitter developer docs.

I've mocked the appropriate endpoint using requests_mock this way:

@pytest.fixture
def mock_post_bearer_token_endpoint(
    requests_mock, basic_auth_string, bearer_token
):
    requests_mock.post(
        "https://api.twitter.com/oauth2/token",
        request_headers={
            "Authorization": f"Basic {basic_auth_string}",
            "Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
        },
        json={"token_type": "bearer", "access_token": f"{bearer_token}"},
    )

And my test method is :

@pytest.mark.usefixtures("mock_post_bearer_token_endpoint")
def test_basic_auth(api_key, api_key_secret, bearer_token):
    response = requests.post(
        'https://api.twitter.com/oauth2/token',
        data={"grant_type": "client_credentials"},
        auth=TwitterBasicAuth(api_key, api_key_secret),
    )
    assert response.json()['access_token'] == bearer_token

(Where TwitterBasicAuth is the authentication class I wrote, and the fixture basic_auth_string is a hardcoded string that would be obtained from transforming the fixtures api_key and api_key_secret appropriately).

And it works.

The Problem

But I'm really bothered by the fact that the mocked endpoint doesn't check the payload. In this particular case, the payload is vital to obtain a bearer token.

I've combed through the documentation for requests_mock (and responses, too) but haven't figured out how to make the endpoint respond with a bearer token only when the correct payload is POSTed.

Please help.

Answer 1

Updated Answer

I went with gold_cy's comment and wrote a custom matcher that takes a request and returns an appropriately crafted OK response if the request has the correct url path, headers and json payload. It returns a 403 response otherwise, as I'd expect from the Twitter API.

@pytest.fixture
def mock_post_bearer_token_endpoint(
    requests_mock, basic_auth_string, bearer_token
):
    def matcher(req):
        if req.path != "/oauth2/token":
            # no mock address
            return None
        if req.headers.get("Authorization") != f"Basic {basic_auth_string}":
            return create_forbidden_response()
        if (
            req.headers.get("Content-Type")
            != "application/x-www-form-urlencoded;charset=UTF-8"
        ):
            return create_forbidden_response()
        if req.json().get("grant_type") != "client_credentials":
            return create_forbidden_response()

        resp = requests.Response()
        resp._content = json.dumps(
            {"token_type": "bearer", "access_token": f"{bearer_token}"}
        ).encode()
        resp.status_code = 200

        return resp

    requests_mock._adapter.add_matcher(matcher)
    yield

def create_forbidden_response():
    resp = requests.Response()
    resp.status_code = 403
    return resp

---

Older Answer

I went with gold_cy's comment and wrote an additional matcher that takes the request and checks for the presence of the data of interest in the payload.

@pytest.fixture(name="mock_post_bearer_token_endpoint")
def fixture_mock_post_bearer_token_endpoint(
    requests_mock, basic_auth_string, bearer_token
):
    def match_grant_type_in_payload(request):
        if request.json().get("grant_type") == "client_credentials":
            return True
        resp = Response()
        resp.status_code = 403
        resp.raise_for_status()

    requests_mock.post(
        "https://api.twitter.com/oauth2/token",
        request_headers={
            "Authorization": f"Basic {basic_auth_string}",
            "Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
        },
        json={"token_type": "bearer", "access_token": f"{bearer_token}"},
        additional_matcher=match_grant_type_in_payload,
    )

I opted to raise an Http403 error (instead of just returning False) in order to reduce the cognitive load of determining the reason exceptions are raised — returning False would lead to a requests_mock.exceptions.NoMockAddress being raised, which I don't think is descriptive enough in this case.

I still think there's a better way around this, and I'll keep searching for it.

Answer 2

I think the misconception here is that you need to put everything in the matcher and let NoMatchException be the thing to tell you if you got it right.

The matcher can be the simplest thing it needs to be in order to return the right response and then you can do all the request/response checking as part of your normal unit test handling.

additional_matchers is useful if you need to switch the response value based on the body of the request for example, and typically true/false is sufficient there.

eg, and i made no attempt to look up twitter auth for this:

import requests
import requests_mock

class TwitterBasicAuth(requests.auth.AuthBase):

    def __init__(self, api_key, api_key_secret):
        self.api_key = api_key
        self.api_key_secret = api_key_secret

    def __call__(self, r):
        r.headers['x-api-key'] = self.api_key
        r.headers['x-api-key-secret'] = self.api_key_secret
        return r


with requests_mock.mock() as m:
    api_key = 'test'
    api_key_secret = 'val'

    m.post(
        "https://api.twitter.com/oauth2/token",
        json={"token_type": "bearer", "access_token": "token"},
    )

    response = requests.post(
        'https://api.twitter.com/oauth2/token',
        data={"grant_type": "client_credentials"},
        auth=TwitterBasicAuth(api_key, api_key_secret),
    )

    assert response.json()['token_type'] == "bearer"
    assert response.json()['access_token'] == "token"
    assert m.last_request.headers['x-api-key'] == api_key
    assert m.last_request.headers['x-api-key-secret'] == api_key_secret

https://requests-mock.readthedocs.io/en/latest/history.html