What I want to achieve is to see the changes (diff only) for a single file per commit.
There is no existing Rest API to meet your needs. But you could refer to the following steps to get the content of the git diff.
Step1: You could use the Rest API to get the commit id.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits?api-version=5.0
Step2: You could use the Rest API to get the commit by commit id.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits/{commitId}?api-version=5.0
In the Rest API Result, you need to record the value of parentsid.
Step3: You could use the Rest API to Get the file path.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits/{commitId}/changes?api-version=6.0
Step4: You could use the following API to get the diff content.
Post https://dev.azure.com/Organization/Project /_api/_versioncontrol/fileDiff?__v=5&diffParameters={value}&repositoryId={repositoryid}
The {value} is Json type.
Here is an example:
{"originalPath":"filepath","originalVersion":"Parentsid","modifiedPath":"filepath","modifiedVersion":"commitid","partialDiff":true,"includeCharDiffs":true}
You could add the value to the API URL.
Then run the API and the result will contains the git diff content. (2 means remove, 1 means add)
Here is a result sample:

Here is the ticket, you could refer to it.