If your access to the private repository is removed, any forks of that private repository will be deleted as well.
See https://github.com/blog/2034-greater-control-over-forks-of-your-private-repositories :
July 2, 2015 New Features
Previously, if you removed collaborator permissions from someone
contributing to a private repository on your personal account, that
person would retain their fork (if they had created one).
Today, we're changing that behavior: if you remove a collaborator's
permissions from one of your private repositories, their fork will be
deleted, giving you greater control over access to your private code.
This matches the behavior of organization-owned forks, which hasn't
changed.
The updated documentation reflects this (https://help.github.com/articles/removing-a-collaborator-from-a-personal-repository/#deleting-forks-of-private-repositories):
Deleting forks of private repositories
While forks of private repositories are deleted when a collaborator is
removed, the person will still retain any local clones of your
repository.