I don't really understand what you're asking... you cannot use a TFS client without a TFS server to communicate with.
VisualStudio can talk to any git repository. It need not be hosted in TFS, it need not even be hosted, it could just be a local repository on your machine. But if that git repository is not hosted by TFS, then you cannot work with other TFS functionality like work item tracking.
Team Foundation Server, however, is a lot more than just version control. If you want to take advantage of TFS functionality for things like agile project management and work item tracking, then you'll need a TFS client. If you want to link a git commit to a work item, then you'll need to have that git commit in a git repository in TFS.
It sounds like you're asking if you can use TFS but not use the version control that's inbuilt to TFS. Technically, the answer is yes, but obviously you would lose any functionality in TFS that depends on version control. You would lose the ability to link work items to git commits (and thus, lose the ability to quickly identify the lines of code that fix a bug.) You would lose the ability to perform continuous integration builds (and thus, you would lose the ability to identify which build contained a new feature or bug fix.)
If that's what you're asking, then this probably isn't a technical question for us but a policy question for your management.