Files aren’t stored on the file system like they would be when git is running on your local machine. Instead they are stored in the TFS SQL server database.
If there are many users who frequently modify and commit the changes, the tables you mentioned will be growing accordingly.
You can refer to this article to see how the git files are stored, then figure out what caused the Database growth.
Normally, the files you committed/attached will be stored in tbl_Content Table, so, you can also check if there is the abnormal growth for that table based on this article.
To clean up and reduce the size of the TFS database, there are a couple of ways to clean your TFS database:
You can also reference this article:
https://social.technet.microsoft.com/wiki/contents/articles/19932.reduce-the-size-of-tfs-databases-after-cleaning-some-source-control-files.aspx