Your application are trying to referencing dlls Microsoft.TeamFoundation.VersionControl.Client
from the GAC.
One solution is just as you done, install VS 2010 on the development environment and referencing the Microsoft.TeamFoundation dlls from the GAC. VS 2010 installed version 10 of the Team Foundation dlls into the development GAC.
However, this will cause you need to also install VS2010 on the build agent. A better solution is using Nuget Package. These have been moved into a Nuget package.
They're also stored under the "CommonExtensions" folder of your Visual
Studio installation, but these are not meant to be referenced anymore,
that's what the NuGet package is for.
C:\Program Files (x86)\Microsoft Visual Studio 1x.0\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer
Take a look at this similar question: Where can I find Microsoft.TeamFoundation.VersionControl.Client.dll in Visual Studio 2015 installation?