Checking NuGet packages into Version Control is not a good idea. You need to remove them, and then use NuGet restore:
If you work with the vNext build system in VSO, all you need to do is to check the Restore NuGet Packages checkbox in the build definition, and choose the .sln file you would like to build.

If you work with one XAML build definition, there are some steps (described here) you need to follow in order to have these NuGet packages restored during the VSO (TFS) build process.
- Add following items to the solution. (Content of the nuget.config and .tfignore file can be found here)

Add one build.proj file under the root path of the solution folder. (Content of the build.proj file can be found here)
Create one folder named tools under the root path of the solution folder. Create NuGet sub-folder under tools folder, download and save nuget.exe under tools\NuGet path.
Check in nuget.config, .tfignore, build.proj and tools\NuGet\nuget.exe into TFS version control.
Modify the build definition to choose to build the build.proj file.

Then you will have NuGet packages restored successfully during the TFS build process.