Hey ash, here is what I would to to investigate the problem: Go to your build machine (aka build server) and check the state of the machine, the build service, the workspaces where your source code is built by TFS build.
It looks like this error occurs during build time, and based on the description of the error code from the other SO thread Preet provided, it's possible that your build machine is messed up.
Now you said that you have poor performance with TFS server. What exactly is performing poorly? Are your TFS server and build machine separate or stacked together? There is a memory fragmentation problem with the build service in TFS 2010 RTM (which is improved much in SP1), especially when you build large projects. I'm wondering if the error here could be related to that, so please also check the memory usage of TfsBuildServiceHost.exe (aka the build service).
Good luck!