When Visual Studio downloads solutions from TFS (double click sln file in solution explorer) it appears to download files one by one and load them up. Unfortunately it seems to try opening project files before it downloads the .nuget directory, which is why it can't find the file. The last thing it appears to do is download that file, which explains why it is on disk but gave the error. If you reopen the solution it's already there and works fine.
When TFS Build server downloads a solution to build, it does so on the solution directory instead. Which means it will get the .nuget directory before it tries to build so it shouldn't cause issues on the build server.
I believe this is a bug in Visual Studio, it really should download all the solution items first. Although it would be nice if it had the same behaviour as TFS Builds.
A work around for this issue is to get latest on the solution folder before you open the solution for the first time. Not ideal but it works.
I'd also suggest logging a bug with either the nuget or visual studio team, however I suspect they're probably already aware of it.