Builds fail due to timeout during Get Sources step in TFS 2018

Question

Our TFS build agents are having seemingly random issues with the Get Sources step where the git fetch command hangs and times out after an hour.

The last portion of the logs shows it got everything it needed but decided to just stop until it timed out.

2018-04-12T10:00:11.9145204Z Resolving deltas:  97% (123/126)   
2018-04-12T10:00:11.9145204Z Resolving deltas:  99% (125/126)   
2018-04-12T10:00:11.9145204Z Resolving deltas: 100% (126/126)   
2018-04-12T10:00:11.9145204Z Resolving deltas: 100% (126/126), completed with 17 local objects.
2018-04-12T11:00:03.8383793Z ##[debug]Re-evaluate condition on job cancellation for step: 'Get Sources'.
2018-04-12T11:00:04.1196536Z ##[error]The operation was canceled.
2018-04-12T11:00:04.1352781Z ##[debug]System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.VisualStudio.Services.Agent.ProcessInvoker.<ExecuteAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.GitCommandManager.<ExecuteGitCommandAsync>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.GitCommandManager.<GitFetch>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.GitSourceProvider.<GetSourceAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension.<GetSourceAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.JobExtensionRunner.<RunAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.<RunStepAsync>d__1.MoveNext()
2018-04-12T11:00:04.1352781Z ##[section]Finishing: Get Sources

We have 4 build agents setup and usually one or two have this problem each night during builds.

When I look into the agent's directory and run a git status command, I get a message saying:

Rename from 'C:\BuildAgent1\_work\1\s\.git\index.lock' to C:\BuildAgent1\_work\1\s\.git\index' failed. Should I try again? (y/n)

Entering 'y' just causes that message to repeat. Looking into the .git/ folder, there is no index.lock file present.

Issuing another build manually and the build gets past the Get Sources step without issue.

Why are the builds hanging after fetching everything?

Just try to set **clean** to `true` and **Clean options** select `Sources directory` under Get Sources, then check if that works. — Andy Li-MSFT, Apr 13 '18 at 09:46
I'll give that a shot and see how the builds work more reliably. Some of the build definitions already have the clean option set to true and has the sources directory option set which have failed previously. — shaboinkin, Apr 13 '18 at 15:45
You can also try to delete the `index` and `index.lock` file directly on agent machine under the `.git` folder (Something like `C:\Agent\_work\2\s\.git`). Then check it again. You can also reference [this thread](https://stackoverflow.com/questions/13635429/git-rename-from-index-lock-to-index-failed) for the troubleshooting. — Andy Li-MSFT, Apr 16 '18 at 09:41
The thing is, there was no index.lock file. But from looking at today's nightly builds, this particular problem did not happen after changing all definitions to clean, so thanks for that suggestion. I ran into another (probably unrelated) issue, which I am engaged with microsoft's support at the moment. — shaboinkin, Apr 16 '18 at 15:01
Yes, if `index.lock` existing just delete it, if not just delete the `index` file. I posted the solution as an answer below. You can check and if that works please [Accept it as an Answer](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work), This can be beneficial to other community members reading this thread. — Andy Li-MSFT, Apr 17 '18 at 01:54

score 1 · Accepted Answer · answered Apr 17 '18 at 01:55

So, the solution for this issue is cleaning the sources directory.

Set Clean to true and Clean options select Sources directory under Get Sources step.
Alternatively you can delete the source folder manually on agent machine. (e.g.: C:\Agent\_work\2\s\, basically delete the .git folder will resolve the issue C:\Agent\_work\2\s\.git)

Builds fail due to timeout during Get Sources step in TFS 2018

1 Answers1