I have recently upgraded TFS 2017 update 3 to azure Devops server 2020 update 1.1. After installing a new agent in Azure Devops server 2020, the agent installation is successful but the event viewer gives below error after every 50mins. But the very next minute the event viewer shows: Agent reconnected. I have tried installing the agent using PAT and integrated as well but same error appears. The agent is not causing any issue, it builds and deploys correctly. I am using windows server 2019 and IIS 10. There were no existing agents to test for this error and this upgrade is done on Lab. Before i proceed with production upgrade, i wanted to find the root cause of this error.
What could be the reason for this error? I have tried almost all the solutions online.
Error Snippet:
[2022-04-26 09:55:37Z WARN VisualStudioServices] GET request to https://tfs-infosys.blfdev.lab/tfs/_apis/distributedtask/pools/6/messages?sessionId=ae9f5c0e-588b-4c92-8b2fe9b6a5e3b2a8 timed out after 60 seconds.
[2022-04-26 09:55:37Z ERR MessageListener] Catch exception during get next message.
[2022-04-26 09:55:37Z ERR MessageListener] System.TimeoutException: The HTTP request timed out after 00:01:00.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial) at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[2022-04-26 09:55:37Z INFO MessageListener] Retriable exception: The HTTP request timed out after 00:01:00.
[2022-04-26 09:55:37Z ERR Terminal] WRITE ERROR: 2022-04-26 09:55:37Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
[2022-04-26 09:55:37Z INFO AgentServer] Refresh MessageQueue VssConnection to get on a different AFD node.
[2022-04-26 09:55:37Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2022-04-26 09:55:37Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2022-04-26 09:55:38Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2022-04-26 09:55:38Z INFO MessageListener] Sleeping for 25.931 seconds before retrying.
[2022-04-26 09:56:53Z INFO Terminal] WRITE LINE: 2022-04-26 09:56:53Z: Agent reconnected.
[2022-04-26 10:06:54Z WARN VisualStudioServices] Authentication failed with status code 401.
X-TFS-ProcessId: feb6477a-c4d5-430e-a011-1ba53e898d8d
ActivityId: a2d19d52-ab72-4c5c-b860-8e9d5ec34c9d
X-TFS-Session: e1d0318d-7539-4cb8-9e76-49e1decf910b
X-VSS-E2EID: db35685b-98ba-409e-8c59-cb0e0482d691
X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
X-TFS-ServiceError: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
WWW-Authenticate: Bearer, Basic realm="https://tfs-infosys.blfdev.lab/tfs/", Negotiate, NTLM
X-Powered-By: ASP.NET
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
Date: Tue, 26 Apr 2022 10:06:54 GMT
[2022-04-26 10:06:54Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\testlab\.credentials_rsaparams
[2022-04-26 10:06:54Z INFO VisualStudioServices] AAD Correlation ID for this token request: Unknown
[2022-04-26 10:08:34Z INFO MessageListener] No message retrieved from session 'ae9f5c0e-588b-4c92-8b2f-e9b6a5e3b2a8' within last 30 minutes.
[2022-04-26 10:10:24Z WARN VisualStudioServices] GET request to https://tfs-infosys.blfdev.lab/tfs/_apis/distributedtask/pools/6/messages?sessionId=ae9f5c0e-588b-4c92-8b2f-e9b6a5e3b2a8 timed out after 60 seconds.
[2022-04-26 10:10:24Z ERR MessageListener] Catch exception during get next message.
[2022-04-26 10:10:24Z ERR MessageListener] System.TimeoutException: The HTTP request timed out after 00:01:00.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[2022-04-26 10:10:24Z INFO MessageListener] Retriable exception: The HTTP request timed out after 00:01:00.
[2022-04-26 10:10:24Z ERR Terminal] WRITE ERROR: 2022-04-26 10:10:24Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
[2022-04-26 10:10:24Z INFO AgentServer] Refresh MessageQueue VssConnection to get on a different AFD node.
[2022-04-26 10:10:24Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2022-04-26 10:10:24Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2022-04-26 10:10:24Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2022-04-26 10:10:24Z INFO MessageListener] Sleeping for 26.055 seconds before retrying.
[2022-04-26 10:11:40Z INFO Terminal] WRITE LINE: 2022-04-26 10:11:40Z: Agent reconnected.
[2022-04-26 10:26:01Z WARN VisualStudioServices] GET request to https://tfs-infosys.blfdev.lab/tfs/_apis/distributedtask/pools/6/messages?sessionId=ae9f5c0e-588b-4c92-8b2f-e9b6a5e3b2a8 timed out after 60 seconds.
[2022-04-26 10:26:01Z ERR MessageListener] Catch exception during get next message.
[2022-04-26 10:26:01Z ERR MessageListener] System.TimeoutException: The HTTP request timed out after 00:01:00.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[2022-04-26 10:26:01Z INFO MessageListener] Retriable exception: The HTTP request timed out after 00:01:00.
[2022-04-26 10:26:01Z ERR Terminal] WRITE ERROR: 2022-04-26 10:26:01Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
[2022-04-26 10:26:01Z INFO AgentServer] Refresh MessageQueue VssConnection to get on a different AFD node.
[2022-04-26 10:26:01Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2022-04-26 10:26:01Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2022-04-26 10:26:01Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2022-04-26 10:26:01Z INFO MessageListener] Sleeping for 22.767 seconds before retrying.
[2022-04-26 10:27:14Z INFO Terminal] WRITE LINE: 2022-04-26 10:27:14Z: Agent reconnected.
