TFS 2017 agent error (lost communication with the server) - tfs

I have TFS2017 installed on Windows Server 2008 R2 virtual machine. Yes, it's a weird combination, but it is what it is.
Also I installed agent(vsts-agent-win7-x64-2.117.2) on the same virtual machine. Agent is successfully installed as a service, it's up and running, I can see it in Default pool(showing as Online, Idle).
The problem starts when I request new build. It's always failing with error:
The agent agent3 lost communication with the server. Verify the machine is running and has a healthy network connection. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610
I also checked logs in _diag folder:
[2018-02-27 09:33:48Z INFO JobDispatcher] Retrying lock renewal for jobrequest 9. Job is still locked for: 10 seconds.
[2018-02-27 09:33:58Z ERR JobDispatcher] Catch exception during renew agent jobrequest 9.
[2018-02-27 09:33:58Z ERR JobDispatcher] Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Not Found
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponse(HttpResponseMessage response)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__451.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__271.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__261.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Listener.JobDispatcher.d__12.MoveNext()
[2018-02-27 09:33:58Z INFO JobDispatcher] Lock renewal has run out of retry, stop renew lock for jobrequest 9.
[2018-02-27 09:33:58Z INFO JobDispatcher] Unable to renew job request for job 88ba8e32-bdca-402c-8a94-d1e39f95819b for the first time, stop dispatching job to worker.
Seems that agent can't update item in job queue for some reason.
I checked firewall - it's off, and moreover TFS and agent are installed on the same machine.
Any ideas?
Update.
Screenshot of Services:
I tried to create different agents running under different users - NETWORK Service or current user, which is logged into TFS. Also I always check that service is started.
Update 2.
I tried to install another agent on another VM in the same network. It was installed and appeared in Default pool. But when it goes to build, same error occurs. Btw, another VM works under Windows Server R2.

Wow,
That was a real nightmare. But it's finally solved.
The root of the issue is a tool called UrlScan. It was installed on the server and by default it blocks all http verbs except GET/POST/OPTIONS.
Enabling all known verbs didn't help, because seems that Agent use some custom http header. The only way was to uninstall UrlScan completely.
Good luck.

Related

How to run Docker Desktop with Windows Containers on Amazon AWS EC2 Windows (or is this not possible)?

I have a DESKTOP machine running Windows 10 running Windows 10, Microsoft 365, Docker Desktop running in Windows Containers mode.
I have an Amazon EC2 machine running Windows Server 2019 and have installed Microsoft 365 on there too and access via RDP.
Essentially what I'd like to is create an EC2 machine which allows me to run Microsoft 365 AND Docker Desktop running in Windows Containers mode.
I've tried creating AWS EC2 instance and installing Docker for Desktop but I get lots of cryptic errors. (see below).
I tried using this AMI from Microsoft with Hyper-V built in https://aws.amazon.com/marketplace/server/procurement?productId=d94e0325-6111-4c69-8c29-87e784a6d0aa
[Or should I be using Docker Enterprise for this?]
[Or is it not possible to get a Windows machine which will run Docker Windows Containers on AWS at all?]
[Or do I need to choose a specific type of machine size [t4.large or whatever] - I have experienced on Microsoft Azure before that running Docker Desktop was possible only on machines of a certain spec, which allowed "virtualization on top of virtualization"]
System.InvalidOperationException:
job failed with message: 'DockerDesktopVM' failed to start. (Virtual machine ID 677B7F36-8A44-42CE-AD29-7884DDCE164C)
The Virtual Machine Management Service failed to start the virtual machine 'DockerDesktopVM' because one of the Hyper-V components is not running (Virtual machine ID 677B7F36-8A44-42CE-AD29-7884DDCE164C).
at Docker.Backend.HyperVInfrastructure.HandleUpdatedJob(CIMConcreteJob updatedJob) in C:\workspaces\PR-15633\src\github.com\docker\pinata\win\src\Docker.Backend\HyperV\HyperVInfrastructure.cs:line 762
at Docker.Backend.HyperVInfrastructure.<HandleMaybeAsyncJob>d__38.MoveNext() in C:\workspaces\PR-15633\src\github.com\docker\pinata\win\src\Docker.Backend\HyperV\HyperVInfrastructure.cs:line 696
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Docker.Backend.HyperVInfrastructure.<StartVMAsync>d__30.MoveNext() in C:\workspaces\PR-15633\src\github.com\docker\pinata\win\src\Docker.Backend\HyperV\HyperVInfrastructure.cs:line 544
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Docker.Backend.HyperV.<StartAsync>d__6.MoveNext() in C:\workspaces\PR-15633\src\github.com\docker\pinata\win\src\Docker.Backend\HyperV\HyperV.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Docker.Backend.HttpAPI.HyperVController.<StartAsync>d__7.MoveNext() in C:\workspaces\PR-15633\src\github.com\docker\pinata\win\src\Docker.Backend\HttpAPI\HyperVController.cs:line 42
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.TaskHelpersExtensions.<CastToObject>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
Update 2022
You can now install WSL (Windows Subsystem for Linux) on Windows Server 2019/2022 OS
https://learn.microsoft.com/en-us/windows/wsl/install-on-server
Docker Desktop is not supported on Windows Server 2019 OS host, Docker Desktop is only supported on Windows 10 host, Mac and planned for Linux Desktop ( there are kernel difference b/w Windows server host and Windows 10 desktop)
With Windows Server 2019 you can install Docker Engine Enterprise (now known as Mirantis Container Runtime (MCR) ) and run windows containers.
Following documentation from Microsoft has information on installing Docker Engine on Windows Server :
https://learn.microsoft.com/en-us/virtualization/windowscontainers/quick-start/set-up-environment?tabs=Windows-Server
If you want to install via script checkout this link (Docker Engine Enterprise now known as Mirantis Container Runtime):
https://docs.mirantis.com/containers/v3.1/mcr-deployment-guide/mcr-windows.html
You can also checkout this blog here and follow the steps to install Docker Engine on Windows Server host :
https://mechdeveloper.medium.com/windows-server-2016-containers-bc0baa2222c1

Right click publish custom docker images to existing ACR with Visual Studio 2019

I get below error when I try to publish DotNet 3.1 web app via Visual Studio 2019 to ACR.
I followed this link's instructions to publish but instead of creating new one, I selected existing ACR from list. I also looked in Visual studio to set ACR credentials with any special switches shown in the error, but could not find it.
My Docker Desktop is of version - 2.5.0.1 and VS version is - 16.7.6
Getting below error:
Publish has encountered an error.
Running the docker.exe login command failed.
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
A diagnostic log has been written to the following location:
C:\Users\Admin\AppData\Local\Temp\tmp2719.tmp
Below is full stack trace file which was provided by Visual Studio after failing:
26-11-2020 08:40:30 PM
Microsoft.WebTools.Azure.Publish.Docker.DockerCommandException:
Running the docker.exe login command failed.
WARNING! Using --password via the CLI is insecure. Use
--password-stdin. at Microsoft.WebTools.Azure.Publish.Docker.DockerOperations.ThrowDockerCommandError(String
dockerCommand) at
Microsoft.WebTools.Azure.Publish.Docker.DockerOperations.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Microsoft.WebTools.Azure.Publish.Docker.DockerPublish.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Microsoft.WebTools.Azure.Publish.ContainerRegistry.ManageContainerRegistry.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Microsoft.WebTools.Azure.Publish.PublishProviders.ContainerRegistryProfileVisual.d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Microsoft.Publish.Framework.ViewModel.ProfileSelectorViewModel.d__180.MoveNext()
According to the message you provided, it shows the docker login failed. As I know when you log in to the Visual Studio, then you can select the Docker support and publish the image to the existing Azure Container Registry if you have the permission to control the ACR. There is no option to set the credential for the ACR. But it gives the login failed error. So the possible reason is that your existing ACR does not enable the admin user. Try to enable the admin user for the ACR and then try to publish again.

Web deploy fails s the server experienced an issue processing the request [duplicate]

This question already has an answer here:
WebDeploy not listening on port 8172
(1 answer)
Closed 2 years ago.
Web deploy has been working fine for me via Microsoft Visual Studio Enterprise 2019. Yesterday, it failed. I've made no changes to the server, although Windows Updates are on
4>MSDEPLOY(0,0): Error : (04/03/2020 18:45:08) An error occurred when the request was processed on the remote computer.
4>MSDEPLOY(0,0): Error : The server experienced an issue processing the request. Contact the server administrator for more information.
4>MSDEPLOY(0,0): Error count: 1.
I tried to FTP my site, which connects fine.
I turned off the firewall on my VPS, and the same error message is shown.
The website builds fine locally.
The .tmp file shows the following stack
04/03/2020 18:09:20
System.AggregateException: One or more errors occurred. ---> System.Exception: Build failed. Check the Output window for more details.
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Microsoft.WebTools.Publish.PublishService.VsWebProjectPublish.<>c__DisplayClass43_0.<PublishAsync>b__3()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Publish.Framework.ViewModel.ProfileSelectorViewModel.<RunPublishTaskAsync>d__213.MoveNext()
---> (Inner Exception #0) System.Exception: Build failed. Check the Output window for more details.<---
System.Exception: Build failed. Check the Output window for more details.
I don't know how to debug this further
I am clueless as to why but, it seems the fix was simply to accept connections in the firewall on Port 8172

Can't publish to Docker Hub from Visual Studio

Docker .NetCore app running in a container runs fine in Visual Studio. When I try to publish to Docker Hub, it returns an “unknown” error (log below).
The profile is set to my user and pw (for Docker Hub). When viewing in VS, on the Publish screen there is a url to the base Docker Hub directory. Clicking on it returns a “not found” error from Docker Hub. Any ideas on what would cause this? I did check to make sure I was logged in with Docker Desktop, if that matters.
From VS:
Publish has encountered an error.
Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details.
Log: at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Microsoft.Publish.Framework.ViewModel.ProfileSelectorViewModel.d__213.MoveNext()
---> (Inner Exception #0) System.Exception: Publish has encountered an error. We were unable to determine the cause of the error. Check the
output log for more details. <---
System.Exception: Publish has encountered an error. We were unable to
determine the cause of the error. Check the output log for more
details.
===================

TFS 2017 server required firewall settings for Build agent

I have TFS 2017 on Windows Server 2016 Standard.
I am now setting up Build agent on Windows 10 pro using PAT token authentication. (having "Agent Pools (read, manage)", "Agent Pools (read)", "Build (read"), "Code (read)", "Build (read and execute)", though this part seeems to be fine.
I am getting close to the very end of the configuration
Testing agent connection.
An error occured while sending the request.
Log file says
[2017-06-27 14:58:18Z ERR Agent] System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Der Servername oder die Serveradresse konnte nicht verarbeitet werden
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.WinHttpHandler.d__101.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.d__58.MoveNext()
More by chance (since this is nowhere mentioned as required) I disabled the firewall on the server and now the connection and configuration of the build agent goes to the very end.
I definitely have 443 and 80 open, I am using TFS via https/443 and also entered that in the build agent config. In the TFS console, I see :80 and :443, nothing like :8080.
What else do I need to consider?
TFS server is in a domain, build agent is not, https certificate is self-signed and added to Root CAs on build agent.
After opening all TCP ports outgoing and incoming and doing some "netstat -a", I also tried UDP.
After some testing and "binary search", I opened UDP 1434 and now the
Testing agent connection
results in asking for the _work folder.
Update:
After ~ 5 builds I got "An error occured while sending the request.. retry every 15 seconds..", looks like opening 5353 and 5355 UDP fixes this for now.

Resources