I am using TFS to execute a nightly build that includes several steps that use the TFS Test Agent. I am running the latest version of TFS/Test Agent(2015 - Update 3) and there are no other builds being run at this time. Often(maybe half the time), when the nightly job is run the step "Visual Studio Test Agent Deployment" fails with the following error:
The job has been abandoned because agent Agent-XXX did not renew the
lock. Ensure agent is running, not sleeping, and has not lost
communication with the service.
This is due to the error found in the Test Agent's log file(under _diag):
The session for this agent already exists. Sleeping for 30 seconds
before next retry.
Microsoft.TeamFoundation.DistributedTask.WebApi.TaskAgentSessionConflictException:
The task agent Agent-XXX already has an active session for owner XXX.
This issue is directly referenced here, and indirectly talked about here.
The solution I've found to this issue is to restart the server that the test agent is running on, this clears any dead sessions, and after the server starts back up, the tests run just fine. I think this is effectively what is being done in the previously mentioned post. The result of resetting the configs is that the service is restarted.
While being presented as a solution in the linked article, it is only temporary. Even after the server has been restarted and the build runs successfully, the next day the issue will again reappear necessitating manual intervention to get the build to run.
I could schedule a task to reset the service or even restart the server directly before the nightly build is run, but it strikes me as a bandage rather than a fix. Has anyone experienced this issue before, and if so is there any way to prevent it from occurring in the first place?
Update 1
I simply set up a build that runs 5 minutes before my main tests that runs a Bat script to restart all my servers hosting my test agents. This is a workaround, but one that seems to resolve the issue. Hopefully someday someone can come up with a better solution than this, but for now, it's how I have to run automated testing in TFS.
Update 2
I have three servers now, all three exhibit the same issue, though it is hard to pin down exactly when it occurs. Scaling up the workaround without creating downtime it proving to be quite challenging.
Update 3
A better day came, I upgraded TFS to 2018, and the build agent to the latest version, this issue no longer occurs, I think its a bug in the old build agent. I still don't have a solution for the original version of the build agent...
t sounds like a process Agent.Listener.exe was running under somewhere on the machine, maybe as a service (not a logged in user session).
note, if an agent process is abruptly terminated while it has an active session, the session will eventually timeout (after 5 minutes i think). and on startup, if an agent encounters session conflict then it will retry for up to 5.5 minutes i think before giving up (enough time for an abruptly terminated session to expire).
i'm going to go ahead and close this and assume a process was running somewhere. we havent had any issues in this area and haven't heard any other reports, so i dont think there is an issue here with the agent. if you find a repro, or it looks like i'm wrong then please reopen.
Related
Here is the thing, I have got a program that may get stuck sometimes, and when it happens I need to reboot my machine.
So, I want to reboot my Jenkins slave when the program gets stuck then continue to execute the rest of my program without marking the whole job as failed.
Can Anyone tell me how to do that?
Actually I wanted to add this as a comment but I don't have enough reputation.
You may want to use Restart from stage feature as documented here
We're running TFS 2015 update 2. When a build starts, it says "Waiting for an agent" then, 2 seconds after, the build fails. I looked whether the agent pools were running, and all of them were on green (as I understand this is expected). Also, I looked whether the TFSJobsAgent was running, and it's ok. If I download the zip log from de build process, it is empty, so i don't know what i'm doing wrong. i tried to create a new agent pool, and run the build process on that agent, but i got the same result.
Any ideas on how to tackle/solve this?
PS: All the builds were working fine a week ago.
The build service account needs to be a member of the Build Service Accounts group.
You also need to check whether the build agent service is running.
I am trying to run some automated acceptance tests on a windows VM but am running into some problems.
Here is what I want, a job which runs on a freshly reverted VM all the time. This job will get an MSI installer from an upstream job, install it, and then run some automated tests on it, in this case using robotframework (but that doesn't really matter in this case)
I have setup the slave in the vSphere plugin to only have one executor and to disconnect after one execution. On disconnect is shutsdown and reverts. My hope was this meant that it would run one Jenkins job and then revert, the next job would get a fresh snapshot, and so would the next and so on.
The problem is if a job is in queue waiting for the VM slave, as soon as the first job finishes the next one starts, before the VM has shutdown and reverted. The signal to shutdown and revert has however been sent, so the next job is almost immedieatly failed as the VM shuts down.
Everything works fine as long as jobs needing the VM aren't queued while another is running, but if they are I run into this problem.
Can anyone suggest a way to fix this?
Am I better off using vSphere build steps rather than setting up a build slave in this fashion, if so how exactly do I go about getting the same workflow to work using buildsteps and (i assume) pipelined builds.
Thanks
You can set a 'Quiet period' - it's in Advanced Project Options when you create a build. You should set it at the parent job, and this is the time to wait before executing the dependent job
If you'll increase the wait time, the server will go down before the second job starts...
Turns out the version of the vSphere plugin I was using was outdated, this bug problem is fixed in the newer version
Our Release Managemeng has a job that is stuck "In Progress".
The error is
Communication with the deployer was lost during the deployment. Please
make sure (1) the deployer machine has not rebooted during
installation and (2) the component timeout is sufficient to copy the
files from the drop location to the deployer machine and install the
package.
I can't stop or abandon the release. The buttons are all disabled. How can I kill this?
From the Release Manager, go to the Release tab. Enter in the details of the actual release, go to the step that is pending and you will see a a "Stop" Button at the top. That will stop the step and change the step of the Release.
Is the build stuck? Can you restart the build controller and / or the build agent? You can look for them by editing the build definition.
Don't trust me as Release Management is pretty new, but the error is about the connection between the RM Server and the RM Deployer service (i.e. the RM agent). RM Server don't know anything more about the agent, so your option is to connect to the target machine(s) and manually check deployment status. If completed, restart the RM Deployer service and cross fingers.
I faced the same issue of the release being stuck in 'In Progress' state. Turned out, the password of the credentials I was using, changed. Once the new password was specified in the deployment agent, the release managed to complete. This was months ago, and now I am facing same issue on other server. No clue what is the reason this time.
We has had this problem in which all releases got stuck on TFS 2018
As there is a connectivity issue with SQL when release is completed , it may not update the status in DB in some cases if load is more, so the release is stuck in InProgress state and started consuming pipeline in SQL . Other releases will also not move ahead, as there is blockage in pipeline. Once we increase the pipeline count, the problematic release could move out as processing of releases started happening.
Once the problematic release is canceled by the system, we set the pipeline back to original count of 1, then you could see their releases progressing and not being stuck.
Solution:
You need to increase the count of pipeline to let say 25 after this create a new Release Pipeline and queue this pipeline this will push all those pipeline which got stuck .Once pipeline start queuing make the count back to one or original count.
Reference - https://blogs.msdn.microsoft.com/tfssetup/2017/11/14/understanding-build-and-release-pipelines-visual-studio-team-servicesteam-foundation-server/
TFS215085: An error occurred while connecting to agent X
TFS215074: Failed to start build agent on 'X' because another build was unexpectedly in progress. Investigate build agent status to determine the cause.
I am queuing up a large number of builds most of which run fine, occasionally a build will fail with these messages and the agent will become unreachable. Often it will automatically reset it self and become enabled again and builds will continue on that agent. Sometimes it will stay unreachable, but if I manually change its status back to enabled it will start processing builds again.
There are no TFS messages or other messages in the application logs related to this. The web IIS logs for the webservices give no indication there as well.
This is for Team Foundation Server 2008 SP1 running on both Windows Server 2008 and Windows Server 2003. I have turned logging on when this has failed, but there is no additional information there as well.
Any suggestions or hidden switches for managing the build agent that would resolve this issue would be greatly appreciated.
This error occurs since there is no official synchronization between the TFS application tier and the build machine(s) other than the server queue state. For instance, if a build is issued a stop command but doesn't actually stop on the machine, the server (in 2008) will mark it as stopped after a timeout period no matter what the outcome which immediately allows the next build in the queue to start. When this build attempts to start, it will fail with the error message above (TF215074) since we only allowed one build to run on an agent at any given time and this constraint has been violated. Is this issue happening often? Do you know how to reproduce it?
Patrick