Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
We are on TFS 2017, on-premises installation. We have one TFS application server / source server and one build server. On the build server we have only one default build agent installed and the builds / releases are getting queued if more than one build / release is getting submitted.
Can we install more than one build agent on the single build server?
If so, having more than one build agent, will it ease the build by running them parallelly?
When setting up build / release definitions, do we have to tell the definition what build agent to use or will the TFS build automagically decide it?
Are there any cost / licence implications of running more than one build agent on one single server?
Can we install more than one build agent on the single build server?
Yes.
If so, having more than one build agent, will it ease the build by
running them parallely?
There's no conclusive answer.
Will multiple agents let you run multiple builds in parallel? Yes.
Will this speed things up? Maybe.
Builds are typically I/O limited, which means the bottleneck is with how fast stuff can be read from and written to disks. More CPU/RAM, past a certain point, won't help.
One agent per server is definitely overkill. My rule of thumb is no more than two build agents per physical disk in a machine. RAM/CPU adjusted as necessary, depending on the size/complexity of the applications being compiled.
When setting up build / release definitions, do we have to tell the
defintion what build agent to use or will the TFS build automagically
decide it?
You can register build agents into agent pools and queues. You specify an agent queue to use in your build definition. The agent will be picked from that set of agents.
Are there any cost / licence implications of running more than one
build agent on one single server?
Licensing is based on concurrency, not number of agents or number of build servers.
Documentation states:
"Users who have Visual Studio Enterprise subscriptions are assigned to
VS Enterprise access level in the Users hub of TFS instance. Each of
these users contributes one additional concurrent pipeline to each
collection. You can use this benefit on all Team Foundation Servers in
your organization."
Beyond that, there is no licensing consideration.
ref: https://learn.microsoft.com/en-us/vsts/build-release/concepts/licensing/concurrent-pipelines-tfs?view=vsts
Related
What are the best practices for a jenkins installation like the one below?
I do have a quite small dedicated server with 16 gb of ram and 2tb of diskspace with enabled hardware virtualization, and one use of it would be to host my own projects (opensource), and there are applications set up such as git repository manager and stuff.
I would like to set up jenkins there for automatic building, but I want to make it secure.
This installation is small enough to require only a master node, but I am planning to disable building on master completely, and to run a virtual machine as an agent, for the reason that it would be isolated as much as possible on the same physical server, so that a job would be unable to destroy jenkins master data.
Should I go for master only anyway? or, if using a virtual machine agent, should I have only one executor there or multiple ones? I probably cannot isolate multiple parallel jobs running on one agent without using one agent per job, but maybe I am overthinking all this. Using one agent per job, at least in case of virtual machines, would exhaust server resources very quickly, or alternatively, money.
You can use Jenkins own database of users, which I have used in commercial settings and it has worked perfectly well. If you have Active Directory you can also integrate with this if you want to go to extra effort so people only have to remember one login.
Once users are logged in you should provide authorisation via the Role-Strategy plugin
On Team Foundation (TFS2017) which is the maximum number of build agents that you can have connected to your TFS instance?
There is not any official document statement the limitation of build agent numbers with TFS for now. Also didn't get any related prompt info such as: build agents have reached the maximum.
For multiple machines, you could configure as much as you require, there is no evidently limitation.
For a single machine, it depends on the hardware. If your agent server is virtual, then it is already slower as compared to the physical, you also need to allocate sufficient RAM for it.
Can I install multiple private agents on the same machine?
Yes. This approach can work well for agents that run jobs that don't
consume a lot of shared resources.
You might find that in other cases you don't gain much efficiency
by running multiple agents on the same machine. For example, it might
not be worthwhile for agents that run builds that consume a lot of
disk and I/O resources.
You might also run into problems if concurrent build processes are
using the same singleton tool deployment, such as NPM packages. For
example, one build might update a dependency while another build is in
the middle of using it, which could cause unreliable results and
errors.
Source Link
It depends on how many cores agent server has. One Agent will take up one core.
I am not a developer, but reading about CI/CD at the moment. Now I am wondering about good practices for automated code deployment. I read a lot about the deployment of code to a pre-existing environment so far.
My question now is whether it is also good-practice to use e.g. a Jenkins workflow to deploy an environment from scratch when a new build is created. For example for testing of a newly created build, deleting the environment again after testing.
I know that there are various plugins to interact with AWS, Azure etc. that could be used to develop a job for deployment of a virtual machine.
There are also plugins to trigger Puppet to deploy infra (as code) and there are plugins to invoke an infrastructure orchestration.
So everything is available to be able to deploy the infrastructure and middleware before deploying code (with some extra effort of course).
Is this something that is used in real life? How is it done?
The background of my question is my interest in full automation of development with as few clicks as possible, and cost saving in a pay-per-use model by not having idle machines.
My question now is whether it is also good-practice to use e.g. a Jenkins workflow to deploy an environment from scratch when a new build is created
Yes it is good practice to deploy an environment from scratch. Like you say, Jenkins and Jenkins pipelines can certainly help with kicking off and orchestrating that process depending on your specific requirements. Deploying a full environment from scratch is one of the hardest things to automate, and if that is automated, it implies that a lot of other things are also automated, such as infrastructure, application deployments, application configuration, and so on.
Is this something that is used in real life?
Yes, definitely. A lot of shops do this. The simpler your environments, the easier it is, and therefore, a startup with one backend app would have relatively little trouble achieving this valhalla state. But even the creation of the most complex environments--with hundreds of interdependent applications--can be fully automated; it just takes more time and effort.
The background of my question is my interest in full automation of development with as less clicks as possible and cost saving in a pay-per-use model by not having idling machines.
Yes, definitely. The "spin up and destroy" strategy benefits all hosting models (since, after full automation, no one ever has to wait for someone to manually provision an environment), but those using public clouds see even larger benefits in terms of cost (vs always leaving AWS environments running, for example).
I appreciate your thoughts.
Not a problem. I will advise that this question doesn't fit stackoverflow's question and answer sweet spot super well, since it is quite general. In the future, I would recommend chatting with your developers, finding folks who are excited about this sort of thing, and formulating more specific questions when you all get stuck in the weeds on something. Welcome to stackoverflow!
All is being used in various combinations; the objective is to deliver continuous value to end user. My two cents:
Build & Release
It depends on what you are using. I personally recommend to use what is available with the tool. For example, VSTS (Visual Studio Team Services) offers complete CI/CD pipeline. But if you have a unique need which can only be served by Jenkins then you must use that and VSTS offers that out of the box.
IAC (Infrastructure as code)
In addition to Puppet etc. You can take benefits of AZURE ARM (Azure Resource Manager) Template to Build and destroy an environment. Again, see what is available out of the box with the tool set you have.
Pay-per-use
What I have personally used is Azure Dev/Test Labs and have the code deployed to that via CI/CD pipeline. Later setup Shutdown policy on the VM so it will auto-start and auto-shutdown based on time provided. This is a great feature to let you save cost on the resources being used and replicate environments.
For example, UAT environment might not needed until QA is signed off. But using IAC you can quickly spin up the environment automatically and then have one-click deployment setup to deploy code to UAT.
Trying to set up build server after upgrading to TFS 2015.
The way I envision is:
Single Agent Pool, that will have 3 queues:
1. Nightly builds
2. CI builds
3. Gated/validation builds.
Each of them will have some agents, the goal is to have some control, to make sure nightly builds wouldn't consume all agent, so gated queue will always have some available agent.
The problem I have now is when I try to add new queue, the option "Use existing pool" is disabled, I can only add queue with creating new agent pool.
It doesn't work the way you want it to work.
One agent can be a member of one and exactly one agent pool. The agent pool exists at the server level, not the Team Project Collection level.
One agent queue is tied to one and exactly one agent pool. However, agent pools can be referenced by different agent queues across Team Project Collection boundaries.
So, the upshot of this is that you can share your agent pools across multiple team project collections.
In VSTS, the distinction exists but is less relevant -- you can't have multiple Team Project Collections, so an agent pool and an agent queue are more or less equivalent, you just have to manage both of them.
You can use custom Capabilities (on your agents) and Demands (on your build definitions) to ensure that particular agents are always reserved for particular build scenarios.
Of course, task-based builds don't support gated checkin for TFVC yet, so your concern about gated agents always being available is moot, at least for now.
Now that all of that is out of the way, the answer to your question is simple:
Q: I'm trying to create a queue that uses an existing pool, but the
controls are grayed out. Why?
A: On the Create Queue dialog box, you can't use an existing pool if it
is already referenced by another queue. Each pool can be referenced by
only one queue. If you delete the existing queue, you can then use the
pool.
Ref: https://msdn.microsoft.com/en-us/Library/vs/alm/Build/agents/admin
I work on a large open source project based on ruby on rails. We use Github, Travis, Code Climate and others. Our test suite takes a long time to run and we have many pull requests opened and updated through the day, which creates a large backlog. We even implemented a build killer in our bot to prevent any unnecessary builds, however we still have a backlog. Is it possible for us to host our own runner to increase the number of workers?
There's Travis CI Enterprise (https://enterprise.travis-ci.com/) that lets people host their own runners, but that's probably mostly only for paid-for customers. Have you guys swapped over to the container-based builds? Might speed things up a bit. What's the project?