yarn takes a lot of time on vsts hosted agent due to more than few dependencies .
Our monorepo contains three somewhat identical but totally different apps which share lot of node dependencies.
Each app is very huge and takes considerable time to build. So we build individual app based on path filter
Release contains artifacts from all three builds
What I need
download node modules once
use same downloaded dependencies in three different conditional builds
release app after all or any build with artifacts latest for each build
any pointers how to configure this
There isn't any way to do this with Hosted Agent. The Hosted Agent is a group of virtual machines hosted on Azure. Every time you queue a new build, it will initialize an available agent from these machines with a clean environment. So the build machine you used may different for every build. And when the build is finished, the files downloaded/generated during the build will also be cleared. So there isn't any way to share the files between them.
Related
We have a PoC on deploying a file to an old mainframe. There are many types of deployments that we do but this question focuses on individual files. We are able to SSH into the mainframe and we have a deployment pipeline with the steps needed to get one file into the correct location.
The problem is we have over 54,000 of these individual files. During a release we may deploy as little as 1-5 files or large deployment may be 250 files. Each of them will have a different source and target destination. Some of them may be sources from the same folder and deployed to the same folder but that is not guaranteed.
We can make the assumption that the files are immutable. There are issues on both build and release to consider:
Build - what is the artifact? Do we use one artifact for each release that could contain 1-250 files? We don't want to have 250 build scripts for a release, that we know.
Release - How do we use the pipelines. If you batch them together then is it a one click deploy to that environment? How would you determine if someone added a file to the release? I guess we would need a new build that would create a new pipeline?
There are a few other things that come up like we need to check the status in our change management system to confirm that the ticket for that File is in a status that is approvable. That is a deployment step currently.
I'm not sure this is the "answer" or not but this is our take on it so far:
The Artifact
We are going to create a "release" data file. In this file there will be a list of files going with each deployment. We will organize the files by product line and create a branch of all files for a specific product. Then the build will read the files and create the artifact from the list of files related to that release. We will also include the data file in the artifact.
Deployment
We will create a Parent/Child release process. The Parent script will loop through the data file and call the child script. The Child script will deploy an individual file which will be represented by a row in the data file. To deploy to Production the Parent will be deployed only. The child will not every be deployed individually.
Multiple Deployment Times/Dependencies
We have a requirement to Deploy certain files at certain times. One production file deployment may be at 1 PM and another at 7 PM in the same release. To accommodate this
we will include deployment time in the data file. After each file is deployed we will some how keep track that this file has been deployed.
Change Management
We will do our change management system check in each child script to make sure the file is ready to deploy. If the individual file is not approved we will not stop processing, we will finish the deployment for any other files in the list that are approved and then as the last step in the deployment we will fail the deploy. We need to make the "tracking" available to the teams to see what caused the deploy to fail.
Making some assumptions here and is happy path, but perhaps this will help get you to the ultimate solution.
Have a master branch that has a products folder. This folder would then have subfolders for each product, which has the files:
master/
products/
productA
productB
productN
Dev Team would work on files in separate fix branches then merge into master via pull requests. You can setup policies and gates for audit
Create a build pipeline with powershell script task that checks for deltas (possible example) in master and copy/publish only those changes to an artifact destination folder with the same product subfolder layout
Create a release pipeline that has a stage for each product and/or destination path on the mainframe. Each stage would have a custom task that copies the files from the appropriate product folder to the destination via SSH. You could even create a task group that gets re-used then just use variables for folder paths, etc. NOTE: The will be quite a few stages, but that's what release pipelines are for :)
Schedule the release pipeline to run at the desired times. You can setup notifications on failures so someone or process can investigate/retry etc.
I have a Jenkins setup running in production, I want to automate jenkins setup(installation) along with all the jobs that are setup in jenkins.
One crude way I can think of is to copy the whole jobs directory to the new Jenkins setup.
I want to know how other people in industry do deal with this problem.
I have used the plugin Thinbackup to move jobs, users, and plugins. You can make a full backup and restore it to the new server. The plugin is not perfect and is up for adoption. I had issues with the restore. I ended up using the plugin only for creating the archive, but then I copied manually the folders (users, jobs, plugins, nodes, email-templates, secrets, JENKINS_HOME files) from the archive to the new server.
Before creating the archive or copying the jobs, ensure that no more than 30 builds per job are kept, this will keep your archive small. I have seen 5000+ builds per job, which were totally unnecessary and were blocking the creation of the archive.
When you create or restore the archive, or copy files, the server should be in quiet mode, no builds should be executed.
http://<jenkins.server>/quietDown
After you copy the files or restore the archive, you should restart Jenkins or even better, restart the server.
Another option is to use RSync as mentioned here. I am not sure what is the OS of your Jenkins server. If it is Linux you can check out this guide that I have written.
I have set up a CI Build that is also executing some tests.
In the GetSources step Clean is set to true.
I use a git repo.
When I run the build in a hosted agent, "getSources" takes about 20 seconds. When I run the build on a on premise agent, "getSources" takes about 20 minutes!
I can see, that the on premise agent is mostly idle in terms of cpu and memory. I also verified that network speed is at around 50 MBit/s.
Why does getSources take so long?
The Clean option has no effect for Hosted agent.
No matter what you set for the clean option (false or true for clean sources/all build directories/output directory etc), when you queue build with Hosted agent, it always download the sources only.
Assume if you set Clean as true and clean all build directories. For private agent, it will delete the entire working folder that contains the sources folder, binaries folder, artifact folder, and so on. But for Hosted agent, it only download the sources each time.
So the execute time for Hosted agent is different from private agent for most time.
To speed up private agent to build, you can follow any of below aspect:
set Clean as false (more efficient).
Since you are queuing CI build, if you set Clean as false, the private agent will only update the files which has been modified/created/deleted to your local source folder.
Only clean sources if you still need to set Clean as true.
It will only clean up the files and subfolders source folder (s/). But if your project is large enough, you'd better use the way to set Clean as false.
To answer the "why?", it's the fact that you're copying all those files between your system and Microsoft's data center. I've experienced the same and bought some additional pipelines to accommodate.
I don't like when it comes to release my projects on production server.. May be i just don't have enough experience, nobody taught me how to do this in a right way.
For now i have several repos with scala (on top of spray). I have everything to build and run this projects on my local machine (of course, i develop them). So installed jenkins on my production server in order to sync from git, build and run. It works for now but i don't like it, because i need to install jenkins on every machine i want to have run my projects. What if i want to show my project to my friend in cafe?
So i've come with idea: what if i run tests before building app, make portable build (e.q. with sbt native packager) and save it on remote server "release server". That server just keeps these ready to be launched apps.
Then i go to production server, run bash script that downloads executables from release server and runs my project on a machine
In future i want to:
download and run projects inside docker containers.
keep ready to be served static files for frontend. Run docker
container with nginx and linked volume with static files
I heard about nexus (http://www.sonatype.org/nexus/), that artist use to save their songs, images, so on. I believe there should be open source projects that expose idea like mine
Any help is appreciated!
A common anti-pattern, in my opinion, is to build the software every time you perform a deployment.You are best advised to separate the process of build from the act of deployment by introducing a binary repository manager (you've mentioned on such example, nexus).
Best Practice - Using a Repository Manager
Binary repository manager
How can I automatically deploy a war from Nexus to Tomcat?
Only successfully tests builds get pushed to the repository, so you can treat each successful build as a mini-release. A by-product of this is that your production server does not have to have all the build software pre-installed (like, Jenkins, ANT , Maven, etc).
It should be noted that modern repository managers like Nexus and Artifactory now support Docker registries too, so that you use these for deploying docker images too.
Update
A related chef question, a technology where there is no intermediate binary file (like a jar). In this case the software is still "released" by creating a tar distribution stored in the repo.
chef cookbook delivery - chef server vs. artifactory + berkshelf
I'm using TFS 2013 on premises. I have four build agents configured on a Build machine. Several build definitions compile ASP .NET websites. I configured the msbuild parameters to deploy the IIS application to the integration server, which sits out there in Rackspace.
By default webdeploy does differential deployments by comparing file dates. In my case that's a big plus because copying files from our network to Rackspace takes quite some time. Now, in order to preserve file dates the build agent has to compile the same base set of source code. On every build only the differential source code yields a new DLL, minimizing the number of files deployed.
All of that works fine, with a caveat: a given build definition has to be assigned to a build agent (by agent name or tag). The problem is I create a lot of contingency when all builds assigned to the same agent are queued up. They wait in line until the previous build is done.
In an ideal world any agent should be able to take care of any build, but the source code being compiled has to be the same, regardless of the agent.
I tried changing the working folder of all agents to point to the same location but I get an error because two agents can't be mapped to the same folder. I guess there is one workspace per agent.
Any ideas?
Finally I found a way to do this. Here are all the changes you need to do:
By default the working folder of each agent is $(SystemDrive)\Builds\$(BuildAgentId)\$(BuildDefinitionPath). That means there's one working folder per BuildAgentId. I changed it so that all Agents share the same folder: $(SystemDrive)\Builds\WorkingFolder\$(BuildDefinitionPath)
By default at runtime the workflow creates a workspace that looks like "[BuildDefinitionId][AgentId][MachineName]". Because all agents share the same working folder there's an error trying to create each separate workspace. The solution to this is in the build definition: Edit the xaml and look for an activity called "Get sources from Team Foundation Version Control". There's a property called WrokspaceName. Since I want to have one workspace per build definition I set that property to the BuildDetail.BuildDefinition.Name.
Save your customized build template and create a build that uses it.
Make sure the option "1. TF VersionControl/1. Clean workspace" is set to False. Otherwise the build will wipe out all the source code on every build.
Make sure the option "2. Build/3. Clean build" is set to false. Otherwise the build will wipeout the output binaries on every build.
With this setup you can queue up the same build on any agent, and all of them will point to the same source code and bin output. When the source code changes only the affected binaries are recompiled. I have a custom step in the template that deploys the output files to IIS, to all the servers in our webfarm, using msdeploy.exe. Now my builds+deployments take one or two minutes, because only the dlls or content that changed during the build are synchronized to the servers.
You can't run two build agents in the same folder. The point of build agents is to run multiple builds in parallel, usually on separate PCs. If you try to run them on the same source code, then (a) it's pointless as two build of exactly the same source should produce identical results, and (b) they are almost certainly going to trip over each other and cause the builds to fail or produce unexpected results.
If you want to be able to build and then deploy a series of versions of your codebase, then there are two options:
if you queue up multiple builds, then the last one will "win", so the intermediate builds are of no real value. So if you check in New code before your first build completes, you may as well stop the active build and start a new one. you should be asking yourself why the build is so slow, or why you are checking in changes so often that this is necessary.
if each build produces an incremental update to the deployed result, then you need to pass the output of your builds to some deployment agent that is able to diff it against the deployed version and send only the changes to be deployed. This could be set up to gather results from multiple build agents if that would be beneficial.
but I wonder if perhaps your build Is slow because you are doing a complete build each time (which cleans the build folder, gets all the sources, and does a full rebuild), when what you want is an incremental build (which gets the latest changes, compiles only what is affected, and complete quickly). perhaps you should investigate making your build incremental.