We have a PoC on deploying a file to an old mainframe. There are many types of deployments that we do but this question focuses on individual files. We are able to SSH into the mainframe and we have a deployment pipeline with the steps needed to get one file into the correct location.
The problem is we have over 54,000 of these individual files. During a release we may deploy as little as 1-5 files or large deployment may be 250 files. Each of them will have a different source and target destination. Some of them may be sources from the same folder and deployed to the same folder but that is not guaranteed.
We can make the assumption that the files are immutable. There are issues on both build and release to consider:
Build - what is the artifact? Do we use one artifact for each release that could contain 1-250 files? We don't want to have 250 build scripts for a release, that we know.
Release - How do we use the pipelines. If you batch them together then is it a one click deploy to that environment? How would you determine if someone added a file to the release? I guess we would need a new build that would create a new pipeline?
There are a few other things that come up like we need to check the status in our change management system to confirm that the ticket for that File is in a status that is approvable. That is a deployment step currently.
I'm not sure this is the "answer" or not but this is our take on it so far:
The Artifact
We are going to create a "release" data file. In this file there will be a list of files going with each deployment. We will organize the files by product line and create a branch of all files for a specific product. Then the build will read the files and create the artifact from the list of files related to that release. We will also include the data file in the artifact.
Deployment
We will create a Parent/Child release process. The Parent script will loop through the data file and call the child script. The Child script will deploy an individual file which will be represented by a row in the data file. To deploy to Production the Parent will be deployed only. The child will not every be deployed individually.
Multiple Deployment Times/Dependencies
We have a requirement to Deploy certain files at certain times. One production file deployment may be at 1 PM and another at 7 PM in the same release. To accommodate this
we will include deployment time in the data file. After each file is deployed we will some how keep track that this file has been deployed.
Change Management
We will do our change management system check in each child script to make sure the file is ready to deploy. If the individual file is not approved we will not stop processing, we will finish the deployment for any other files in the list that are approved and then as the last step in the deployment we will fail the deploy. We need to make the "tracking" available to the teams to see what caused the deploy to fail.
Making some assumptions here and is happy path, but perhaps this will help get you to the ultimate solution.
Have a master branch that has a products folder. This folder would then have subfolders for each product, which has the files:
master/
products/
productA
productB
productN
Dev Team would work on files in separate fix branches then merge into master via pull requests. You can setup policies and gates for audit
Create a build pipeline with powershell script task that checks for deltas (possible example) in master and copy/publish only those changes to an artifact destination folder with the same product subfolder layout
Create a release pipeline that has a stage for each product and/or destination path on the mainframe. Each stage would have a custom task that copies the files from the appropriate product folder to the destination via SSH. You could even create a task group that gets re-used then just use variables for folder paths, etc. NOTE: The will be quite a few stages, but that's what release pipelines are for :)
Schedule the release pipeline to run at the desired times. You can setup notifications on failures so someone or process can investigate/retry etc.
Related
I am new to the world of scripting with TFS2015. I created a script that builds all of the projects within my solution (it is a rather large solution) and puts it out in a shared folder (where each project has its own subfolder).
I would like to create a separate script for each project that simply copies the bin folder from the shared and pastes it out on my Test environment. I rarely need to deploy everything, so the idea is one build...multiple deploys.
However, when I run my deploy script using the Copy Files step it is doing another build. Although it copies the files that I expect, it is after a full build that creates the folder structure for the build.
Am I able to make the Copy Files step NOT do a Build?
Here is the steps that my script is curently doing:
As you can see, there is only one step (Copy Files) but it still does the Get sources and copies everything into a new folder on the build box like so (where the number keeps incrementing up with each run of the script):
I just want to copy the files from the Source to the Target and not do a build or Get Sources.
It looks like you're still on TFS 2015 RTM or Update 1. Which is already pretty old technology if you compare it to the lifetime of the new build system which was introduced with this version.
TFS 2015 update 2 has introduced a similar system to the Build pipelines to orchestrate Releases. This doesn't require you to map any workspaces or git repositories and can act on the artefacts of your builds or simply on the contents of file shares.
It makes sense that a Build has to build something and in order to build something, it has to get the things to build. If you're actually not building something, then you're probably deploying or releasing or packaging something else. Hence the distinction between Build and Release pipelines.
TFS 2017+ has an option to disable the syncing of sources. Primarily to allow people to get the sources themselves in creative ways (e.g. a custom powershell script that invokes git.exe).
My primary advice would be to upgrade to TFS 2018 update 3 or at least TFS 2017 update 3.1, worst case TFS 2015 update 4.1. The fact that versions older than update 2015.4.1 have a known XSS scripting security bug may be reason enough to convince your organisation to perform this update.
Barring that option you're left with one solution:
Link your build definition either to a git repository with only a single commit (If I remember correctly the 2015 agent still crashes when syncing an empty Git repo) or link it to a TFVC repository and set the workspace settings to cloak everything. This essentially causes the build to sync an empty folder, which it can cache, before calling your powershell script.
I'm trying to understand what it does. Currently, this is the value that I see - dist/.tgz
From what I understand, our grunt scripts makes a tgz file. However, I don't know what Jenkins does.
I got an error when I didn't specify any pattern
ERROR: No artifacts are configured for archiving.
You probably forgot to set the file pattern, so please go back to the configuration and specify it.
If you really did mean to archive all the files in the workspace, please specify "**"
Build step 'Archive the artifacts' changed build result to FAILURE
Most importantly, it allows you to archive items from your job's workspace in a persistent and accessible way, linked to the specific build number.
I.e. you have a job Build that compiles your sources into program.exe, archiving it linked to the build it was produced by, and keeping it accessible for developers or other jobs can come in very handy.
Additionally, archived artifacts are transferred to your jenkins master, so your job can run on any slave, but your archived files will be always accessible, even when that particular slave is offline.
Also, with the right configuration and plugins, other projects can access archived artifacts from other projects. I.e. a job Deploy that uploads your program.exe to some location is as trivial as copying the archived artifact of the last successful build into its workspace for the upload.
Theres quite some information on SO already, i.e. here.
I am not a developer nor am I the Application Lifecycle Manager. I pull the latest builds from TFS Build Server along with the latest code and I import them in AppScan source for analysis (security tool). Against each security findings there is a class name within which the findings were found along with the directory where the class is located.
Builds are created automatically during a certain time of the day. As mentioned earlier the directory names change every time after a new build and a new number (highlighted below) is introduced. It is not sequential, does anyone know of TFS build server assigns these numbers.
C:\Builds**12**\Development
C:\Builds**14**\Development
C:\Builds**13**\Development
C:\Builds**15**\Development
TFS will create folders based on the following.
DriveLetter\Builds\$(BuildAgentId)\$(BuildDefinitionPath)
so if you have 1 controller with 4 agents on a clean install you would get
Drive:\Builds\1\$(BuildDefinitionPath)
Drive:\Builds\2\$(BuildDefinitionPath)
Drive:\Builds\3\$(BuildDefinitionPath)
Drive:\Builds\4\$(BuildDefinitionPath)
Your Build definition is the name of the build definition and then each subsequent build is stored underneath, with each individual build getting its own folder.
these settings are available on the Build Agent properties screen.
These are transient files and you cant rely on them being there. They are only used as part of the build. You would be better doing a "Get" from TFS at the appropriate Changeset number for the build and using the binaries from the drop folder.
I'm using TFS 2013 on premises. I have four build agents configured on a Build machine. Several build definitions compile ASP .NET websites. I configured the msbuild parameters to deploy the IIS application to the integration server, which sits out there in Rackspace.
By default webdeploy does differential deployments by comparing file dates. In my case that's a big plus because copying files from our network to Rackspace takes quite some time. Now, in order to preserve file dates the build agent has to compile the same base set of source code. On every build only the differential source code yields a new DLL, minimizing the number of files deployed.
All of that works fine, with a caveat: a given build definition has to be assigned to a build agent (by agent name or tag). The problem is I create a lot of contingency when all builds assigned to the same agent are queued up. They wait in line until the previous build is done.
In an ideal world any agent should be able to take care of any build, but the source code being compiled has to be the same, regardless of the agent.
I tried changing the working folder of all agents to point to the same location but I get an error because two agents can't be mapped to the same folder. I guess there is one workspace per agent.
Any ideas?
Finally I found a way to do this. Here are all the changes you need to do:
By default the working folder of each agent is $(SystemDrive)\Builds\$(BuildAgentId)\$(BuildDefinitionPath). That means there's one working folder per BuildAgentId. I changed it so that all Agents share the same folder: $(SystemDrive)\Builds\WorkingFolder\$(BuildDefinitionPath)
By default at runtime the workflow creates a workspace that looks like "[BuildDefinitionId][AgentId][MachineName]". Because all agents share the same working folder there's an error trying to create each separate workspace. The solution to this is in the build definition: Edit the xaml and look for an activity called "Get sources from Team Foundation Version Control". There's a property called WrokspaceName. Since I want to have one workspace per build definition I set that property to the BuildDetail.BuildDefinition.Name.
Save your customized build template and create a build that uses it.
Make sure the option "1. TF VersionControl/1. Clean workspace" is set to False. Otherwise the build will wipe out all the source code on every build.
Make sure the option "2. Build/3. Clean build" is set to false. Otherwise the build will wipeout the output binaries on every build.
With this setup you can queue up the same build on any agent, and all of them will point to the same source code and bin output. When the source code changes only the affected binaries are recompiled. I have a custom step in the template that deploys the output files to IIS, to all the servers in our webfarm, using msdeploy.exe. Now my builds+deployments take one or two minutes, because only the dlls or content that changed during the build are synchronized to the servers.
You can't run two build agents in the same folder. The point of build agents is to run multiple builds in parallel, usually on separate PCs. If you try to run them on the same source code, then (a) it's pointless as two build of exactly the same source should produce identical results, and (b) they are almost certainly going to trip over each other and cause the builds to fail or produce unexpected results.
If you want to be able to build and then deploy a series of versions of your codebase, then there are two options:
if you queue up multiple builds, then the last one will "win", so the intermediate builds are of no real value. So if you check in New code before your first build completes, you may as well stop the active build and start a new one. you should be asking yourself why the build is so slow, or why you are checking in changes so often that this is necessary.
if each build produces an incremental update to the deployed result, then you need to pass the output of your builds to some deployment agent that is able to diff it against the deployed version and send only the changes to be deployed. This could be set up to gather results from multiple build agents if that would be beneficial.
but I wonder if perhaps your build Is slow because you are doing a complete build each time (which cleans the build folder, gets all the sources, and does a full rebuild), when what you want is an incremental build (which gets the latest changes, compiles only what is affected, and complete quickly). perhaps you should investigate making your build incremental.
If you follow some of MS's recommended branching strategies you can easily end up with a project structure such as:
$PROJECT\
DEV\
MyProject
STAGE\
MyProject
PROD\
MyProject
Now let's say I have three different build definitions. One each for DEV, STAGE, and PROD. This should be common considering that the build definition will define the exact solutions to build.
If I turn on CI for each of them, STAGE will be built even though the checkin occurred in DEV...
Now my question. How can I limit the build definition to execute only when a check in occurs in either a path or a solution that is part of the build definition?
When defining the working folder configuration screen - only have it start at the root of the branch you want to build.
For example, your DEV branch would be configured so that $/TEAMPROJECT/DEV/MyProject was mapped to $(SourceDir) rather than the default mapping which would have been set to just $/TEAMROJECT.
FYI - Personally, I only have CI Builds on Dev branches and queue manual builds for a push to QA. I also normally don't do a re-build for production but just push the build binaries that were QA'd. I also keep by build configuration folder inside the branch i.e. $/TEAMPROJECT/DEV/TeamBuild rather than the default $/TEAMPROJECT/TeamBuildTypes and therefore changes to the build configuration are also pushed up through the branches. That said, you have to stick with the default if you wanted the build configuration to be visible to VS2005 clients.
Hope that helps,
Martin.