I want to set up an Incremental Build in TFS as we want to deploy only modified files into Physical path, not the entire code.
We want the feature to build & deploy only the files that have been changed from the previous deployment. This will reduce the build and deployment time and the developers won't have to wait longer to see their changes deployed.
What you're describing is not an "incremental build". You a describing a much more complex situation than an incremental build.
What you're describing has never been an out-of-the-box option, and is in fact incredibly difficult to do properly, and ultimately would probably not impact things as much as you're hoping, anyway.
First of all, it's actually very difficult to determine a subset of files that have changed between deployments. And if you're building and deploying properly, then you're making a single build and deploying it along a pipeline of environments. This means that "what's different" at any given time is potentially different for every environment in your pipeline. Ex: DEV has version 5, QA has version 4, and PROD has version 3. So you have to start by assuming that you're going to use the oldest version. Build systems have no innate knowledge of "releases", so you'd have to build something into your build and release pipelines to track what source version constitutes the latest code in production.
Let's say you've solved that problem. You now have the ability to retrieve just the delta between what's deployed to production and the commit being built.
If you're working with compiled code, then you still need all of the source code, because you're going to have to rebuild the whole thing. Every assembly is going to get regenerated, and different metadata at compilation time is going to mean those assemblies are different even if the code that constitutes those assemblies is the same. And since assemblies can reference other assemblies, you have no straightforward way of determining at build time which assemblies have actually changed and need to be deployed. So you pretty much have no choice but to deploy all compiled assets every time. Note that this still applies to TypeScript or anything else that goes through a compiler/transpiler process; you need all of the code available, and it has to go through the entire build process.
So at this point, you still have to build your entire application to get the deployable output. Build time hasn't gone down at all. You've managed to bring down just a subset of static content (i.e. HTML pages, images, etc) to be deployed, though. That may have sped your deployments up a bit!
However, if the thing that's making your build and deployment process slow is that you have a ton of non-code-related static content, then you've gone through a very long and convoluted process to arrive at a much simpler solution: Move static content to a CDN and get it out of source control, or have a separate process that manages static content so that it can be deployed independently of unrelated application code.
You haven't really provided any information that can be used to provide a recommendation on how to proceed, but hopefully this answer is helpful in understanding why what you want to do is not going to solve your problem, unless you are dealing entirely with static content or scripts that don't require building.
Related
originally, I use gradle to build my android project, but recently, I migrate it to bazel, and I find that bazel is truly fast than gradle, so I want to know why, but the doc of bazel doesn't give much idea about this, can anyone help me？
Thanks very much!
Full disclosure: I work on Bazel.
That's not an easy question to answer for two reasons. First, performance is highly dependent on the scenario. For example, we'd generally expect a clean build to be slower than a build where only a single file has changed. Second, I don't know how Gradle works internally, and they've done a lot of work recently to improve Gradle performance.
But I can talk about Bazel and what we're doing to make it fast. We've been working on build performance for ~10 years, starting long before we made it public.
The key feature is that we require all dependencies to be declared, and we track them explicitly. If you use a header file in C++, or depend on a Java library, you must declare this dependency in your BUILD file (and we enforce that these are declared by sandboxing individual actions). There are three effects from this:
First, we can heavily parallelize the build, because we know which things depend on which other things.
Second, we can make incremental builds very fast, because we can tell what parts of the build have to be re-done when you change a specific file (BUILD file, header file, source file, ...).
Third, we almost never have to do clean builds. Other build tools often require 'make clean' to get into a predictable state - since Bazel knows all the dependencies, it can get to a predictable state on every single build.
Another effect is that we can cache remotely (i.e., across users), and even execute on another machine, although neither of these are fully supported at the time of this writing.
I am in the process of setting up continuous integration in our TFS system. One major part of our system are the development of about 50 DotNetNuke modules to support our CMS infrastructure. Right now, each of those projects have their own solution since their code bases are mostly siloed (with common code in 1 or 2 common projects). Keeping them in their own solution is done because it makes the development process faster (loading, compiling, etc....)
However, this has proven difficult to maintain when setting up TFS team build as each solution has to be manually added to the build definition and MSBuild seems unable to take advantage of parallel compiling due to each project being in its own solution. This causes about 5 minute full build times, which while isn't horrible isn't ideal. Mostly though, it's not ideal from a build definition maintenance aspect.
To solve this I creating a global solution that included all projects. The idea being that if you want your project to be automatically compiled and deployed by TFS you will have to include your project in the global solution. This seems works well, as it's easy to maintain from a build definition standpoint and brings the total build time down to 70 seconds.
The one problem is that the displayed TFS build log groups all warnings and errors together under the solution instead of separating them out by project. This makes it difficult to quickly see what project caused which errors and warnings.
Is there a good way to see project level error/warning messages in the build log summary view without delving into the cluttered build log?
To answer your direct question, I believe the answer is no (at least not without some heavy customization).
For me this is never a big concern as I am pretty aggressive about getting my teams to bring errors/warnings down to zero, then enforcing it via TFS Build (/p:TreatWarningsAsErrors=true). This means you should never have to wade through hundreds of warnings in the build summary.
If you add all your individual solutions to the build definition, you can always use the TFS Power Tools to "clone" a build def to make maintenance easier. You could also modify the Build Template to build the solutions/projects in parallel, although this runs the risk of having file contention issues.
Our project group stored binary files of the project that we are working on in SVN repository for over a year, in the end our repository grew out of control, taking backups of SVN repo became impossible at one point since each binary that is checked in is around 20 MB.
Now we switched to TFS,we are not responsible for backing the repository up, our IT tream takes care of it and we have more network and storage capacity for backups because of that but we want to decide what to do with the binaries. As far as I know TFS stores deltas and for binary files but deltas will be huge, but we might end up reaching our disk space quota one day, so I would like to plan things better from the start, I don't want to get caught up in a bad situation when it's too late to fix the problem.
I would prefer not keeping builds in the source control but our project group insists to keep a copy of every binary for reproducing the problems that we see in the production system, I can't get them to get the source code from TFS, build it and create the binary, because it is not straightforward according to them.
Does TFS offer a better build versioning method? If someone can share some insight I'd really be grateful.
As a general rule you should not be storing build output in TFS. Occasionally you may want to store binaries for common libraries used by many applications but tools such as nuget get around that.
Build output has a few phases of its life and each phase should be stored in a separate place. e.g.
Build output: When code is built (by TFS / Jenkins / Hudson etc.) the output is stored in a drop location. This storage should be considered volatile as you'll be producing a lot of builds, many of which will be discarded.
Builds that have been passed to testers: These are builds that have passed some very basic QA e.g. it compiles, static code analysis tools are happy, unit tests pass. Once a build has been deemed good enough to be given to test it should be moved from the drop location to another area. This could be a network share (non production as the build can be reproduced) there may be a number of builds that get promoted during the lifetime of a project and you will want to keep track of what versions the testers are using in each environment.
Builds that have passed test and are in production: Your test team deem the build to be of a high enough quality to ship. As part of your go live process, you should take the build that has been signed off by test and store it in a 3rd location. In ITIL speak this is a Definitive Media Library. This can be a simple file share, but it should be considered to be "production" and have the same backup and resilience criteria as any other production system.
The DML is the place where you store the binaries that are in production (and associated configuration items such as install instructions, symbol files etc.) The tool producing the build should also have labelled the source in TFS so that you can work out what code was used to produce the binary. Your branching strategy will also help with being able to connect the binary to the code.
It's also a good idea to have a "live like" environment, this should be separate from your regular dev and test environments. As the name suggests it contains only the code that has been released to production. This enables you to quickly reproduce bugs in production
Two methods that may help you:
Use Team Foundation Build System. One of the advantages is that you can set up retention periods for finished builds. For example, you can order TFS to store the 10 latest successful builds, and the two latest failed ones. You can also tell TFS to store certain builds (e.g. "production builds"/final releases) indefinitely. These binaries folders can of course also be backed up externally, if needed.
Use a different collection for your binaries, with another (less frequent) backup schedule. TFS needs to backup whole collections, but by separating data that doesn't change as frequently as the source you can lower the backup cost. This of course depends on the frequency you are required to have the binaries backed up.
You might want to look into creating build definitions in TFS to give your project group an easy 'one button' push to grab the source code from a particular branch and then build it and drop it to a location. That way they get to have their binaries, and you don't have to source control them.
If you are using a branching strategy where you create Release or RTM branches when you push something to production, then you can point your build definitions at those branches and they can manually trigger them from the TFS portal or from within Visual Studio.
Some times ago I asked the question about how to integrate an application using dependencies on a build server and I had quite satisfying answers. Today I am facing a different case. For a project I have to use non-redistribuable depedencies (RDL object model for SSRS). It means that out-of-the-box, these assemblies are not made to be deployed for development purpose. But somehow, I need to...
My first guess was to publish them in the GAC. Fine, it worked and the build server was able to compile the project smoothly. But then I realised that it broke some applications like the Report Server and the Report Builder (probably it would also break BIDS). So publishing in the GAC is definitely not a decent solution.
My second guess was to check the assemblies in source control. Well, it could work if I had only 2 assemblies for an amount of about 1MB. But here it is 23 assemblies and 29MB I have to check in, so it is definitely not suitable either.
I don't know much about MSBuild targets and maybe it could be a solution but I really have no idea on how to use it. I have been scratching my head hard and now I have to chose between breaking my builds or breaking my services!
As some people stated in comments we finally decided to source control the assemblies.
But as we are in an environment where we sometimes need to move a lot, which means not always in office, and need to work from distance with occasionally somewhat unreliable Internet connection, we decided to put some strict condition on whether we source control the assemblies or we deploy them on the build server and development machines.
Assemblies will be source controlled if all these criterias are met:
Assemblies/Framework is not deployable/redistribuable
Assemblies/Framework deployment may interfere with local machine services stability
Total amount of deployed assemblies on the project collection does not exceed 100MB
You could try using a different repository just for these assemblies, and do a checkout/update during the build job.
Also, if you want to keep it in the main repo as well, you could use svn:externals (http://svnbook.red-bean.com/en/1.0/ch07s03.html) to automatically update the DLLs when you update your working copy.
I apologize for the length of this post but I needed to include a lot of information for proper answers. I hope this does not discourage responses...
Our shop historically has coded web sites using Classic ASP with some newer ASP.NET sites configured as web sites. As everyone knows this means that the source files (*.asp, *.aspx, and *.aspx.vb (or *.aspx.cs)) files are deployed to development and production servers as is.
The configuration management process was (and still is) entirely manual and includes the following steps (requirements):
Taking copies of the modified files and storing them in a "release" folder for archiving.
Taking copies of the production files that will be replaced and storing them in a "archive" folder for easier rollback.
Generating a diff report of before and after source files for code review or general reference when diagnosing a post-release issue.
The developer who coded the changes is not the person who performs the production release. The original developer is required to hand off the source files to another developer for some additional testing and production deployment.
To make the situation more difficult (not with the above..but with what I talk about below) we do not follow a formal release schedule. As individual bugs or enhancements are completed they are released. This means we could easily be making several releases to a site a week. It is even possible that a given site gets two different releases to individual pages on the same day!
Since I came on board I have been trying to transition the team to newer technologies like ASP.NET web applications and ASP.NET MVC. (We have also taken on responsibility for stand-alone applications and console utilities used for non-web processes...so my dilemma still applies.)
The difference between these technologies and the legacy technologies is the pre-compiling. Instead of deploying the code-behind files (*.aspx.vb (or *.aspx.cs)) a dll or exe gets deployed. This type of deployment package has raised several questions (issues ??).
Generating difference reports when the source has been compiled. While the newly modified source files are sitting on the developers system the production copy is a compiled copy.
Making sure that changes related to other bugs or enhancements are not included in the particular release. This would apply to both the original developer and the person performing the release.
Allowing the original developer to pass along the changed files to another developer for build, testing, and deployment.
Up to now I was the only developer on the team working on these types of sites and applications so the conflicts and issues mentioned above where non-existent. (I skip the difference report step and the I do my own deployments.) However, I am trying to push the rest of the team to embrace this plus allow for better distribution of bugs and enhancement tasks.
We are currently using VSS but I am pushing (and will most likely succeed) in getting us moved over to TFS. Some ideas I have are
Setting up a separate build system for use by the developer to do the deployment. This will solve two problems -- (1) Different versions/patches of Visual Studio and other libraries between developers and (2) instances where the person performing the release has checked out files locally for another change. (Of course this does not guarantee differences between the build system and the original developer but at least that means the release is from a consistent config.
Using labels to tag just the modified files. My problem is that while I can identify (and pull down for a build) the modified files, how do I identify the files that need to be included in the build but have not changed. Again, the idea is to not included checked in files that are related to un-released changes.
Using labels to tag all the files for the release (the modified files and the unchanged files). My problem with this is similar to the last one...how do I make sure that a file checked in by another developer (say they went on vacation) for an un-related change is not labelled and included in this build.
Using the labels I could probably write a script to generate difference reports for the labeled version and the previously labeled version. If the process works properly that should result in exactly what changes are included in the the particular release..?
Any other ideas, concerns, points of interest? While I do have some flexibility of the process some of the requirements (like difference report or some way to easily view differences and having separate developer/deployer) are most likely untouchable.
Thank you so much for any help you can provide on this.
To keep track of different versions of the code and to help you manage very fast release cycles (daily) vs long term enhancements you can use branches in TFS.
There is a ton of information out there on branching, but in general I like to try to keep things simple. For example, have one branch called "release" and another "development". Everybody works on the development branch but the code to be deployed to production is merged into the release branch right before release.
This blog post describes the process:
http://team-foundation-server.blogspot.com/2008/01/how-we-branch-our-code-in-tfs.html
Well, based on my experience with VS2003 vs VS2010 for example is that the project structures are different and allowing VS to do a conversion often times results in a solution that either requires a lot of refactoring or is unusable. Having said that; if you can transition everything over to TFS2010 then one way to handle it is to setup different projects for each solution and use the TFS built in version handling for the different releases. You can also set up a build server and schedule nightly builds. If the build is ok then you can push this version into testing and ultimately production. You should really read up on TFS because it's totally different from VSS and is definitely a huge upgrade in allowing you to do team-focused development.
P.S. TFS has a really good Sharepoint integration which will help you and your team keep track of all the bugs and tasks.