I am in the process of setting up continuous integration in our TFS system. One major part of our system are the development of about 50 DotNetNuke modules to support our CMS infrastructure. Right now, each of those projects have their own solution since their code bases are mostly siloed (with common code in 1 or 2 common projects). Keeping them in their own solution is done because it makes the development process faster (loading, compiling, etc....)
However, this has proven difficult to maintain when setting up TFS team build as each solution has to be manually added to the build definition and MSBuild seems unable to take advantage of parallel compiling due to each project being in its own solution. This causes about 5 minute full build times, which while isn't horrible isn't ideal. Mostly though, it's not ideal from a build definition maintenance aspect.
To solve this I creating a global solution that included all projects. The idea being that if you want your project to be automatically compiled and deployed by TFS you will have to include your project in the global solution. This seems works well, as it's easy to maintain from a build definition standpoint and brings the total build time down to 70 seconds.
The one problem is that the displayed TFS build log groups all warnings and errors together under the solution instead of separating them out by project. This makes it difficult to quickly see what project caused which errors and warnings.
Is there a good way to see project level error/warning messages in the build log summary view without delving into the cluttered build log?
To answer your direct question, I believe the answer is no (at least not without some heavy customization).
For me this is never a big concern as I am pretty aggressive about getting my teams to bring errors/warnings down to zero, then enforcing it via TFS Build (/p:TreatWarningsAsErrors=true). This means you should never have to wade through hundreds of warnings in the build summary.
If you add all your individual solutions to the build definition, you can always use the TFS Power Tools to "clone" a build def to make maintenance easier. You could also modify the Build Template to build the solutions/projects in parallel, although this runs the risk of having file contention issues.
Related
I want to set up an Incremental Build in TFS as we want to deploy only modified files into Physical path, not the entire code.
We want the feature to build & deploy only the files that have been changed from the previous deployment. This will reduce the build and deployment time and the developers won't have to wait longer to see their changes deployed.
What you're describing is not an "incremental build". You a describing a much more complex situation than an incremental build.
What you're describing has never been an out-of-the-box option, and is in fact incredibly difficult to do properly, and ultimately would probably not impact things as much as you're hoping, anyway.
First of all, it's actually very difficult to determine a subset of files that have changed between deployments. And if you're building and deploying properly, then you're making a single build and deploying it along a pipeline of environments. This means that "what's different" at any given time is potentially different for every environment in your pipeline. Ex: DEV has version 5, QA has version 4, and PROD has version 3. So you have to start by assuming that you're going to use the oldest version. Build systems have no innate knowledge of "releases", so you'd have to build something into your build and release pipelines to track what source version constitutes the latest code in production.
Let's say you've solved that problem. You now have the ability to retrieve just the delta between what's deployed to production and the commit being built.
If you're working with compiled code, then you still need all of the source code, because you're going to have to rebuild the whole thing. Every assembly is going to get regenerated, and different metadata at compilation time is going to mean those assemblies are different even if the code that constitutes those assemblies is the same. And since assemblies can reference other assemblies, you have no straightforward way of determining at build time which assemblies have actually changed and need to be deployed. So you pretty much have no choice but to deploy all compiled assets every time. Note that this still applies to TypeScript or anything else that goes through a compiler/transpiler process; you need all of the code available, and it has to go through the entire build process.
So at this point, you still have to build your entire application to get the deployable output. Build time hasn't gone down at all. You've managed to bring down just a subset of static content (i.e. HTML pages, images, etc) to be deployed, though. That may have sped your deployments up a bit!
However, if the thing that's making your build and deployment process slow is that you have a ton of non-code-related static content, then you've gone through a very long and convoluted process to arrive at a much simpler solution: Move static content to a CDN and get it out of source control, or have a separate process that manages static content so that it can be deployed independently of unrelated application code.
You haven't really provided any information that can be used to provide a recommendation on how to proceed, but hopefully this answer is helpful in understanding why what you want to do is not going to solve your problem, unless you are dealing entirely with static content or scripts that don't require building.
I am new in configuration of TFS.
Currently our project is 50% done but we found that we have very bad code. We consider the need for static code analysis like Resharper or another product like StyleCop, CodeAnalysis and FxCop.
We want configure the TFS to reject a checkin when that check in contains code that triggers code analysis warnings.
But for the previous code we want to suppress the existing warning to prevent the code from becoming worse than it already is.
As Ivan mentions, your root cause it not in the lack of analysis tools, but probably in the level of quality and rigor agreed (or currently being enforeced between team members) between the development team and their project's sponsor. It may be that the pressure on the team is too high, causing important review actions to be skipped, or that the team (or the sponsor!) doesn't have the same desire to quality as you or the sponsor. Or that the team doesn't have the right level of knowledge to prevent these issues from happening.
The best way out of this is to fix as much as you can in a short period of time.
Warning: I've experienced with a number of teams the effect of turning on too many rules all at once. Generally, there is a reluctance for people to concede that their work hasn't been up to par and turning on rules that do not directly cause bug ("The identifier is cased incorrectly" for example) can cause frustration that can severely hamper your momentum. Carefully selecting which rules need to be solved now and which can wait for later worked much better in my experience. Once the team has developed a way to solve these kinds of problem, you can easily apply more.
Turning on Tools like configuring Code Analysis for your solution or using the Solution Wide Analysis feature of Resharper, can help you spot issues, but it won't solve them or prevent similar issues from popping up in the future unless your team stops creating them.
Tip: Note that you can turn on Resharper during your build as well using the Resharper CLI features.
StyleCop I would not enforce on this team (just yet) if the code itself is bad enough to trigger massive warnings that may hold bugs and issues. Fix these problems first, make the code it pretty later. Your priorities are now to remove any possible bugs.
CodeAnalysis and FxCop are the same things, so you won't need to turn on both. A tool like Resharper can help your developers to quickly remove a lot of the issues by using the magic-key ALT+ENTER.
If you want to create a clean baseline you can run code analysis once, then select all warnings that are generated and then select Suppress in global suppression file. This will work for Code Analysis issues, but won't suppress any Compiler Warnings, there is no easy way to quickly suppress all current compiler warnings.
Tip: It sometimes helps to temporarily rename any existing globalsupressions.cs files, so that this "baseline" is stored separately. You then know which warnings you'll have to fix at a later point in time.
Tip: When a developer suppresses a warning, have them add a Justification="Reason for suppression" to the suppression that is generated, that way you can distinguish between carefully considered suppression and temporary ones.
Depending on whether you already have a build server your next step is to install Team Build and once you have a build server you'll need to setup a Build Definition. This blog post covers most of the steps.
In the build definition set the trigger to "Gated Checkin" and on the Process tab make sure you set Code Analysis to "Always". If you want to fail your build based on Code Analysis errors, you need to create a custom ruleset and configure that for your solution.
To have compiler errors fail the build you can also enable the "Treat Warnings as Errors
Once you have enabled your gated check-in build all developers changes will be prompted to wait for their build to finish. You can turn on alerts (using Web access) or use the Build Notification Tool to get notified when the changes were successfully submitted.
Tip: Instead of turning on all rules at once (or switching them all to cause an ERROR during builds) you can also opt to turn on rules a couple at a time and fix them. Turning on rules by category gives you a nice opportunity to teach people the importance of the rules being turned on and possible solutions for fixing them.
A far more advanced solution would be to install and configure SonarQube alongside your Team Build environment. The ALM Rangers and Sonar have recently worked together to create installation guidance and a number of extensions to enable Team Build and SonarQube integration. You can find the installation guide here.
I have a TFS 2010 with some projects and a common library used in 5 of them. We use VS 2013 and we have Rolling Builds enabled in most if not all build definitions. When the common library is checked in, all of the projects referencing it are recompiled - but the order is poor, the most commonly used project is compiled as the last one. Is there a way to change that so it gets compiled first?
This question hints at a lot of problems and possible solutions. The simplest answer is probably to just add more build servers to run all the builds in parallel.
Otherwise you need to consider turning off rolling builds and writing your own build scheduler. That or other strategies such as building the DLL once and checking it in as source to the other builds or running the build on a branch that is outside of the other 5 builds and merging the source into those builds when they want to pick up the change.
If you are otherwise happy with how things are now and don't want to do a lot of work to solve the problem.... then just scale out your build farm with additional agents.
There's a "Priority in queue" on the build definition, but it sounds like you might want to change your solution's > Project Dependencies' > Build Order?
I apologize for the length of this post but I needed to include a lot of information for proper answers. I hope this does not discourage responses...
Our shop historically has coded web sites using Classic ASP with some newer ASP.NET sites configured as web sites. As everyone knows this means that the source files (*.asp, *.aspx, and *.aspx.vb (or *.aspx.cs)) files are deployed to development and production servers as is.
The configuration management process was (and still is) entirely manual and includes the following steps (requirements):
Taking copies of the modified files and storing them in a "release" folder for archiving.
Taking copies of the production files that will be replaced and storing them in a "archive" folder for easier rollback.
Generating a diff report of before and after source files for code review or general reference when diagnosing a post-release issue.
The developer who coded the changes is not the person who performs the production release. The original developer is required to hand off the source files to another developer for some additional testing and production deployment.
To make the situation more difficult (not with the above..but with what I talk about below) we do not follow a formal release schedule. As individual bugs or enhancements are completed they are released. This means we could easily be making several releases to a site a week. It is even possible that a given site gets two different releases to individual pages on the same day!
Since I came on board I have been trying to transition the team to newer technologies like ASP.NET web applications and ASP.NET MVC. (We have also taken on responsibility for stand-alone applications and console utilities used for non-web processes...so my dilemma still applies.)
The difference between these technologies and the legacy technologies is the pre-compiling. Instead of deploying the code-behind files (*.aspx.vb (or *.aspx.cs)) a dll or exe gets deployed. This type of deployment package has raised several questions (issues ??).
Generating difference reports when the source has been compiled. While the newly modified source files are sitting on the developers system the production copy is a compiled copy.
Making sure that changes related to other bugs or enhancements are not included in the particular release. This would apply to both the original developer and the person performing the release.
Allowing the original developer to pass along the changed files to another developer for build, testing, and deployment.
Up to now I was the only developer on the team working on these types of sites and applications so the conflicts and issues mentioned above where non-existent. (I skip the difference report step and the I do my own deployments.) However, I am trying to push the rest of the team to embrace this plus allow for better distribution of bugs and enhancement tasks.
We are currently using VSS but I am pushing (and will most likely succeed) in getting us moved over to TFS. Some ideas I have are
Setting up a separate build system for use by the developer to do the deployment. This will solve two problems -- (1) Different versions/patches of Visual Studio and other libraries between developers and (2) instances where the person performing the release has checked out files locally for another change. (Of course this does not guarantee differences between the build system and the original developer but at least that means the release is from a consistent config.
Using labels to tag just the modified files. My problem is that while I can identify (and pull down for a build) the modified files, how do I identify the files that need to be included in the build but have not changed. Again, the idea is to not included checked in files that are related to un-released changes.
Using labels to tag all the files for the release (the modified files and the unchanged files). My problem with this is similar to the last one...how do I make sure that a file checked in by another developer (say they went on vacation) for an un-related change is not labelled and included in this build.
Using the labels I could probably write a script to generate difference reports for the labeled version and the previously labeled version. If the process works properly that should result in exactly what changes are included in the the particular release..?
Any other ideas, concerns, points of interest? While I do have some flexibility of the process some of the requirements (like difference report or some way to easily view differences and having separate developer/deployer) are most likely untouchable.
Thank you so much for any help you can provide on this.
To keep track of different versions of the code and to help you manage very fast release cycles (daily) vs long term enhancements you can use branches in TFS.
There is a ton of information out there on branching, but in general I like to try to keep things simple. For example, have one branch called "release" and another "development". Everybody works on the development branch but the code to be deployed to production is merged into the release branch right before release.
This blog post describes the process:
http://team-foundation-server.blogspot.com/2008/01/how-we-branch-our-code-in-tfs.html
Well, based on my experience with VS2003 vs VS2010 for example is that the project structures are different and allowing VS to do a conversion often times results in a solution that either requires a lot of refactoring or is unusable. Having said that; if you can transition everything over to TFS2010 then one way to handle it is to setup different projects for each solution and use the TFS built in version handling for the different releases. You can also set up a build server and schedule nightly builds. If the build is ok then you can push this version into testing and ultimately production. You should really read up on TFS because it's totally different from VSS and is definitely a huge upgrade in allowing you to do team-focused development.
P.S. TFS has a really good Sharepoint integration which will help you and your team keep track of all the bugs and tasks.
I'm trying to work with MSBuild and TFS.
I've managed to create my own MSBuild script, that works great from the command-line. The script works with csproj files, and compiles, obfuscate, sign and copies everything that's needed.
However, looking at the documentation of TFS & Team Build, it appears that it expect solutions as the "input" for the script.
Also, I haven't found an easy/intuitive way of performing a "Get Latest Version" from the TFS as part of the script. I'm assuming that the Team Build automatically do a "Get Latest" on the solutions it's suppose to compile, but again - I don't (want to) work with solutions...
Any insights? any pointers? any links?
Team Build defines about 25 targets of its own. When you queue a Team Build, they are automatically run for you in the predefined order listed # MSDN. Don't modify this process. Instead, simply set a couple of these properties that determine how the tasks behave. For example, set <IncrementalGet> to "true" if you want ordinary Get behavior, or "false" if you want something closer to tf get /force.
As far as running your own MSBuild script, again this shouldn't be necessary. Start with the TFSBuild.proj file that's provided for you. It should only require minimal modifications to do everything you describe. Call your obfuscation & signing code by overriding a task like AfterCompile or AfterTest. Put your auto-deploy code in AfterDropBuild. Etc.
Even really complex scenarios are possible if you refactor appropriately. See past answers #1 #2.
As far as the actual compile, you're right that Team Build operates on solutions. I recommend giving it what it wants. I'll be the first to admit that *.sln files are ugly and largely undocumented, but at least you're offloading the work to a well tested & supported product.
If you really wanted to, you could give it a blank/dummy solution and override the CoreCompile task with your custom compiler logic. But this is really asking for trouble. At bare minimum, you lose all of Team Build's flexibility WRT building multiple platforms and flavors. More practically, you're bound to spend a lot of time debugging something that's designed to "just work" -- and there are no good MSBuild debuggers yet (that I know of). Not worth it, IMO.
BTW, the solution files do not affect the Get process. As you can see in the 1st link, the Get is done very early on, long before Team Build even reads the solution file(s). Apart from a few options like <IncrementalGet>, this is not controlled from MSBuild at all -- in particular, the paths to be downloaded are determined by the workspace mappings associated with the build definition. I.e., they are stored in the Team Build SQL database, not the filesystem, and managed with tools (like Team Explorer) that call the TFS webservice API.