I am putting together a bunch of applications to create an automatic building for microsoft platform (the products I chose and the software I will build, both, runs on windows). The products I've chosen are:
Code repository: SubVersion
Continuous integration: CruiseControl
Unit testing: NUnit
Test coverage: NCover
Static code analysis: FXCop
Now I need to choose a bug/issue tracking system (free if possible) that can be, in some way, integrated with the previous products.
What I mean by integration? Well, all these products have a file as output I want to be able to publish errors and bugs found by them into the tracking system.
Do you know some product, some technique or trick that can help me to do this?
Thanks in advance.
First off, these are all tools I have experience with and congratulate you on your choices - these tools will serve you well if you use them wisely.
The most common usage of these tools is that CC would fail the build if certain criteria are not met, e.g.:
A unit test fails
Code coverage falls below a certain threshold
FXCop detects a violation of a certain severity
Because the build would fail and in continuous integration a failed build should be fixed immediately, you wouldn't really need to put those issues into a bug tracking system. Think of build-failing errors as being as severe as the code not compiling - you drop everything and fix right away.
Related
My managers want we to determine which tests might have to be run, based on coding changes that were made to the application we are testing.
But, it is hard to know which tests are actually needed to be re-verified as a result of a code change. What we have done is common to test the entire area where the code change occurred / or the entire proj, solution.
We were told this could be achieved by TFS build or MTM tools. Could someone share the details?
PM:We are running on TFS 2015 update4,VS2017.
There is a concept of Test Impact Analysis which helps in analysis of impact of development on existing tests. Using TIA, developers know exactly which tests need to be verified as a result of their code change.
The Test Impact Analysis (TIA) feature specifically enables this – TIA
is all about incremental validation by automatic test selection. For a
given code commit entering the pipeline TIA will select and run only
the relevant tests required to validate that commit. Thus, that test
run is going to complete faster, if there is a failure you will get to
know about it faster, and because it is all scoped by relevance,
analysis will be faster as well.
Test Impact Analysis for managed automated tests is available via a checkbox in the 2.* preview version of the VSTest task.
If enabled, only the relevant set of managed automated tests that need to be run to validate a given code change will run. Test Impact Analysis requires the latest version of Visual Studio, and is presently supported in CI for managed automated tests.
However this is only available with TFS2017 update1(need 2.* preview version of VSTS task). More details please refer this blog: Accelerated Continuous Testing with Test Impact Analysis
How does CircleCI and other CI tools help ?
I am not able to fully understand the internals of these tools and how they help with faster deployment of apps.
Are these tools useful only for github based open source projects ? Since the testing requirements for every app is different how is it possible to seamlessly automate it using the CI tools ?
The reason you are using Continous Integration (CI) is to have a well defined build system and always a releasable latest successfullbuild.
You can also integrate unit tests or integration tests.
I think it is not only useful for github-based projects, but also for projects with projects where more developers develop in parallel.
For more information: Wikipedia: Continous-Integration
CI is a developer practice which allows them to integrate code into their shared repository several times a day along with other developers in their team.
Each time a developer checks-in code its been verified auto-build and detect errors early.
It solves issues like
long and intense builds
spend less time in debugging
build more features quickly
How it works
CI server monitors for any changes in the code repo and starts building unit & integration tests. It assigns label for each version of build and also informs the team for a successful build or if it fails, the team fixes the issue and again starts to integrate.
I am new in configuration of TFS.
Currently our project is 50% done but we found that we have very bad code. We consider the need for static code analysis like Resharper or another product like StyleCop, CodeAnalysis and FxCop.
We want configure the TFS to reject a checkin when that check in contains code that triggers code analysis warnings.
But for the previous code we want to suppress the existing warning to prevent the code from becoming worse than it already is.
As Ivan mentions, your root cause it not in the lack of analysis tools, but probably in the level of quality and rigor agreed (or currently being enforeced between team members) between the development team and their project's sponsor. It may be that the pressure on the team is too high, causing important review actions to be skipped, or that the team (or the sponsor!) doesn't have the same desire to quality as you or the sponsor. Or that the team doesn't have the right level of knowledge to prevent these issues from happening.
The best way out of this is to fix as much as you can in a short period of time.
Warning: I've experienced with a number of teams the effect of turning on too many rules all at once. Generally, there is a reluctance for people to concede that their work hasn't been up to par and turning on rules that do not directly cause bug ("The identifier is cased incorrectly" for example) can cause frustration that can severely hamper your momentum. Carefully selecting which rules need to be solved now and which can wait for later worked much better in my experience. Once the team has developed a way to solve these kinds of problem, you can easily apply more.
Turning on Tools like configuring Code Analysis for your solution or using the Solution Wide Analysis feature of Resharper, can help you spot issues, but it won't solve them or prevent similar issues from popping up in the future unless your team stops creating them.
Tip: Note that you can turn on Resharper during your build as well using the Resharper CLI features.
StyleCop I would not enforce on this team (just yet) if the code itself is bad enough to trigger massive warnings that may hold bugs and issues. Fix these problems first, make the code it pretty later. Your priorities are now to remove any possible bugs.
CodeAnalysis and FxCop are the same things, so you won't need to turn on both. A tool like Resharper can help your developers to quickly remove a lot of the issues by using the magic-key ALT+ENTER.
If you want to create a clean baseline you can run code analysis once, then select all warnings that are generated and then select Suppress in global suppression file. This will work for Code Analysis issues, but won't suppress any Compiler Warnings, there is no easy way to quickly suppress all current compiler warnings.
Tip: It sometimes helps to temporarily rename any existing globalsupressions.cs files, so that this "baseline" is stored separately. You then know which warnings you'll have to fix at a later point in time.
Tip: When a developer suppresses a warning, have them add a Justification="Reason for suppression" to the suppression that is generated, that way you can distinguish between carefully considered suppression and temporary ones.
Depending on whether you already have a build server your next step is to install Team Build and once you have a build server you'll need to setup a Build Definition. This blog post covers most of the steps.
In the build definition set the trigger to "Gated Checkin" and on the Process tab make sure you set Code Analysis to "Always". If you want to fail your build based on Code Analysis errors, you need to create a custom ruleset and configure that for your solution.
To have compiler errors fail the build you can also enable the "Treat Warnings as Errors
Once you have enabled your gated check-in build all developers changes will be prompted to wait for their build to finish. You can turn on alerts (using Web access) or use the Build Notification Tool to get notified when the changes were successfully submitted.
Tip: Instead of turning on all rules at once (or switching them all to cause an ERROR during builds) you can also opt to turn on rules a couple at a time and fix them. Turning on rules by category gives you a nice opportunity to teach people the importance of the rules being turned on and possible solutions for fixing them.
A far more advanced solution would be to install and configure SonarQube alongside your Team Build environment. The ALM Rangers and Sonar have recently worked together to create installation guidance and a number of extensions to enable Team Build and SonarQube integration. You can find the installation guide here.
I am in the process of setting up continuous integration in our TFS system. One major part of our system are the development of about 50 DotNetNuke modules to support our CMS infrastructure. Right now, each of those projects have their own solution since their code bases are mostly siloed (with common code in 1 or 2 common projects). Keeping them in their own solution is done because it makes the development process faster (loading, compiling, etc....)
However, this has proven difficult to maintain when setting up TFS team build as each solution has to be manually added to the build definition and MSBuild seems unable to take advantage of parallel compiling due to each project being in its own solution. This causes about 5 minute full build times, which while isn't horrible isn't ideal. Mostly though, it's not ideal from a build definition maintenance aspect.
To solve this I creating a global solution that included all projects. The idea being that if you want your project to be automatically compiled and deployed by TFS you will have to include your project in the global solution. This seems works well, as it's easy to maintain from a build definition standpoint and brings the total build time down to 70 seconds.
The one problem is that the displayed TFS build log groups all warnings and errors together under the solution instead of separating them out by project. This makes it difficult to quickly see what project caused which errors and warnings.
Is there a good way to see project level error/warning messages in the build log summary view without delving into the cluttered build log?
To answer your direct question, I believe the answer is no (at least not without some heavy customization).
For me this is never a big concern as I am pretty aggressive about getting my teams to bring errors/warnings down to zero, then enforcing it via TFS Build (/p:TreatWarningsAsErrors=true). This means you should never have to wade through hundreds of warnings in the build summary.
If you add all your individual solutions to the build definition, you can always use the TFS Power Tools to "clone" a build def to make maintenance easier. You could also modify the Build Template to build the solutions/projects in parallel, although this runs the risk of having file contention issues.
What kind of agile tools are you using for Erlang development? What continuous integration (CI) server are you using to build Erlang code? The only reference I got was from Quora question How do I integrate Erlang unit tests in Jenkins (Hudson)?.
I am also interested in the nifty details of setting them up and making talk to each other.
As a company using Erlang actively, Klarna (www.klarna.com) use Jenkins (formerly Hudson) for daily regression test on nearly every dev commit. It's an org with about 80 people total in rnd and we use distribute mode of Jenkins which allows us to have more than 10 build slaves mastered by only one Jenkins server. Basically we have a code base with Eralng code which is version controlled by tools like svn or git. All these testcases are under common test framework and all works well under Jenkins.
Previously, we tried Cruise Control and gave it up since Jenkins does much better.
As Lukas mentioned, you probably will need a tool to gen xml files sine common test doesn't export them directly. Haven't really tried that module though, we do have an implementation of common test event handler to do the job, but it was abandoned due to performance, we do have a a critical requirement on test time. right now, we use a own made script to export xml from common test log directly.
There are a lot more you could do with Erlang and Jenkins, like code coverage analyze if you compile properly and export formatted xml to Cobertour plugin, gui test with selenium etc.
For setting up Jenkins, I think Jenkins home page has a good introduction.
Regarding agile tools, I guess it's really hard to define what a agile tool. Also what I believe is it's very much depend on the size of you org. You will probably need a good process view tool (team level or depart level), a good ticket tracking tool, code review tool, communication tool. There are bunch of them implemented under open source. According to our exp, none of them seems to be able to work seamlessly with Jenkins which means you will need to select and tweak by your own requirement. BUT that's the beauty of open source isn't it :)?
If you want to do it using Jenkins, I have written a common test hook which generates JUnit XML output for your tests which Jenkins can use to produce test statistics.
https://github.com/garazdawi/cth_tools/blob/master/src/cth_junit.erl
We use Jenkins for our Python code, so I think you may use Jenkins with Erlang code.
We use buildbot with our own recipes to hook unit tests.