I was just wondering if there was a way to analyse TFS, to figure out code coverage results against a specific developer.
Say I want to determine the code coverage statistics for one of the developers in my team.
It’s not supported out of the box.
Don’t think it will be in the future, since its a very hard measurement:
Developes write source code. Source code within a project is usually divided in functionality not the developers. Code coverage results is the coverage of a given assembly being exercised in a testrun. We therefore need to analyse the coverage per code line and relate that code line back to a given developer changeset. Regardless of the unittest dll being instrumented, both the code in the unittest as the code being exercised is involved in the code coverage result. So which codeline being covered is counted for a specific developer ? The lines unittest, a line in a shared library, a line that was changed by 4 developers (is coverage shared), othes issues ?
But why are you asking this question, if you are trying to improve the quality of a specific individual code reviews and pair programming would be a more efficient approach. Even if it would be possible, terrorizing individuals on their code coverage would only result in dysfunctional measurement. Source code of a given product is shared in a Team, therefore the team is responsible for the coverage. Require your team to take that responsibility.
Related
I am trying to understand the concept of code coverage and a complete novice to this topic.
I am using Eclemma to measure the code coverage of an open-source code. Can somebody help me to know what are important insights I should consider into the below snapshot?
Code coverage is a metric, which expresses which portion of the (application) code gets executed when you run your test cases. However, it is just a measure of completeness; it provides no information about how thoroughly the executed code was tested by the test cases.
In your screenshot, the third line of the table (src/main/java) is a relevant one. It expresses that the application code consists of 3,846 (bytecode) instructions; out of these, roughly 67% were executed (presumably by the automated test cases residing in src/test/java). This means that the test cases cannot reveal any fault in one third of the whole application because the test cases do not touch that code at all. The remaining code (other two thirds) is executed by at least one test case. Test cases can reveal faults in this code; how effectively they do depends on their used input data and oracles.
Note that it is often not possible or sensible to achieve 100% coverage.
I'm a young maths teacher and I'm introducing algorithms to my students.
They learn C basics to write their programs.
I have more 400 students and testing their programs takes me a lot of time.
Already wrote a shell script but they need to send me the binary.
So I would like to set up a platform where they can upload their binaries or their sources and test it.
The school gave me a Jenkins and a web instance.
I did read some docs about Jenkins and JUnit but I don't see how can I test output of another program.
Thanks for your answers.
You don't have to use Jenkins and JUnit necessarily. You don't need anything else than your own laptop and a bash script. They are just doing C programming...
First, gather your students code in an organized manner (send code by mail or download them from the platform). Then, prepare some input and output files (which correct results obviously) and write a tiny script which will execute your students code with these prepared inputs. The script will finally compare the results with your prepared outputs.
Some notes to make things easier because students always do things their own way:
be precise on how they have to compile their program (flags, given makefile...)
be clear on the input and output format
I tell them to give their program in a specific way (name of program with NAME or ID)
If they don't follow my guidelines they get minus points. This is so much important because they have to learn to be rigorous and in the same time you also gain tremendous time. They have to put the effort to make your correction process efficient especially when you have so much students.
But well you can also dig in Jenkins doc it might be worth it (although I do not use it personnally). And why can't you ask your school to help you for that ?
Edit: Maybe this CMake/C++ Jenkins topic might be help you.
I find entering the data in the feature file of specflow very painful specially when it is repetitive and large data. Can we use an external data source like spreadsheet to enter this data and then use this external datasource in the feature file?
It's theoretically possible, but probably so much effort that you wouldn't want to do it.
The problem is that the feature file is simply a human readable form. When it is saved in Visual Studio it is parsed and converted into the feature.cs file and that is the one that is compiled and used for testing.
So your process would become
edit spreadsheet
export to feature file
get specflow's VS plugin to convert to feature.cs
run msbuild
run tests via Nunit or similar
I wouldn't do this. Instead I'd focus on getting my tests to be better examples. It sounds like you are to trying to exhaustively cover every possibility. Don't come up with examples to cover every possible case, but instead cover as much logic as possible with fewer tests.
I have a fairly large MVC3 application, of which I have developed a small first phase without writing any unit tests, targeted especially detecting things like regressions caused by refactoring. I know it's a bit irresponsible to say this, but it hasn't really been necessary so far, with very simple CRUD operations, but I would like to move toward a TDD approach going forward.
I have basically completed phase 1, where I have written actions and views where members can register as authors and create course modules. Now I have more complex phases to implement where consumers of the courses and their trainees must register and complete courses, with academic progress tracking, author feedback and financial implications. I feel it would be unwise to proceed without a solid unit testing strategy, and based on past experience I feel TDD would be quite suitable to my future development efforts here.
Are there any known procedures for 'converting' a development effort to TDD, and for introducing unit tests to already written code? I don't need kindergarten level step by step stuff, but general strategic guidance.
BTW, I have included the web-development and MVC tags on this question as I believe these fields of development can significant influence on the unit testing requirements of project artefacts. If you disagree and wish to remove any of them, please be so kind as to leave a comment saying why.
I don't know of any existing procedures, but I can highlight what I usually do.
My approach for an existing system would be to attempt writing tests first to reproduce defects and then modify the code to fix it. I say attempt, because not everything is reproducible in a cost effective manner. For example, trying to write a test to reproduce an issue related to CSS3 transitions on a very specific version of IE may be cool, but not a good use of your time. I usually give myself a deadline to write such tests. The only exception may be features that are highly valued or difficult to manually test (like an API).
For any new features you add, first write the test (as if the class under test is an API), verify the test fails and implement the feature to satisfy the test. Repeat. When you done with the feature, run it through something like PEX. It will often highlight things you never thought of. Be sensible about which issues to fix.
For existing code, I'll use code coverage to help me find features I do not have tests for. I comment out the code, write the test (which fails), uncomment the code, verify test passes and repeat. If needed, I'll refactor the code to simplify testing. PEX can also help.
Pay close attention to pain points, as it highlights areas that should be refactored. For example, if you have a controller that uses ObjectContext/IDbCommand/IDbConnection directly for data access, you may find that you require a database to be configured etc just to test business conditions. That is my hint that I need an interface to a data access layer so I can mock it and simulate those business conditions in my controller. The same goes for registry access and so forth.
But be sensible about what you write tests for. The value of TDD diminishes at some point and it may actually cost more to write those tests than it is to give it to someone in India to manually test.
I am not asking the static code analysis which is provided by StyleCop or Fxcop. Both are having different purpose and it serves well. I am asking whether is there a way to find the code coverage of your user control or sub module? For ex, you have an application which uses the helper classes in a separate assembly. Inorder to ensure the unit testing code coverage, we need to run the application and ensure using NCover or similar tool.
My requirement is, without running it, is there any possible to find code coverage of the helper classes or similar kind of assemblies?
See Static Estimation for Test Coverage for a technique that estimates coverage without executing the source code.
The basic idea is to compute a program slice for each test case, and then "count" what the slice enumerates. A (forward) slice is effectively that part of a program that you can reach from a specific starting point in the code, in this case, the test code.
While the technical paper above is hard to get if you're not an ACM member [or you didn't attend the conference where it was presented :], there's a slide presentation here.
Of course, running this static estimator only tells you (roughly) what code will be exercised. It doesn't substitute for actually running the tests, and verifying that they pass!
In general, the answer is no. This is equivalent to the halting problem, which is not computable.
There are (research) tools based on abstract interpretation or model checking that can show coverage properties without execution, for subsets of language. See, e.g.
"Analyzing Functional Coverage in Bounded Model Checking", Grosse, D. Kuhne, U. Drechsler, R. 2008
In general, yes, there are approaches, but they're specialized, and may require some formal methods experience. This kind of stuff is still cutting edge research.
I would say no; with the exception of 'dead code' which a compiler can determine.
My definition of code coverage is a result which indicates how many times each line of code is run in your program: which, of course, means running the program. The determining factor here is usually the values of data passing through the program which the determine the paths of executions taken by conditionals. A static analysis, like a compiler, could deduce lines of code that cannot run under any conditions.
An example here is if your program uses a third-party library, but there is a bug in the library. If your program never uses those parts of the library, or the data you send to the library causes it to avoid the bug, then you won't be affected.
You could write a program that, by reflection, assumes that all conditionals will be taken, and follows all function calls, through all derived classes, but I'm not sure what this will tell you. It certainly can't tell you whether or not there are any bugs in the lines of code covered.
Coverity Static Analysis is a tool that is can identify many secuirty flaws in a program. It can also identify dead code and can be used to help satisfy testing regulations such as D0178B which requires that the developers demonstrate that all code can be executed.
If you are using Visual Studio, you can first run 'Analyze Code Coverage', Then you can export code Coverage results using below Button(marked in Green) in Visual Studio:
Later you can import the Coverage Result file back to Visual Studio