I have a strange coverage result in visual studio:
it appears as the if statement was not covered but since we did enter the block it must have been. Why are those results wrong? (those result cover many runs under many conditions)
screenshot
I suspect what you are seeing is branch coverage due to not testing all combinations that can result in going down each path. Logical AND (&&) allows early escape i.e. it only evaluates the second operator if the first is true.
e.g.
if (bool.TryParse(savePrep, out save) && save)
has 3 possibilities
savePrep = "true"
savePrep = "false"
savePrep = "neither-true-nor-false"
you have probably only exercised #1 and #2
Related
I'm trying to describe a scenario of my app on gherkin language so that I can use it as executable spec. The scenario is more less the following: There's a phase of a process in which a check is performed. If all conditions for the check are fulfilled then the process end. Otherwise, the process waits for any condition to change (it's notified about this) and the checks again, finishing if succesful. What I'm having trouble describin is this waiting part. My current version (simplified) is:
Given condition A
And not condition B
When the check is performed
Then the result is negative, pending condition B
What I'm trying to express with pending condition B is that the test will be repeated once condition B changes, but I don't particularly like this version, since it's hard to turn one to one to a test (the fact that condition B changes would be a new When).
Can anybody with more experience come up with a better formulation?
You can either link the two tests together, like this:
Scenario: When A and not B result is negative, but if B happens then result is positive
Given condition A
But not condition B
Then the check returns negative
But if condition B
Then the check returns positive
Which might not be best practice but is sometimes the pragmatic way of doing things, especially if the tests are slow running because of the system under test or your test environment etc.
Or you could make it into two scenarios with some repetition behind the scenes.
Scenario: When A and not B the result is negative
Given condition A
But not condition B
Then the check returns negative
Scenario: When A and B the result should be positive
Given the system has condition A but not B
And the check is returning negative
When condition B
Then the check returns positive
In your case I would say that which one to choose depends on how long your tests take to run. If they are slow then go for one big scenario. If they aren't, or it doesn't matter for some reason then go for the second suggestion. The second suggestion will give more information about the cause of the failure which is nice to have, but if the tests are slow then I think it would still be quite obvious why the the test was failing even if you are using one big scenario.
I have been struggling with parallel and async constructs in F# for the last couple days and not sure where to go at this point. I have been programming with F# for about 4 months - certainly no expert - and I currently have a series of calculations that are implemented in F# (asp.net 4.5) and are working correctly when executed sequentially. I am running the calculations on a multi-core server and since there are millions of inputs to perform the same calculation on, I am hoping to take advantage of parallelism to speed it up.
The calculations are extremely data parallel - basically the exact calculation on different input data. I have tried a number of different avenues and I continually run into the same issue - it seems as if the parallel looping never gets to the end of the input data set. I have tried TPL, ConcurrentQueues, Parallel.Array.map/iter and all the same result: the program starts out fine and then somewhere in the middle (indeterminate) it just hangs and never completes. For simplicity I actually removed the calculation from the program and I am just calling a print method, and Here is where the code is currently at:
let runParallel =
let ids = query {for c in db.CustTable do select c.id} |> Seq.take(5)
let customerInputArray= getAllObservations ids
Array.Parallel.iter(fun c -> testParallel c) customerInputArray
let key = System.Console.ReadKey()
0
A few points...
I limited the results above to only 5 just for debugging. The actual program does not apply the Take(5).
The testParallel method is just a printfn "test".
The customerInputArray is a complex data type. It is a tuple of lists that contain records. So I am pretty sure my problem must be there...but I added exception handling and no exception is getting raised, so have no idea how to go about finding the problem.
Any help is appreciated. Thanks in advance.
EDIT: Thanks for the advice...I think it is definitely deadlock. When I remove all of the printfn, sprintfn, and string concat operations, it completes. (of course, I need those things in there.)
Is printfn, sprintfn, and string ops not thread-safe?
Another EDIT: Iteration always stops on the last item..So if my input array has 15 items, the processing stops on item 14, or seems to never get to item 15. Then everything just hangs. Does not matter what the size of the input array is..Any ideas what can be causing this? I even switched over to Parallel.ForEach (instead of Array.Parallel) and same behavior.
Update on the situation and how I resolved this issue.
I was unable to upload code from my example due to my company's firewall policy, so in the end my question did not have enough details. I failed to mention that I was using a type provider which was important information in this situation. But here is what I figured out.
I am using the F# type provider for SQL Server and was passing around its Service Types which I suspect are not thread-safe. When I replaced the ServiceTypes with plain old F# Records, the code worked fine - no more deadlocks and everything completed without error.
I'm running some tests for a chip via Verilog, and I've run into a little bit of a problem where I am scratching my head a little. I'm testing coverage on the code to make sure all states happen over randomized testing of all parameters, etc.
In evaluating two values of the following type:
case(state_vector)
STATE1:
...
STATE2:
if(!var1 && var2)
state_vector = STATE1;
else
state_vector = STATE2;
STATE3:
...
Now the problem is that in doing coverage analysis the statement after the else statement is never reached, meaning that the if-statement always evaluates to true.
I originally assumed that the values of var1 and var2 were 0 and 1, respectively. Upon double checking before finishing my report I noticed that this assumption was incorrect, as a waveform analysis shows that var1 is always 1 and var2 is always 0 throughout the entire simulation.
Now I will test to make sure the values change the way I want them, but I'm curious as to how in Verilog this may happen. Essentially I am slipping into a state because the if-statement evaluates to true for infinitesimal unit of time.
Any ideas on how to better evaluate this problem? I'd like to check that another function isn't changing my state_vector at the same time I'm trying to check my current state.
Quick and dirty way is to sprinkle $display("%t %m got here",$time); around the code in question and make sure there are labels for the begin-end blocks eg:
begin : meaningful_label
$display("%t %m got here",$time);
... code ...
$display("%t %m got here too",$time);
end
If the display statement(s) are called, then state_vector is being assigned somewhere else. Otherwise the something something is preventing the code from exciting.
To further debug:
Not display message:
add more display messages to higher levels.
Displaying messages:
Some waveform viewers have active drivers tracing. If your viewer does not have this feature, then add messages around all other assigning statements and watch for time-stamp when the condition should be true.
currently I am using Z3 with Python.
I want to create an assertion-stack, where I can push a level and pop it later.
It should work exactly like any other "stack" with push- and pop-operations.
Therefore the SMTLIB2-standard defines two functions "push(n)" and "pop(n)" with optional numbers n. In my case n would always be 1.
But there seems to be some strange behaviour in Z3.
Why does following code result in "index out of bounds"?
s = Solver()
s.push() # expected: one new level on the stack, reality: emtpy stack
s.pop(1) # expected: stack is empty, reality: exception (index out of bounds)
If I add an assertions, Z3 works as expected.
s = Solver()
s.push()
s.add(True) # now there is one level on the stack,
s.pop(1) # pop is successful
Even this works correct:
s = Solver()
s.add(True)
s.push() # now there is one level on the stack,
s.pop(1) # pop is successful
The problem is, that I do not know, how many levels and how many assertions are created in my program. It is possible, that there is no assertion at all and only one level. Then the program would crash (or catch the exception). A workaround would be adding some simple formula like "True" always as a first step, but this seems ugly.
Is this a bug in Z3 or is this behaviour correct?
This bug has been fixed in the unstable (working-in-progress) branch.
It will be available in the next official release. In the meantime, here are some instructions on how to compile the unstable branch. The fix is also available in the nightly builds available at codeplex.
The article at onjava seems to imply that basis path coverage is a sufficient substitute for full path coverage, due to some linear-independence/cyclomatic-complexity magic.
Using an example similar to the article:
public int returnInput(int x, boolean one, boolean two)
{
int y = x;
if(one)
{
y = x-1;
}
if(two)
{
x = y;
}
return x;
}
with the basis set {FF,TF,FT}, the bug is not exposed. Only the untested TT path would expose it.
So, how is basis path coverage useful? It doesn't seem much better than branch coverage.
[Disclaimer: I've never heard of this technique before, it just looks interesting so I've done a few searches and here's what I think I've found out. Hopefully someone who knows what they're talking about will contribute too...]
I think it's supposed to be a better way of generating branch coverage tests, not a complete substitute for path coverage. There's a far longer document here which restates the goals a bit: http://www.westfallteam.com/sites/default/files/papers/Basis_Path_Testing_Paper.pdf
The onjava article says "the goal of basis path testing is to test all decision outcomes independently of one another. Testing the four basis paths achieves this goal, making the other paths extraneous"
I think "extraneous" here means, "unnecessary to the goal of basis path testing", not as one might assume, "a complete waste of everyone's time".
I think the point of testing branches independently, is to break those accidental correlations between the paths which work, and the paths you test, that occur with terrifying frequency when I write both the code and an arbitrary set of branch coverage tests myself. There's no magic in the linear independence, it's just a systematic way of generating branch coverage, which discourages the tester from making the same assumptions as the programmer about correlation between branch choices.
So you're right, basis path testing misses your bug, and in general misses 2^(N-1)-N bugs, where N is the cyclomatic complexity. It just aims not to miss the 2^(N-1)-N paths most likely to be buggy, as letting the coder choose N paths to test typically does ;-)
path coverage is no better than any other coverage metrics - it is just that metrics that shows how much of 'code' has been tested. The fact that you can achieve 100% branch coverage with (TF,FT) set of TCs as well as (TT,FF) means it is all up to your luck if your exit criteria tell you exit after 100% coverage is done.
The coverage should not be a focus for the tester - finding bugs should be and TC is just a way to show the bug just as well as coverage a proxy showing how much of this showing the bug activity has been done. As with all other white box methods - striving for max coverage with minimum costs require actually understanding the code so that you could actually write a defect w/o a TC. TC is just good for regression and as a documentation of the defect.
As a tester coverage is just a hint on how much has been done - only experience can be really helpful as to say how much is enough. As this is difficult to present in numerical values we use other methods i.e. coverage statistics.
Not sure if this makes sense to you I guess judging on the date you are far gone since the date you publish your question...
My recollection from McCabe's work on this exact subject is: you generate the basis paths systematically, changing one condition at a time, and only changing the last condition, until you can't change any new conditions.
Suppose we start with FF, which is the shortest path. Following the algorithm, we change the last if in the chain, yielding FT. We've covered the second if now, meaning: if there was a bug in the second if, surely our two tests were paying attention to what happened when the second if statement suddenly started executing, otherwise our tests aren't working or our code isn't verifiable. Both possibilities suggest our code needs reworking.
Having covered FT, we go back up one node in the path and change the first T to F. When building basis paths, we only change one condition at a time. So we are forced to leave the second if the same, yielding... TT!
We are left with these basis paths: {FF, FT, TT}. Which address the issue you raised.
But wait, you say, what if the bug occurs in the TF case?? The answer is: we should have already noticed it between two of the other three tests. Think about it:
The second if already had its chance to demonstrate its effect on the code independently any other changes to the execution of the program through the FF and FT tests.
The first if had its chance to demonstrate its independent effect going from FT to TT.
We could have started with the TT case (the longest path). We would have arrived at slightly different basis paths, but they would still exercise each if statement independently.
Notice in your simple example, there is no co-linearity in the conditions of the if statements. Co-linearity cripples basis path generation.
In short: basis path testing, done systematically, avoids the problems you think it has. Basis path testing doesn't tell you how to write verifiable code. (TDD does that.) More to the point, path testing doesn't tell you which assertions you need to make. That's your job as the human.
Source: this is my research area, but I read McCabe's paper on this exact subject a few years back: http://mccabe.com/pdf/mccabe-nist235r.pdf