We use CppUTest to run unit tests.
This is being performed by Cmake/Ninja where after building the tests, we use ninja to execute them ninja test
an example output of this is:
1/3 Test #1: Test1................................................... Passed 0.03 sec
Start 2: Test2
2/3 Test #2: Test2......................................................... Passed 0.00 sec
Start 3: Test3
3/3 Test #3: Test3..............................................................***Exception: SegFault 0.00 sec
66% tests passed, 1 tests failed out of 3
Total Test time (real) = 0.26 sec
The following tests FAILED:
3 - Test3 (SEGFAULT)
Errors while running CTest
FAILED: CMakeFiles/test.util
This is ok if i trigger the build locally on my machine and analyze it manually. Now what i am looking for is an already existing solution to help jenkins analyze the output.
Right now, Jenkins executes the build and exits "successfully", because the command itself ninja test executed successfully, but not all of the tests.
Maybe you already found this but you can create a JUnit output with cpputest with the -ojunit output flag. Jenkins should then be able to import the results from this file.
CppUTest Commandline Switches
Related
My documentation tests are silently not executed in my Docker environment while everything works on both Windows and Ubuntu/Debian hosts.
I created a minimal Github Repository to demonstrate the issue. I tried two different versions of Rust nightly and Rust stable, debug/release, all without success. See my Dockerfile and complete build output.
Example code:
/// Fixes string arrays which can also be objects into string arrays
/// # Examples
///
/// ```
/// assert_eq!(cargo_test_doc_docker::add(1, 2), 3);
/// ```
pub fn add(a: i32, b: i32) -> i32 {
a + b
}
Result when executing on Debian:
arturh#host:~/projects/cargo-test-doc-docker$ cargo test
Compiling cargo-test-doc-docker v0.1.0 (/home/arturh/projects/cargo-test-doc-docker)
Finished test [unoptimized + debuginfo] target(s) in 2.39s
Running target/debug/deps/cargo_test_doc_docker-9d5ae146cd4c3628
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Running target/debug/deps/cargo_test_doc_docker-2a696d2579128ce1
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Doc-tests cargo-test-doc-docker
running 1 test
test src/lib.rs - add (line 4) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
The problem occurs when executing the build on Docker. This is a minimal Dockerfile that reproduces the problem:
FROM ekidd/rust-musl-builder:nightly-2020-01-26-openssl11 as build
COPY --chown=rust:rust . .
RUN cargo test; echo $?
Result for every Rust toolchain I tried:
Step 6/17 : RUN cargo test; echo $?
---> Running in b266fc72f3c1
Compiling cargo-test-doc-docker v0.1.0 (/home/rust/src)
Finished test [unoptimized + debuginfo] target(s) in 0.32s
Running target/x86_64-unknown-linux-musl/debug/deps/cargo_test_doc_docker-7b40e7e5b47f49eb
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Running target/x86_64-unknown-linux-musl/debug/deps/cargo_test_doc_docker-0bfec9752a7bec14
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
0
It does not even try to execute any doc tests and exits with zero so it's not easily noticed. I guess it must be something the Docker base image does, but what could that be?
Cross Compilation
This is a logical, if surprising, outcome of cross-compilation.
To understand why, imagine that you:
Compile on a Linux x64 machine (Host).
Target a Windows ARM machine.
The generated code cannot be executed on the current host (Linux x64): it is prepared for a different CPU (instruction set) and OS (system calls).
Since the tests -- unit tests, integration tests, and documentation tests -- are also generated for the target architecture, they cannot be executed on the host either.
What to do with the tests?
If your code has no specific dependency on a specific platform, then you can content yourself with compiling for the host and running those.
Otherwise, you will need access to a machine that can actually run the cross-compiled binaries. You can still use cross-compilation to speed up building those binaries, and then upload them to either a physical or virtual machine to run them.
AFAIK Cargo does not help with the latter, so you'll need your own scripts.
Shepmaster was right, when I target the x86_64-unknown-linux-musl it also does not work locally on Debian:
arturh#host:~/projects/cargo-test-doc-docker$ cargo test --target=x86_64-unknown-linux-musl; echo $?
Compiling cargo-test-doc-docker v0.1.0 (/home/arturh/projects/cargo-test-doc-docker)
Finished test [unoptimized + debuginfo] target(s) in 0.28s
Running target/x86_64-unknown-linux-musl/debug/deps/cargo_test_doc_docker-8dfff5631875d404
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Running target/x86_64-unknown-linux-musl/debug/deps/cargo_test_doc_docker-eb877250b708174b
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
0
I guess I must have a separate build step for testing the doc tests with the target x86_64-unknown-linux-gnu.
I have created a jenkins "freestyle" job, in which I am trying to run multiple BDD testing process. Following is the "commands" I have put in "Jenins/Build/execute shell" section:
cd ~/FEXT_BETA_BDD
rm -rf allure_reports allure-reports allure-results
pip install behave
pip install selenium
pip install -r features/requirements.txt
# execute features in plan section
behave -f allure_behave.formatter:AllureFormatter -f pretty -o ./allure-reports
./features/plan/*.feature
# execute features in blueprint section
behave -f allure_behave.formatter:AllureFormatter -f pretty -o ./allure-reports
./features/blueprint/*.feature
What I have found is in Jenkins, if there is any test case intermittent failure, such message is shown in the Console Output:
"
...
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
3 steps passed, 1 failed, 1 skipped, 0 undefined
Took 2m48.770s
Build step 'Execute shell' marked build as failure
"
And the leftover test cases are skipped. But if I was to run the behave command on my local host directly, I don't get this type of behaviour. The failure will be detected and the remaining test cases continues till all are finished.
So How may I work around this issue in Jenkins ?
Thanks,
Jack
You may try the following syntax:
set +e
# execute features in plan section
behave -f allure_behave.formatter:AllureFormatter -f pretty -o ./allure-reports
./features/plan/*.feature || echo 'ALERT: Build failed while running the plan section'
# execute features in blueprint section
behave -f allure_behave.formatter:AllureFormatter -f pretty -o ./allure-reports
./features/blueprint/*.feature || echo 'ALERT: Build failed while running the blueprint section'
# Restoring original configuration
set -e
Note:
Goal of set -e is to cause the shell to abort any time an error occurs. If you will see your log output, you will notice sh -xe at the start of execution which confirms that Execute Shell in Jenkins uses -e option. So, to disable it, you can use +e instead. However, it's good to restore it once your purpose is fulfilled so that subsequent commands produce expected result.
Ref: https://superuser.com/questions/1113014/what-would-set-e-and-set-x-commands-do-in-the-context-of-a-shell-script
The ConsoleOutput from the SummaryReporter above indicates that you have only one feature with one scenario (that fails). Behave has no such thing that it stops when the first scenario fails.
An early abortion of the test run can only occur if critical things happen:
A failure/exception in the before_all() hook occurs
A critical exception is raised (SystemExit, KeyboardInterrupt) to end the test run
Your implementation tells behave to abort the test run (make sense on critical failures when all other tests will also fail; why waste the time)
BUT: If the test run is aborted early, all the features/scenarios that are not executed yet are reported as untested counts in the SummaryReporter.
...
0 features passed, 1 failed, 0 skipped, 2 untested
0 scenarios passed, 1 failed, 0 skipped, 3 untested
0 steps passed, 1 failed, 0 skipped, 0 undefined, 6 untested
HINT: Untested counts are normally hidden. They are only shown if the counter is not zero (greater than zero).
This is not the case in your description.
SEE ALSO:
behave: features/runner.abort_by_user.feature
I wrote a function for dbatools called New-DbaSqlConnectionStringBuilder. I wrote unit tests for it. I know these unit tests cover most of the function. I am getting 0% code coverage report with the following command.
Invoke-Pester .\tests\New-DbaSqlConnectionStringBuilder.Tests.ps1 -CodeCoverage .\functions\New-DbaSqlConnectionStringBuilder.ps1
Abridged output below:
**********************
Running C:\Users\zippy\Documents\dbatools\tests\New-
. . .
Unit tests happen
. . .
Passed: 16 Failed: 0 Skipped: 0 Pending: 0 Inconclusive: 0
Code coverage report:
Covered 0.00% of 21 analyzed commands in 1 file.
To get this version of the code:
git clone https://github.com/zippy1981/dbatools.git
cd dbatools
git checkout testing/PesterCodeCoverage
Import-Module .\dbatools.psd1
What am I doing wrong?
Just psychic debugging:
Your module is installed and your test are running against the module instead of the: ' .\functions\New-DbaSqlConnectionStringBuilder.ps1' file.
Doing some proof of concept I've a simple netcore repo with some xUnit tests at NetCoreXunit that I've got to build on both Appveyor and Travis. I've put in a failing test and Appveyor fails the build but I'm struggling to get Travis to do the same. It executes the tests happily and reports one of the tests fails but passes the build.
I've Googled to death and been trying to pipe and parse the output in a script step in the yaml configuration but my script knowledge is not great.
If anyone could help me get Travis to fail the build I'd be grateful. There's a link from the GitHub repo to both my Appveyor and Travis builds and if you commit to the repo it should build automatically.
--UPDATE--
So I got it as far as parsing the output of two test assemblies and correctly identifying if there's been a test failure; but I need to create a variable so both assemblies get tested before throwing the exit. I've had to jump through silly hoops to get this far; and one was I can't seem to define a variable without Travis complaining. It's also hardcoded and I'd like to extend it to finding all test assemblies not just the hardcoded.
after_success:
# Run tests
- dotnet test ./src/NetCoreXunit -xml ./out/NetCoreXunit.xml;
if grep -q 'result="Fail"' ./out/NetCoreXunit.xml ; then
echo 'Failed tests detected.';
else
echo 'All tests passed.';
fi;
- dotnet test ./src/NetCoreXunitB -xml ./out/NetCoreXunitB.xml;
if grep -q 'result="Fail"' ./out/NetCoreXunitB.xml ; then
echo 'Failed tests detected.';
else
echo 'All tests passed.';
fi;
Any advice appreciated: how do I get a list of all test assemblies and how do I declare and set a bool that I can then exitcode with?
Spent way to long trying to get .travis.yml to work; should have just gone straight down the Python route; works as follows called out to from the yml.
import os
import sys
import re
from subprocess import call
root_directory = os.getcwd()
print (root_directory)
regexp = re.compile(r'src[\/\\]NetCoreXunit.?$')
result = False
for child in os.walk(root_directory):
print (child[0])
if regexp.search(child[0]) is not None:
print ("Matched")
test_path = os.path.join(root_directory, child[0])
if os.path.isdir(test_path):
print ("IsDir")
print (test_path)
os.chdir(test_path)
call (["dotnet", "test", "-xml", "output.xml"])
if 'result="Fail"' in open("output.xml").read():
print (test_path + ": Failed tests detected")
result = True
else:
print (test_path + ": All tests passed")
os.chdir(root_directory)
if result:
print ("Failed tests detected")
sys.exit(1)
echo how many testcases
read s1
echo Enter the Testcases
for (( c=1; c<=$s1; c++ ))
do
read a1
a[$c]=$a1
#echo ${a[$c]}
done
for (( c=1; c<$s1; c++ ))
do
str=${a[$c]}'|'
str1=$str1$str
done
str1=$str1${a[$c]}
echo $str1
str1=\($str1\)
echo $str1
CMD="ruby final2.rb --name "\"\/test_$str1\/\"
#echo $CMD
$CMD
i have the testsuite final2.rb which contains test_1 test_2 test_3 test_4 test_5 test_6 test_7 as testcases in it.
Above i have created a script that will take only the number of testcases to run like
1
2
5 these will be converted to the pattern ruby final2.rb --name "/test_(1|2|5)/"
As we know this command runs the testcases:-
test_1 test_2 test_3 in the testsuite final2.rb.
but when executed using Bash Script the test suite runs only for a milli seconds like..
DEMO
*Loaded suite final2
Started
Finished in 0.000135 seconds.
0 tests, 0 assertions, 0 failures, 0 errors*
but if i write the same command ruby final2.rb --name "/test_(1|2|5)/"
in termial myself the desired testcases runs and the output is
***Loaded suite final2
Started
Finished in 124.1212135 seconds.
3 tests, 6 assertions, 0 failures, 0 errors***
so
runnig a commad in terminal is working and then runing same command by script is not working...
any suggestions..
help
to ran system command you need to wrap it in this quotes "`":
`CMD="ruby final2.rb --name "\"\/test_$str1\/\"`
another aproaches: System call from Ruby