I'm writing my own test program and I want to be able to re-use tools like Hudson for displaying the results of the test cases. I've so far gotten the results of the text file all into the same XML file and with success, failure, and errors.
Now I want to add the output of the test into the file. I have it set up so I can get the output of a test for each test individually, but I can't seem to figure out how to get it into the XML file in a way Hudson will recognize.
I want to do something like this...
<testsuite>
<testcase>
<success classname="...">
<stdout>
This is standard output
</stdout>
</success>
</testcase>
</testsuite>
But this doesn't get recognized. I see in the Ant source code that it's defined as "system-out", but I also see that it seems it wants the file in this format.
<testsuite>
<testcase classname="..." />
<system-out>
This is standard output
</system-out>
</testsuite>
Is there anyway to make this file so that I can have a specific stdout for each test case? Or do I have to make a new testsuite for every test case?
Edit: I seem to be able to get this format to work, but I'm still disappointed that I can't print the output during a success. I'd like it it while browsing tests, someone could see the output of that test.
<testsuite>
<testcase name="...">
<failure message="shows up as error message">
standard out (shows up as stacktrace)
</failure>
</testcase>
</testsuite>
Is there anywhere that shows what format Hudson accepts? I feel bad commiting bad revisions to source control just to get it to run on the automated build server.
I also can't seem to find where inside of Hudson the code for this functionality is.
Yes, you can not use the "success".
Code for this functionality is https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/tasks/junit/CaseResult.java
Related
I have configured my tests per the docs: https://support.atlassian.com/bitbucket-cloud/docs/test-reporting-in-pipelines/
I'm generating test reports with the console output captured though and it appears Bitbucket insists on excluding this output from the test tab.
I can find no documentation from Bitbucket on how this is configurable.
My report looks like this:
<?xml version="1.0" encoding="utf-8"?><testsuites><testsuite name="pytest" errors="0" failures="1" skipped="0" tests="1" time="3.402" timestamp="2023-01-28T20:38:30.862709" hostname="6745580f0e58"><testcase classname="tests.mytest" name="test_force_error" time="0.090"><failure message="ValueError: Some error.">Traceback (most recent call last):
File "/workspaces/app/tests/mytest.py", line 29, in test_force_error
assert blah.blah(
File "/workspaces/app/tests/mytest.py", line 91, in blah
raise ValueError(
ValueError: Some error.</failure><system-out>--------------------------------- Captured Log ---------------------------------
--------------------------------- Captured Out ---------------------------------
I WANT TO SEE THIS IN THE TEST TAB
</system-out><system-err>--------------------------------- Captured Err ---------------------------------
WARNING: I WANT TO SEE STDERR TOO
</system-err></testcase></testsuite></testsuites>
I the test tab in the bitbucket UI everything after </failure> is not shown.
How do I have bitbucket include </system-out> and <system-err>? I deliberately included this in the test reports so I could view it in the test tab for each test individually and not have to pour over the entire test output to see it.
The pipeline only detects files, as stated in the documentation you linked, it does not read your console logs. If you don't have option for a pre-made tester that can output such files you can always write the file yourself, as you have attempted but into a file.
My example web service is returning following XML.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<errorResponse>
<errorCode>Wrong ID</errorCode>
<errorId>2</errorId>
</errorResponse>
Following tests are passed.
response.then().body("errorResponse.errorId", Matchers.is("2"));
response.then().body("errorResponse.errorCode", Matchers.is("Wrong ID"));
response.then().body("errorResponse1.errorCode", Matchers.is("Wrong ID"));
response.then().body("errorResponse2.errorCode", Matchers.is("Wrong ID"));
I understand that first two tests are fine, what I am not getting is why the last two are getting passed?
Rest-assured uses its xml-path library to evaluate your hamcrest matcher and that library contains the XMLAssertion class which does the actual checking.
The source can be found on GitHub: https://github.com/rest-assured/rest-assured/blob/master/xml-path/src/main/groovy/io/restassured/assertion/XMLAssertion.groovy
At line 60 of this file you can see it removes the part of the search key before the first dot, because it recognizes we're evaluating from a root node.
Hence your key:
"errorResponse3.errorCode"
becomes
".errorCode"
So it turns out it doesn't matter what this initial path looks like, it assumes it's the name of the root node and discards it anyway.
I am using build.xml with phantom.js to execute the qunit testcases,test are passing but the results are not getting written to output xml file.qunit.done function of qunit.runner.js gives
Cannot write junit results file error in QUnit.done function.
any one has any idea/suggestion
thanks in advance
My first question. Please bear with me!
I have an Ant task that need to get some input from the user before proceeding. I use the Input task to achieve this. The input message will contain Swedish characters (eg å, ä and ö) but I am unable to get Ant to output the message properly. I'm testing this using the command line on a machine running Windows 7 Pro English (but obviously using a Swedish keyboard). Example:
<input message="åäö"/>
will output:
[input] Õõ÷
The build.xml is saved in UTF-8 format. If I do 'chcp' on the command line I get "Active code page: 850".
The same result can be seen when doing an echo:
<echo message="åäö"/>
will output:
[echo] Õõ÷
But in the case of the echo task I'm able to do:
<echo encoding="850" message=åäö">
to get the expected output:
[echo] åäö
The input task does however not have an encoding attribute and I'd very much prefer to not have to define an encoding at all, especially not on a per-task level (since I can't tell for sure on what machine the Ant script will be run from).
PS I have additional problems with the received input if it contains åäö and I set the input as a property that is later used in a filter copy task, but I guess that's a whole other question
I can observe the issue on my Polish Windows.
<script language="javascript">
java.lang.System.out.println("default charset: "
+ java.nio.charset.Charset.defaultCharset());
</script>
reports default charset as "windows-1250" while the console operates at "iso-8859-2" (I guess so).
Looks like <input> task uses the default charset thinking it would match the console input. In case it does not, I managed to override the encoding this way:
set ANT_OPTS=-Dfile.encoding=iso-8859-2
ant
In your case I would try to force 850, as it looks like JRE defaults to something else.
This question helped me to find the property name.
It is also important where ant is run from. If I run it from my ide, jedit console plugin, I don't need to override encoding, because I configured it to operate in windows-1250. So it seems to be another workaround, using an IDE.
I've setup teamcity with my sln file and got the unit tests to show up with the CppUnit plugin that teamcity has. And I get test results in the TeamCity UI.
Now I'm trying to get trending reports to show up for my unit tests and code coverage.
As of code coverage, we're using vsinstr.exe and vsperfmon.exe which produces an XML file.
I'm not quite sure as of what steps I should be taking to make the trending reports and code coverage(not as important) to show up.
I've already seen this post, but the answer seems to require editing the build script, which I don't think would work for my case since I'm building through MSBuild and the .sln file, and the tests are being ran through that build.
So basically I'm trying to get the Statistics tab to show up, and I'm not sure where to begin.
Just add simple Powershell step into your build configuration. Something like this:
function TeamCity-SetBuildStatistic([string]$key, [string]$value) {
Write-Output "##teamcity[buildStatisticValue key='$key' value='$value']"
}
$outputFile = 'MetricsResults.xml'
$xml = [xml] (Get-Content $outputFile)
$metrics = $xml.CodeMetricsReport.Targets.Target[0].Modules.Module.Metrics
$metrics.Metric
| foreach { TeamCity-SetBuildStatistic "$($_.Name)" $_.Value.Replace(',', '') }
It uses XML output from FxCop Metrics. You have to update the script for your actual schema.