I am running a large amounts of specflow tests that take a while to run. I have these running on a build server - and I can see the output while these are running.
[10:59:13][Step 3/3] Results Top Level Tests
[10:59:13][Step 3/3] ------- ---------------
[10:59:13][Step 3/3] Passed Project.Web.Specflow.Specs.Account.ForgotPasswordsFeature.ExistingUserHasForgottenTheirPassword
[10:59:23][Step 3/3] Passed Project.Web.Specflow.Specs.Account.ForgotPasswordsFeature.Non_ExistingUserHasForgottenTheirPassword
[10:59:32][Step 3/3] Failed Project.Web.Specflow.Specs.Admin.AdminTabFeature.AdminUserLoggedIn
[10:59:45][Step 3/3] Failed
etc.
All I can see is whether a scenario has passed or failed. I would like to see the error message when these tests fail. These error messages are only revealed at the end of the test run. Is there a way to reveal the report during the test run rather at the end of the test run? This way I can start analysing the failures rather than having to wait 30 min.
Thanks!
Related
I would like to use the Robot framework to automate a step where the next command is executed in a cmd:
docker-compose logs --no-color --tail=1 the-server
The output of the previous command will be the log of the server named the-server in the docker-compose.yml file.
After I would like to check if the response of this command has the string Started the-server, to check if the server is up.
I'm using the next robot file:
*** Settings ***
Library Process
Library OperatingSystem
Suite Setup log running on ${properties.hostname}
Suite Teardown Terminate All Processes kill=True
Variables C:/Users/TheUser/Desktop/CheckOutRegression/properties.py
*** Test Cases ***
Check if the-server is up
${result} Wait Until Keyword Succeeds 10x 20s Check The-Server
*** Keyword ***
Check The-Server
${resultOfcmd} Run Process docker-compose logs --no-color --tail\=1 the-server shell=yes cwd=${properties.pathToDocker}
Should Contain ${resultOfcmd} 'Started the-server' PASS
Log To Console ${resultOfcmd}
The test-case is always failing with the error:
Keyword 'Check The-Server' failed after retrying 10 times. The last error was: TypeError: 'ExecutionResult' object is not iterable
also I'm not sure if the variable {resultOfcmd} contains the output of the command docker-compose logs --no-color --tail\=1 the-server.
Can anyone help?
Check Process library documentation about Result object from Run Process keyword:
https://robotframework.org/robotframework/latest/libraries/Process.html#Result%20object
You get your results from ${resultOfcmd.stdout}.
You should also use Log To Console before failing keyword to see result before test fails.
Log To Console ${resultOfcmd.stdout} console=yes
I have a build workflow (xaml) that invokes mstest. My expectation is that there would be a link to the test results available on the build summary page. However, this is not the case.
The build ends with the MSTest command - exits with 1, since there are failures. So, MSTest is the last activity in the workflow.
Something is definitely wrong here. The drop location contains just the workflow logs - no test results. These are found in the TestResults folder, but not in the drop location.
So, my question is - how do I need to wire the MSTest activity so that at the end there would be a link somewhere to the test results?
Now the gory details. This is how the last moments of the workflow look like in the activity log (I replaced some stuff with [...] for the sake of brevity). Notice the following message at the end:
Publishing results of test run tfsbuild#TORSVARCH01 2017-03-02 16:03:43_Any CPU_Release to http://tfsserver:8080/tfs/defaultcollection...
................Publish completed successfully.
But where is it published?
MSTest
Inputs
TestLists:
ToolPath:
TestNames:
MinPriority: -1
TestSettings:
CommandLineArguments:
SearchPathRoot: d:\b\test\3269\..\2282\Binaries
Platform: Any CPU
MaxPriority: -1
Category:
RunTitle:
PathToResultsFilesRoot: d:\b\test\3269\..\2282\TestResults
TestContainers: System.Linq.Enumerable+WhereEnumerableIterator`1[System.String]
TestMetadata:
TestConfigName:
Flavor: Release
TestConfigId: -1
Publish: True
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MSTest.exe /nologo /usestderr /searchpathroot:"d:\b\test\3269\..\2282\Binaries" /resultsfileroot:"d:\b\test\3269\..\2282\TestResults" /testcontainer:"d:\b\test\2282\Binaries\BackgroundJobTests.dll" [...] /publish:"http://tfsserver:8080/tfs/defaultcollection" /noprompt /publishbuild:"vstfs:///Build/Build/265747" /teamproject:"DFDev" /platform:"Any CPU" /flavor:"Release"
Loading d:\b\test\2282\Binaries\BackgroundJobTests.dll...
[...]
Starting execution...
Warning: The disabled test 'GetJobReqModels' was removed from the test run.
Results Top Level Tests
------- ---------------
Passed BackgroundJobTests.AsyncParallelSqlRunnerTest.AsyncExceptionInDoIt
[...]
Passed WFMCommonTest.TimesheetApi.UnitTests.TimeSegmentApiTests.Test_RoundTimeOfOneSegment
1778/5035 test(s) Passed, 3224 Failed, 33 Error
Summary
-------
Passed 1778
Failed 3224
Error 33
------------
Total 5035
Results file: d:\b\test\2282\TestResults\tfsbuild_TORSVARCH01 2017-03-02 16_03_43_Any CPU_Release.trx
Test Settings: Default Test Settings
Test Run Error.
Run has the following issue(s):
Warning: Test Run deployment issue: The assembly or module 'dtSearchNetApi2' directly or indirectly referenced by the test container 'd:\b\test\2282\binaries\ruleenginetests.dll' was not found.
[...]
Warning: Test Run deployment issue: The assembly or module 'System.ServiceModel.Web' directly or indirectly referenced by the test container 'd:\b\test\2282\binaries\utilitytest.dll' was not found.
One of the background threads threw exception:
System.Data.OleDb.OleDbException (0x80004005): Cannot open database "851qa89" requested by the login. The login failed.
[...]
One of the background threads threw exception:
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
[...]
Waiting to publish...
Publishing results of test run tfsbuild#TORSVARCH01 2017-03-02 16:03:43_Any CPU_Release to http://tfsserver:8080/tfs/defaultcollection...
................Publish completed successfully.
Exception Message: MSTest.exe returned an exit code of 1 indicating that not all tests passed. (type TestFailureException) Exception Stack Trace: at System.Activities.Statements.Throw.Execute(CodeActivityContext context) at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager) at System.Activities
EDIT 1
We are using TFS 2015. The build template is a custom one.
The mstest command line parameters can be observed from the aforementioned sample, but here they are again broken one per line for convenience:
/nologo
/usestderr
/searchpathroot:"d:\b\test\3269\..\2282\Binaries"
/resultsfileroot:"d:\b\test\3269\..\2282\TestResults"
/testcontainer:"d:\b\test\2282\Binaries\BackgroundJobTests.dll" ...
/publish:"http://tfsserver:8080/tfs/defaultcollection"
/noprompt
/publishbuild:"vstfs:///Build/Build/265747"
/teamproject:"DFDev"
/platform:"Any CPU"
/flavor:"Release"
The d:\b\test\2282\TestResults folder indeed contains the trx file and an auxiliary directory. But that is not good enough - I want to seem them online. And mstest claims they are published:
Publishing results of test run tfsbuild#TORSVARCH01 2017-03-02 16:03:43_Any CPU_Release to http://tfsserver:8080/tfs/defaultcollection...
................Publish completed successfully.
But what is the address?
EDIT 2
I checked the online summary page of the build and the test results link is there, but it is not found on the summary page that one can view in the Visual Studio.
Please, observe the page I can see in the Visual Studio 2015:
Now the same page when accessed through a browser. Notice the hyperlink to the test on the right side of the page:
So, I suppose my question can be narrowed down to how can one see this link on the build summary page in Visual Studio?
EDIT 3
Here is my almost entire build workflow - http://pastebin.com/9z2UE79E
In order to save some space I removed two pieces:
An InvokeProcess activity to upgrade a database
An InvokeProcess activity to modify a certain config file.
I do not understand what do I have to do to include the Test Summary on the overall Summary page.
It seems you use the /publish option in MSTest activity. Use the /publish option to publish test data to the database for the team project collection associated with the project's Team Foundation Server.
Check: https://msdn.microsoft.com/en-us/library/ms243151(v=vs.120).aspx
Machine:Mac
UnityVer:4.6.2
error of the following will appear when you have made a jenkins build of unity.
----- Total AssetImport time: 6.818127s, AssetImport time: 6.808909s, Asset hashing: 0.005416s [60.9 KB, 10.972655 mb/s]
Platform assembly: /Applications/Unity4.6.2/Unity.app/Contents/Frameworks/Mono/lib/mono/2.0/System.Data.dll (this message is harmless)
System memory in use before: 71.7 MB.
Unloading 137 Unused Serialized files (Serialized files now loaded: 0 / Dirty serialized files: 0)
System memory in use after: 66.8 MB.
Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6940.
Total: 18.863014 ms (FindLiveObjects: 0.248059 ms CreateObjectMapping: 0.096515 ms MarkObjects: 1.745080 ms DeleteObjects: 0.012070 ms)
Scripts have compiler errors.
(Filename: /Users/builduser/buildslave/unity/build/Runtime/Utilities/Argv.cpp Line: 127)
Aborting batchmode due to failure:
Scripts have compiler errors.
Thread 'UnityLookForNewInputDevices' is still running!
(Filename: /Users/builduser/buildslave/unity/build/Runtime/Threads/Thread.cpp Line: 68)
Thread was not cleaned up!
(Filename: /Users/builduser/buildslave/unity/build/Runtime/Threads/Posix/PlatformThread.cpp Line: 45)
FATAL: Unity3d command line execution failed with status 1
Build step 'Invoke Unity3d Editor' marked build as failure
FTP: Current build result is [FAILURE], not going to run.
Finished: FAILURE
Try search "error" in Jenkins console output. It's very easy to overlook what actually causing build fail. The error show in the bottom of console output is the result. The useful information is where all warnings and errors are shown like you normally see in IDE like visual studio. I spend a whole afternoon found out I should always use Control + F instead of my eyes to search through things like Jenkins console output.
The output shows you the problem: you have script errors. Until you fix your script errors, you won't be able to build.
You haven't included the script errors, so no-one can give you any more info. When you fix them, you'll probably find that it's nothing to do with Jenkins, and the question title is unrelated.
Usual causes of this:
You have some editor scripts in you project
You forgot to put #if guards around editor scripts and/or external scripts
You forgot to copy/paste one of your 3rd party libraries or plugins onto the machine where you're doign the build - and it's not in source control already (where it should be).
I am running Kubuntu 12.04.4 LTS and ROS Hydro.
When trying to complete the MoveIt! Setup Assistant Tutorial here
http://docs.ros.org/hydro/api/moveit_setup_assistant/html/doc/tutorial.html
everything is working fine until I try Step 4. When I click the "Add Group" button, the MoveIt! Setup Assistant GUI closes and I get the following error:
/opt/ros/hydro/lib/moveit_setup_assistant/moveit_setup_assistant: symbol lookup error: /opt/ros/hydro/lib/libmoveit_setup_assistant_widgets.so: undefined symbol: _ZN3ros7package10getPluginsERKSsS2_RSt6vectorISsSaISsEEb
================================================================================REQUIRED process [moveit_setup_assistant-2] has died!
process has died [pid 14775, exit code 127, cmd /opt/ros/hydro/lib/moveit_setup_assistant/moveit_setup_assistant __name:=moveit_setup_assistant __log:=/home/scratch/.ros/log/a83de36e-06f4-11e4-bcc4-001fd081bf4b/moveit_setup_assistant-2.log].
log file: /home/scratch/.ros/log/a83de36e-06f4-11e4-bcc4-001fd081bf4b/moveit_setup_assistant-2*.log
Initiating shutdown!
================================================================================
[moveit_setup_assistant-2] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
Does anyone know what's going on? Thanks
We're trying to set up continuous integration test on our iOS project and running into some issue running broken from Jenkins.
The error we are getting is the following:
iphone Add measurment from CNX.js
/Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/script.rb:55:in `run': Test Script Failed (Bwoken::ScriptFailedError)
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open3.rb:86:in `popen3'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/script.rb:53:in `run'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/fileutils.rb:1427:in `to_proc'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/simulator_runner.rb:21:in `each'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/simulator_runner.rb:21:in `execute'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/script_runner.rb:33:in `execute_for_family'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/script_runner.rb:28:in `execute_in_simulator'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/script_runner.rb:27:in `each'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/script_runner.rb:27:in `execute_in_simulator'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/script_runner.rb:20:in `execute'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/cli/test.rb:105:in `test'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/cli/test.rb:71:in `run'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/lib/bwoken/cli.rb:56
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /Library/Ruby/Gems/1.8/gems/bwoken-2.0.0.beta.2/bin/bwoken:3
from /usr/bin/bwoken:19:in `load'
from /usr/bin/bwoken:19
Recording cancelled : At least one target failed to launch; aborting run
Error Domain=com.apple.instruments Code=1 "Error Starting Recording" UserInfo=0x7fa5632b19f0 {NSLocalizedDescription=Error Starting Recording, NSLocalizedRecoverySuggestion=At least one target failed to launch; aborting run}[m
Failed to start trace.[m
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I think the important part is here:
Recording cancelled : At least one target failed to launch; aborting run
mError Domain=com.apple.instruments Code=1 "Error Starting Recording" UserInfo=0x7fa5632b19f0 {NSLocalizedDescription=Error Starting Recording, NSLocalizedRecoverySuggestion=At least one target failed to launch; aborting run}
I can't make sense of it right now, so any help is appreciated.
Thanks, olivier
WE got it running using the jenkins app (https://github.com/stisti/jenkins-app). The issue is related to the fact that Jenkins runs as a deamon and thus does not have access to the window server.
There might be another way around this (changing the Jenkins user type), but running jenkins as an app works for us