I am trying to run tests in Jenkins using the Specflow Framework and the Specflow testrunner.
Since I am actually just migrating to another server, I was able to copy most configurations and only had to set up a newer Java version.
Jenkins is able to build the project but stops and skips all test with the following error:
Execution framework error: TechTalk.SpecRun.Framework.SpecRunException: At least one test thread aborted. ---> System.DllNotFoundException: Unable to load DLL 'nameofdll.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
I am pretty sure it is actually a specflow problem, because when this error occurs locally in my VS solution, I just delete the specflow-stepmap-cache file and it works fine. Since Jenkins executes the tests there is no such file generated or I just have not found it yet.
I work with Visual Studio 2017, Specflow Runner 2.3.0, SDK 4.6. and Jenkins 2.190.2.
I know those versions are fairly old, but it worked fine on the old server and I believe there is just one thing missing.
Anyone got an idea?
Thanks in advance.
Related
I have my build definition for my project set up on TFS. I intend to add Task to Run C++ Unit Tests and run them after each build.
I tried and added the task. When the Tests ran on TFSBuild the first test crashed with unhandled c++ exception and did not moved to next test. It hung there itself.
So I tried to run the test manually using command line(cmd). I ran the vstest.console.exe command and executed all tests and they ran fine when ran from cmd on the Build Agent. I also tried running them on Visual Studio and they ran fine on the Build Agent.
I am not sure what is going wrong with the TFS.
I have found the solution but not the root cause of the issue.
The issue is solved by using an interactive tfs agent.
I've been scouring various Nunit articles for the past several days trying to figure out the issue with why my Nunit tests are not running.
I am using TFS 2015 Build vNext engine and the Visual Studio Test step to execute my Nunit test cases. The VS 2015 solution used for generating the Nunit test dlls contains the several .net projects and 2 Nunit test projects. Nunit3.0 and Nunit3.0 Test Nuget packages are being used to create and execute the tests. Here is the nuget configuration and nunit.framework reference in my Visual Studio 2015 solution. The test cases run fine in the Visual Studio IDE and compile without error when the solution is build in vNext.
When the VS Test step executes in Build vNext, I get the following warning:
Warning: Dependent Assembly nunit.framework of E:\TFSBuild_Directory\14\s\EBS\B_WebEnable\ABC.Data\ABC.Data.Core.Nunit.Tests\obj\UnitTesting\ABC.Data.Core.Nunit.Tests.dll not found. Can be ignored if not a NUnit project.
The VS Test step is configured:
VS Test Step Configuration
I have confirmed on the build server that the nuget packages are getting restored to the build server and the Nunit.framework.dll appear to be in the expected file path as compared to my local path.
Any ideas as to what could be causing this issue? Any input is appreciated!
I have restored Nunit and Nunit3TestAdapter from Nuget to have a test on my side, but couldn't reproduce your issue. I'd like to share my steps here for your reference:
Restore Nunit and Nunit3TestAdapter from Nuget in VS for the test project (as your first screenshot), and check in the packages.config file.
In "Visual Studio Test" task, specify $(Build.SourcesDirectory)\packages in "Path to Custom Test Adaptors". As if there is a packages folder in the sources directory, it is automatically searched for testadapters (It seems you have a long path, you may need to check it).
I use the default VS template:
No error or warning occurred during this process. You can compare your steps with mine and correct yours. Hope this helps you.
Ok - after stepping away from the issue and coming back to it with fresh eyes, I found the issue causing the nunit.framework.dll issue.
After going step by step through the build definition, I forgot that I had an MSBuild parameter in the build step that was copying the compiled output to another location /p:OutDir=$(build.stagingDirectory)\ABC. Once I changed the test step to point to the new location, everything worked fine. Thanks everyone for the feedback!
I am having an issue with building a project, with SonarQube integrated, on TFS 2015.
When I build this project without SonarQube, it builds fine. However, when SonarQube runs alongside this build, one of the project files fails to compile (specifically, it's a unit test project). I get the following error:
D:\Builds\....\Project\.sonarqube\bin\targets\SonarQube.Integration.targets (235, 0)
Could not write lines to file "D:\Builds\.....\.sonarqube\out\\f_AnyCPU_Release_0213\FilesToAnalyze.txt".
The process cannot access the file 'D:\Builds\.....\Project\.sonarqube\out\f_AnyCPU_Release_0213\FilesToAnalyze.txt'`
because it is being used by another process.
[D:\Builds\.....\Project\\src\Apps\Solution\obj\Debug\Fakes\spi\f.csproj]
If anyone can offer some hints as to why this is happening, it'd be much appreciated. I've tried recreating this error with a few projects that I've written but it doesn't seem to happen. It might be specific to the project.
My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.
Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.
If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?
Here is our pattern matching for our SpecFlow test project:
And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests:
UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:
Test method
AccessTheNDCHomePage threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)
After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?
If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.
The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.
In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.
Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.
if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.
So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.
EDIT
It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.
Here were some of the issues and solutions we found to set up SpecFlow on our build server:
The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue
WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.
I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.
The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.
https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive
I am attempting to run UI tests (using Windows Automation API) on a WPF application, on a CI basis. I have associated my solution with a .testsettings file and linked my test .dll to the build process. All seems to be well.
When I queue a new build, in TFS's build log I can see the command line arguments passed to MSTest (2010) on the build machine - it appears that the .testsettings file is loaded correctly and that the correct test .dll is passed to MSTest.exe.
However, it seems to be "stuck" after the Executing tests message - both when running through the build and by running it directly through the build machine. In addition, I have specified a startup script to run before the tests, in the .testsettings file (it copies the application's binaries locally so the test can start the application and run the applicable tests). This script doesn't seem to be executed (the directory to which the files are supposed to be copied is empty).
Has anyone seen a similar problem and knows how to deal with it?
As Mike said in the comments, the issue was actually a bug in VS 2010 that made it impossible to run tests through the 2010 IDE if you have VS 2012 installed side by side. Unfortunately, I was unable to install VS 2010 SP1 on the build machine, but it is also possible to solve the issue by adding the /noisolation argument to the standard command line syntax (this runs the tests through the MSTest process, which somehow solves the issue).
However, since you cannot define additional arguments to be passed to MSTest through TFS's integrated automated testing feature, I wrote my own application that invokes as a scheduled task, runs the tests and sends an email containing an HTML report (I used trx2html for that -- note that if you're using VS2012, you'll need the beta 0.7 version, as there's been a change in the .trx format since VS 2010).