I am using MSBuild for building our test project from command prompt which inturn uses nunit and gives the results. I want to schedule it nightly so that it automatically runs the msbuild and get the results. Is there any way I can do that?
I would recommend using a Continuous integration server for this. You can set up your continuous integration server to schedule builds and a number of other tasks. Personally, I use Team City from Jetbrains, but there are other options available, such as Cruise Control.net and Hudson.
What exactly do you want to do with the results? If you just need to run your build script, a simple scheduled task should do it.
The way my company solved the problem was to use Powershell. We have a scheduled task that executes a powershell script that completely automates the process using MSBuild and devenv.exe.
We've been able to do away with the expense of many other continuous integration tools with this process. We have even connected it to our svn server so that it can checkout the latest revision, build it, send out an email if there was a build error and send the output of the screen as a result log.
I realize this might not be simple answer you're looking for, but Powershell was really easy for me to learn, exceedingly customizable, and very powerful. I had doubts when my Director said we were moving this way, but it's been a delight.
Related
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
We are using TeamCity 9.x as are main CI server. I'm looking for ways to run a script (PowerShell, Python, ...) when a build is tagged. Is this possible?
The only thing I can think of is to write a simple service which polls the REST API for the last x builds and reads the <tag/> information.
We are using TFS for source control, so labeling the sources is not an option (because a label is unique in TFS).
Are there any other (simpler) ways to do this? Or is there any other way to define build quality and execute something?
Yes you can
In the build trigger definition, you can mention specific subset of tags(using regex patterns) on which teamcity targets are trigeered. In your case, you have to set the triggers to run on tags only
I know this doesn't answer your question but figured I'd mention it anyways since you have TFS in your environment already.
If you were using TFS Build it has a drop down on each build to indicate quality. And there's a free tool called TFS Deployer that allows you to run scripts when the quality is changed.
I want to setup a continous integration system that upon a commit or similar trigger should:
run tests on a fortran/C/C++ code, if needed.
compile that code using cmake.
run tests on a rails app.
compile the rails ap.
restart the server.
I'm looking at Jenkins. Is it the best choice for this kind of work? Also, what's the difference between using a bash script that makes all that (if possible) and using jenkins? I'm asking not because I'm thinking about using a script, but to better understand jenkins.
It sounds like Jenkins would certainly be a reasonable choice for this. Apart from the ability to run arbitrary scripts as build steps, there's also a large number of plugins, which provide better integration with cmake for example.
Even if you're using a single bash script to do all of this, using Jenkins on top of it would still have a number of advantages. You get a web interface, email notifications and build history for free, with all that this entails. By integrating your tests "properly" with Jenkins, you can also get things like graphs that show how many tests succeeded/failed over time.
I am using Jenkins for java projects and have to say it is easy to configure. I used to add lots of plugins for better configuration of build steps, but tend to go back to using scripting languages for build and deploy steps because of two main reasons. If I have a build script, it's easier to configure the same job on a different Jenkins server or run the script manually if need be and the build configuration is not so cluttered (I still have one maven job with more than 50 post build steps). The second reason is, that it is easier to version the scripts in SVN, compared to having the build config in SVN.
So to answer your questions. I don't know if it is the 'best' tool, but it is good enough for me. Regarding scripting: use each tool for what it is build for. Jenkins a glorified cron deamon with great options when it comes to displaying analysis. The learning curve for people to use it is minimal (i.e. starting a job, seeing whether it failed.) Configuring Jenkins needs a little bit more learning, but it's very easy to set up simple jobs and go then to the more complicated tasks.
For the first four activities Jenkins will do the job and is rather the best choice nowadays, but for things like restarting the server (which is actually "remote execution"), better have a look at:
http://saltstack.com/
or:
https://wiki.opscode.com/display/chef/Home
http://cfengine.com/
http://puppetlabs.com/
http://cfengine.com/
Libraries like Fabric(Python) or Capistrano(Ruby) might be useful too.
Is it possible "and how if you know", to schedule the execution of an ant task?
For example, i want my build.xml to be executed every 5 hours or every day at a certain time?
I have been looking around but no solution found
Thank you
For very simple requirements, I'd echo the use of cron.
If the reason for running ANT is to periodically perform a master build of your project's code, then you're effectively following a practice called "Continuous Integration". In that case I'd highly recommend running a continuous integration server, such as Jenkins.
Jenkins is a very useful piece of software, easy to install and can become your automation framework for more than just building your code.
Just use cron and call ant.sh from it.
If you are working on a windows environment, you can use the "task scheduler" and set the interval. If you are working in unix/linux, you may use cron for scheduling your job.
What kind of agile tools are you using for Erlang development? What continuous integration (CI) server are you using to build Erlang code? The only reference I got was from Quora question How do I integrate Erlang unit tests in Jenkins (Hudson)?.
I am also interested in the nifty details of setting them up and making talk to each other.
As a company using Erlang actively, Klarna (www.klarna.com) use Jenkins (formerly Hudson) for daily regression test on nearly every dev commit. It's an org with about 80 people total in rnd and we use distribute mode of Jenkins which allows us to have more than 10 build slaves mastered by only one Jenkins server. Basically we have a code base with Eralng code which is version controlled by tools like svn or git. All these testcases are under common test framework and all works well under Jenkins.
Previously, we tried Cruise Control and gave it up since Jenkins does much better.
As Lukas mentioned, you probably will need a tool to gen xml files sine common test doesn't export them directly. Haven't really tried that module though, we do have an implementation of common test event handler to do the job, but it was abandoned due to performance, we do have a a critical requirement on test time. right now, we use a own made script to export xml from common test log directly.
There are a lot more you could do with Erlang and Jenkins, like code coverage analyze if you compile properly and export formatted xml to Cobertour plugin, gui test with selenium etc.
For setting up Jenkins, I think Jenkins home page has a good introduction.
Regarding agile tools, I guess it's really hard to define what a agile tool. Also what I believe is it's very much depend on the size of you org. You will probably need a good process view tool (team level or depart level), a good ticket tracking tool, code review tool, communication tool. There are bunch of them implemented under open source. According to our exp, none of them seems to be able to work seamlessly with Jenkins which means you will need to select and tweak by your own requirement. BUT that's the beauty of open source isn't it :)?
If you want to do it using Jenkins, I have written a common test hook which generates JUnit XML output for your tests which Jenkins can use to produce test statistics.
https://github.com/garazdawi/cth_tools/blob/master/src/cth_junit.erl
We use Jenkins for our Python code, so I think you may use Jenkins with Erlang code.
We use buildbot with our own recipes to hook unit tests.