This is sort of an open-ended question/request (hope that's allowed).
On my team we are using Karate API testing for our project, which we love. The tests are easy to write and fairly understandable to people without coding backgrounds. The biggest problem we're facing is that these API tests have some inherent degree of flakiness (since the code we're testing makes calls to other systems). When running the tests locally on my machine, it's easy to see where the test failed. However, we're also using a Jenkins pipeline, and when the tests fail in Jenkins it's difficult to see why/how they failed. By default we get a message like this:
com.company.api.OurKarateTests > [crossdock] Find Crossdock Location.[1:7] LPN is invalid FAILED
com.intuit.karate.exception.KarateException
Basically all this tells us is the file name and starting line of the scenario that failed. We do have our pipeline set up so that we can pass in a debug flag and get more information. There are two problems with this however; one is that you have to remember to put in this flag in every commit you want to see info on; the other is that we go from having not enough information to too much (reading through a 24MB file of the whole build).
What I'm looking for is suggestions on how to improve this process, preferably without making changes to the Jenkins pipeline (another team manages this, and it will likely take a long time). Though if changing the pipeline is the only way to do this, I'd like to know that. I'm willing to "think outside the box" and entertain unorthodox solutions (like, posting to a slack integration).
We're currently on Karate version 0.9.3, but I will probably plan to upgrade to 0.9.5 as part of this effort. I've read a bit about the changes. Would the "ExecutionHook" thing be a good way to do this? I will be experimenting with this on my own a bit.
Have other teams/devs faced this issue? What were your solutions? Again we really love Karate, just struggling with the integration of it to Jenkins.
Aren't you using the Cucumber Reporting library as described here: https://github.com/intuit/karate/tree/master/karate-demo#example-report
If you do - you will get an HTML report with all traffic (and anything you print) in-line with the test-steps, and of-course error traces, and most teams find this sufficient for build troubleshooting, there is no need to dig through logs.
Do try upgrade as well, because we keep trying to improve the usefulness of the logs, and you may see improvements if you had failed in a JS block or karate-config.js.
Else, yes the ExecutionHook would be a good thing to explore - but I would be really surprised if the HTML report does not give you what you need.
Related
I am curious if it is possible to automatically measure whether a test suite is flaky from the Circle CI interface. I would measure flaky as - fail and pass with a re-trigger. Is this possible to easily do?
Not at the moment, as far as I'm aware. I've done an extensive research on build insights in general, which included flaky tests analysis and monitoring, and finally decided to build my own tool. The good news is that last I checked, they seem to be focusing on creating better insights tools in addition to what they currently have. They'll tell you all about it if you reach out to them.
In the interim, you have a few options:
Ask them how far away are they from supporting for your idea of what a flaky is (I'm hoping this point gets outdated shortly as they work on it)
Consume their data through their decent enough API, and build your own tool in the interim and crunch the numbers yourself (this is what I ended up doing and it isn't too bad)
For example: generally speaking, a flaky for my team is a test that failed more than a few times over a large timespan. Their API gives you whether a build failed or not, which test failed, when and how. This gave me enough to work with and figure out whether I consider that spec failure as flaky or not. I'd assume yours is sort of similar, with maybe the only difference being whether it was re-triggered (unsure if they provide that info specifically, but you could refer to the workflow, commit and build ID to figure that out; e.g. if the build ID of a new run is the same).
With that beind said, the "how easy is it?" part of your question is something I can't really say for certain. It was a relatively easy learning curve to go through their APIs, get familiar with it, run a couple of requests, look at the data, massage it, store in the DB, then build a web interface around it. But I'm not sure how much familiarity and experience the people building the tool on your end have.
I am trying to track the page speed of certain urls of my project on each merging of the pull requests in Github and output the results of report in HTML format or JSON file. On the CI side, I am going to use Jenkins. I have no prior knowledge on performance testing. I want to know about the best approach to automate the speed test, integrate it with Jenkins and output the result.
On researching over the internet, I noted few possibilities which could be done to achieve this goal.
Installing "Page Speed Insights (psi) node package", creating the script that uses the psi for fetching the speed of certain pages, generating the test reports for use with Jenkins. (Referred to this link by Oxagile)
Performance testing using Jmeter and integrating with Jenkins.
Performance analysis using LightHouse. (Referred to this link by Timo Stollenwerk)
Choosing the right approach is very important. Therefore, I would be very grateful if anyone can suggest me different approaches and thus the right one to use(with examples if possible)in my case to achieve this goal.
Thank you in advance.
After quite a bit of research, I found out that sitespeed.io is the best solution for achieving this goal. It is a complete web performance tool that helps us to measure the performance of the website. It is best for running in the continuous integration to find web performance regressions on commits and monitoring them in production and alerting on regressions.
Is Continuous Integration feasible for iOS.
Again my Jenkins build fails with “Your session has expired. Please log in.” and I begin to ask myself if Continuous Integration is at all feasible for iOS development.
How can you set up a reliable unattended build system that demands a user logged into some web service which might expire your login at any moment? As far as I see it the answer is: You can't.
It's just not possible. It will always be an unreliable system which will stop building for no apparent reason.
Or am I missing something?
As an answer to the discussion in the comments, it is feasible. There is no simple way to explain everything that you need but in a short summary.
If you using Jenkins,
You need access to MacOS, Jenkins running on MacOs or Jenkins Slave which is with MacOS.
Xcode on the machine from point 1. with xcode(xcode comand tool) xcbuild.
Fastlane integrations (check here)
The last thing is fine-tuning with your process, haw and when to increase builds and etc.
At some providers like Azure DevOps, it was easy to add all these BIG steps with build in components of the system.
In general, I never found a good tutorial about this how to be done end to end. But with a lot of reading and trying it was done from me, without to have big experience in the dev-ops world.
P.S. It will be nice if you do a good tutorial for how this can be done when you made it(I didn't have time to do it :( ).
I've been learning Travis CI and I want to use it to help automate tests on a MEAN application, then deploy it. However, there are some ways to go about this.
After reading, I learned I can create two separate repositories, thus maintaining two separate applications: a client application and a backend application. Since they are separate repositories, I can have separate .travis.yml files on each and perform continuous integration on the client application and backend application. However, I need advice on this approach because I have questions:
For the client app, I have to write tests. Since I'll be using angular, I want to test responsiveness and if components are working as intended. The client application also has to communicate with the backend application and I want to see if it is properly getting the correct results (such as clicking a button triggers a GET request and see if I'm getting the correct response body). Since the client app is on a separate repository, and when I build it on TravisCI, how will I connect the client application to the backend application if it exists on a separate repository?
I read around and I can use submodules in git. Thus, the client application and the backend application can be submodules for a 'master repository'. Therefore, how will the trigger in TravisCI work? Will I have separate travis.yml files in each submodule, or will I have to have one in the "master repository"?
If I were to get everything to work properly and have the client application and backend application both successfully deploy and the two are hosted on different servers, how will I fix the cross-domain issue?
The other approach is to host the static files produced by ng build --prod and have the node backend application host them. When Travis CI is triggered, I can first build the node backend application and run the tests on it first and then run the tests on the angular client application. After all of the tests are passed, where do I deploy? I know I have to deploy the node application since it will host the static files, so I how exactly will I deploy the backend application in Travis CI?
I know this is going to push it, but I'll ask it anyway. In the future, I want to learn how to implement microservices, and I want to use Nginx for the purpose of load balancing. How will I go about that? Docker can help me create a production environment where I can see if the Nginx server and node application are working well, but how do I include that in Travis CI?
If my question is a bit vague, please let me know what parts of it are vague so I can edit it that way I can make more sense of what I'm asking for. Thank you, and I look forward to an answer :)
Question is ultra-broad. You should solve one problem at a time, because by the time you solve 1 and 2 I doubt that 3 will be your only concern, and all of these issues are not really related.
try spending a bit of time reading Travis CI documentation, but also how to write tests and what different types of tests will do for you. Your issue is less about Travis than about what are unit tests vs. what are integration tests. So write simple standalone tests for your frontend, simple standalone tests for your backend, maybe run integration tests manually for a while, and when it becomes a real issue, then you will have better knowledge of how everything works together and you will find a way. Long story short: there is no single best way to run integration tests and it mostly depends on many, many things in your app (how does it run, what type of DB do you use, etc.)
you should read about submodules. Perhaps you need them, perhaps not. There is no way to tell. You can use submodules with Travis CI, but you can also not use submodules. Depends on what you want to achieve. Focus on the end goal for your architecture, not on what Travis CI needs!
what cross-domain issue? Again, this is a very different problem, and probably not the most prominent one you will face. Since I have no idea what server technology you will use there is no way I can answer that question properly. If you use express, this may be what you are looking for: https://expressjs.com/en/resources/middleware/cors.html
General bit of advice: all of your questions boil down to experience. Try solving one problem at a time, get started with your project and when you hit a specific issue, it's much much easier to solve than asking about "microservices". There are many ways to do microservices right, each solving a different issue. Without any knowledge of what your application is about and what issues you want to solve, microservices may or may not be what you are looking for, but there are also many other components that can affect your stack. Just get started and don't think about all of this too much for now - it's better to have something soon that you can test and learn upon, than think for weeks about something that you will never get because it is only theory.
At my workplace, I've been tasked to look into some metrics that the Jenkins tool provides and somehow pull them programatically and display them in some presentable format. The metrics that I need to pull are:
How many unit tests are passing? Failing? Skipping? The total % of passing?
How many integration tests are passing? Failing? Skipping? The total % of passing?
How many acceptance tests are passing? Failing? Skipping? The total % of passing?
How long does it take to execute the test? Make the build?
What is the number of tests executing in pipelines?
... the list goes on
Now I have a very small 1000 ft understanding of Jenkins, and an even smaller understanding of the steps that I need to take to make this program come to life. I am an intern with not much programming experience either, but after some research, I learned that I can navigate through the Jenkins API by adding '.../api' to the link that I want to find API elements for, and I know that I'm going to need to develop a plugin. Aside from that I don't have much direction at all. I don't know what environment I need to develop these plugins (Maven? Never heard of it)... I don't know what languages are supported (I only know C++, Java, and JS)... I don't know how to even install a plugin or get to the plugin on the Jenkins site. I feel like I'm drinking from a firehose with this task and need some guidance.
Does anyone have good guides, advice, tips, tricks, videos... anything that might help me get started on Jenkins plugin development? Any insight into how I might solve this problem too would be much appreciated.
Thanks so much.
First of all there are tools out there which generates HTML reports. You can start there.
For example: MSTest report (.trx) can be converted to HTML by TRXER
and can be published using the HTML Publisher Plugin
However if you're into building your own plugin use NetBeans (I have tried it; and it works)
But creating Jenkins graphs you have to google and see.