I have a problem with writing custom sink with Python SDK.
To debug it, I used logging.info or print method, but it could not work well.
Are there any good way to debug custom sink?
What problems did you have with logging?
When run with the Dataflow pipeline runner, the logs should be accessible in the Cloud Console as described in Logging Pipeline Messages.
You can also use the direct runner to execute the pipeline locally, which may make it easier to debug.
Related
Our requirement for API testing is:
To deploy test-automation module (Karate feature files, custom java classes) into AWS ECS-Fargate cluster.
Trigger the tests via Jenkins pipeline after every build of the actual microservice.
In addition to above, test-automation module should be triggered to run test suite on-demand and/or at scheduled intervals (say nightly) and send reports.
I have gone through Karate Distributed Testing and stand-alone executable jar options, but doesn't seem suitable for my case. Is Distributed Testing supported only for "Web-UI" automation testing?
Any thoughts would be helpful.
For this use-case, just use a Maven + JUnit project and then there is no difference between Karate and any other Jenkins Java pipeline.
It should be Jenkin's responsibility to do a scheduled build. It is up to you how to get all this into Fargate, maybe building a Docker container is part of the answer, but I would recommend trying to keep it simple.
Here is some Docker related discussion that may help: https://github.com/intuit/karate/issues/396
Open a new question with specifics next time.
I have an automated Jenkins workflow that runs and tests a java project. I need to get all the data and results that are outputted by Jenkins into Report Portal (RP).
Initially, I was under the impression that you have to install the ReportPortal.io Jenkins plugin to be able to configure Jenkins to communicate with RP.
However, it appears that the plugin will eventually be deprecated.
According to one of the RP devs, there are APIs that can be used, but investigating them on our RP server does not give very clear instructions on what every API does or if it is what is required to get test data from Jenkins to RP.
How then do I get Jenkins to send all generated data to RP?
I am very familiar with Jenkins, but I am extremely new to Report Portal.
ReportPortal is intended for test execution results collection, not for jenkins logs gathering.
In two words, you need to find reporting agent at their github organization which depends on your testing framework (e.g. junit, testng, jbehave) and integrate it into your project.
Here is example for TestNG framework:
https://github.com/reportportal/example-java-TestNG/
Developed few scripts using GATLING tool
Able to execute those scripts as a standalone basis
Execution done through GATLING provided .bat file
Triggering of the batch file will ask for user inputs during run time to
select which scenario to simulate
Trying to integrate these scripts and trigger via JENKINS -
As parameterizing the argument does not support the GATLING provided bat
file
Configuring the GATLING provided bat file results in failure of build -
as it requires a user input during run time
Please anyone can provide a step wise approach to integrate GATLING scripts in JENKINS
Gatling support passing parameter : http://gatling.io/docs/current/cookbook/passing_parameters/
Gatling seems a Java tool and execute script by a .bat/.sh file, so that you can inspect what command and parameters it used in backgounnd sence with some tool's help. like 'processhacker' is a enhancement of Windows Task Management. with it you can see the whole command to help you find out how to pass scenario you want to run in command line. If 'processhacker' can't help that, you need to read the souce code of Gatling or send support mail to its company to ask help.
Is it possible to register logback appenders in Dataflow?
With Beam, I'm able to define an appender for the DirectRunner, but when I deploy to Dataflow, it no longer seems to work. Is this just my logback.xml getting lost or is it because the runner has its own separate root logger?
The Dataflow runner is in charge of orchestrating and paralelizing your pipeline to run in a distributed environment. As part of that it manages logging by using SLF4J.
If you can get logback to work on top of one of the supported libraries, or simply use one of them directly, you should be able to get your job's log messages on cloud logging.
The doc is for the Dataflow SDK, but it should apply equally for beam: https://cloud.google.com/dataflow/pipelines/logging
Recently, in our company, we decided to use Ansible for deployment and continuous integration. But when I started using Ansible I didn't find modules for building Java projects with Maven, or modules for running JUnit tests, or JMeter tests.
So, I'm in a doubtful state: it may be I'm using Ansible in a wrong way.
When I looked at Jenkins, it can do things like build, run tests, deploy. The missing thing in Hudson is creating/deleting an instance in cloud environments like AWS.
So, in general, for what purposes do we need to use Ansible/Jenkins? For CI do I need to use a combination of Ansible and Jenkins?
Please throw some light on correct usage of Ansible.
First, Jenkins and Hudson are basically the same project. I'll refer to it as Jenkins below. See How to choose between Hudson and Jenkins?, Hudson vs Jenkins in 2012, and What is the most notable difference between Jenkins and Hudson from a user perpective? for more.
Second, Ansible isn't meant to be a continuous integration engine. It (generally) doesn't poll git repos and run builds that fail in a sane way.
When can I simply use Jenkins?
If your machine environment and deployment process is very straightforward (such as Heroku or iron that is configured outside of your team), Jenkins may be enough. You can write a custom script that does a deploy as the final build step (or a chained step).
When can I simply use Ansible?
If you only need to "deploy" without needing to build/test, Ansible might be enough. For instance, you can run a deploy from the commandline or using Ansible Tower. This is great for small projects, static sites, etc.
How do they work together?
A good combination is to use Jenkins to build, test, and save artifacts. Add a step to call Ansible or Ansible Tower to handle the actual deployment process. That allows Ansible to handle machine configuration and lets Jenkins handle the CI process.
What are the alternatives to Jenkins?
I strongly recommend Thoughtworks Go (not to be confused with Go the language) instead of Jenkins. Others include CruiseControl, TravisCI, and Integrity.
Ansible is just a "glorified SSH loop".
CI is not only the software running, but the whole process of how success and failure is handled, who gets notification, and how the change is merged into the target version control.
If we only focus on the software, CI is a reactive scheduler triggered by code changes, and triggering typical build-validate-release-deploy sequence of "steps".
So in respect of software, Ansible without additional "sugaring" is just a toolkit to run things, which can be those very steps, but it is not CI.
The Ansible (without tower) totally lacks this reactive nature.
If you want to marry Ansible with CI, you can.
Ansible tower is a very Ansible oriented scheduler, but if you need CI software, I think you not necessarily need it. Any CI app capable of running shell script would be capable to launch Ansible playbooks.
Yet unlike Ansible tower - CI tools know to display test reports of all test frameworks, trigger notifications, etc.
Ansible tower can make sense in a complex environment with lots of groups touching Ansible code... The truth is I haven't seen a single real reason to pay for it. But if a manager liked the web interface nothing can stand "but others use it" logic.
I suspect the concept of Ansible tower was in response to puppet enterprise.
:)