The command I am running:
python3 -m apache_beam.examples.wordcount --region us-central1 --input gs://dataflow-samples/shakespeare/kinglear.txt --output gs://STORAGE_BUCKET/results/output --runner DataflowRunner --project traffic-prediction-317805 --temp_location gs://STORAGE_BUCKET/temp/
Getting the following error when running a Beam pipeline in Dataflow as a job:
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: Error processing pipeline.
Checked Dataflow Job Log and seeing the time and error message:
2022-10-26 01:26:40.307 BST Error processing pipeline.
Related
I want to rerun, below shell script in the same project, once my build is i completed with errors, so I can rerun my failed test cases in testng-failed.xml.
FILE=./target/surefire-reports/testng-failed.xml
if test -f "$FILE"; then
echo "$FILE exists."
mvn clean test -Dhttp.proxyHost=proxy-dev.aws.skynet.com -Dhttp.proxyPort=8080 -Dproxy-dev.aws.skynet.com -Dhttps.proxyPort=8080 -Dbrowser="${Browser}" -Dbrowser_version="${browser_version}" -Dos_version="10" -Dos="Windows" -Dlocalrun="false" -Dtestxml=FILE
else
echo "$FILE is not there"
fi
But in Jenkins, post build section, I don't see an option to add Execute shell. Please help.
I installed post build plugin , but it's only show for maven projects and not for my free style project.
In free style project, No post steps option.
I managed a walkaround to do this without any plugins and still run in the Jenkins free style project. Add two execute shells and in first shell enter the below shell commands.
export M2_HOME=/var/jenkins_home/tools/hudson.tasks.Maven_MavenInstallation/Maven3
export PATH=$PATH:$M2_HOME/bin
mvn --version
cd CommonPagesStorage
mvn clean install -DskipTests
cd ..
cd MultiWebdriverDemo
mvn clean test -Dmaven.test.failure.ignore=true -Dsurefire.suiteXmlFiles=TestSuites/${TestPlanName}
echo "${TestPlanName}is ran"
#Dmaven.test.failure.ignore=true added this so even the script failed it doesn't mark as fail and mark it as unstable and continue.
In the second shell command ,conditionally checks the testng-failed.xml is exists if so run it to run the failed test cases.
cd MultiWebdriverDemo
export M2_HOME=/var/jenkins_home/tools/hudson.tasks.Maven_MavenInstallation/Maven3
export PATH=$PATH:$M2_HOME/bin
mvn --version
#this to print current directory.
echo "the second Script executed from: ${PWD}"
if [ -e ./target/surefire-reports/testng-failed.xml ]
then
echo "ok file found pass"
mvn test -Dmaven.test.failure.ignore=true -Dsurefire.suiteXmlFiles=target/surefire-reports/testng-failed.xml
else
echo "file not found passed in first attempt ..."
exit 0
#exit 0 means forcefully mark it as passed
fi
if [ -e ./target/surefire-reports/testng-failed.xml ]
then
echo "Rerun also failed exiting with buil state as failure "
exit 1
#exit 1 means forcefully mark it as failed
else
echo "file not found rerun is passed marking the build as passed ..."
exit 0
fi
S0 even in the second time, my webdriver test case is failed, it's failed build mark as a failure.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16.199 s
[INFO] Finished at: 2021-08-12T14:05:36Z
[INFO] ------------------------------------------------------------------------
+ [ -e ./target/surefire-reports/testng-failed.xml ]
+ echo Rerun also failed exiting with buil state as failure
Rerun also failed exiting with buil state as failure
+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
TestNG Reports Processing: START
Looking for TestNG results report in workspace using pattern: **/testng-results.xml
Saving reports...
Processing '/var/jenkins_home/jobs/August-FreeStyle/builds/37/testng/testng-results.xml'
100.000000% of tests failed, which exceeded threshold of 0%. Marking build as UNSTABLE
TestNG Reports Processing: FINISH
Finished: FAILURE
I'm running a basic pipeline that executes pylint on my repository code.
My Jenkins runs on Debian etch, the Jenkins version is 2.231.
At the end of the pipeline, I'm getting the following error :
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 24
Finished: FAILURE
As this page https://wiki.jenkins.io/display/JENKINS/Job+Exit+Status explains, the error code 24 refers to a "too many open files" error.
If I remove the pylint part of the job, the pipelines executes smoothly.
Il tried to configure the limit in /etc/security/limits.conf
jenkins soft nofile 64000
jenkins hard nofile 64000
and in the Jenkins config file /etc/default/jenkins :
MAXOPENFILES=64000
If I put "ulimit -n" in the pipeline, the configured value is displayed but it has no effect on the result that remains : ERROR: script returned exit code 24.
The problem comes from pylint that doesn't return a 0 code even if it ran successfully.
The solution is to use the --exit-zero option when running pylint.
pylint --exit-zero src
Jenkins with pylint gives build failure
I am trying to run sonar-scanner and access quality gate results and am kind of stuck after trying various options suggested on forums. This is my first time trying to post, so please let me know if I am missing any details. I do see the json payload in sonarqube server webhooks console but it is in failed status (red cross-mark). Ours is a shared CBJ and SonarQube server with limited access for me on both of those. Any help/guidance is really appreciated. Thank you so much.
======================================
SonarQube Configuration
Project_Name > Administration > Webhooks
Name: Webhook_Name
URL: https://CloudBeesJenkins_Server_FQDN/dev-master/sonarqube-webhook/
Secret: 'webhook_secret_text'
======================================
CBJ Configuration
CredentialsID: 'SonarQubeToken': Value: Scope: Global credentials (unrestricted)
======================================
Jenkins Job - Pipeline Script
/* this stage succeeds */
stage('SonarQube Analysis') {
def scannerHome = tool 'Sonar-Prod';
withSonarQubeEnv('Sonar-Prod') {
sh """${scannerHome}/bin/sonar-scanner -X \
-Dsonar.projectKey=ProjKey \
-Dsonar.sources=src \
-Dsonar.host.url=https://sonarqube_server_fqdn \
-Dsonar.login=sonar_project_secret_text"""
}
}
/* fails at waitForQualityGate */
stage("Quality Gate Status Check") {
timeout(time: 1, unit: 'HOURS')// Just in case something goes wrong, pipeline will be killed after a timeout
// had previously tried using waitForQualityGate() and waitForQualityGate(webhookSecretId: 'webhook_secret_text' with same result
def qg = waitForQualityGate(webhookSecretId: 'webhook_secret_text', credentialsId: 'sonar_project_secret_text') // Reuse taskId previously collected by withSonarQubeEnv
if (qg.status != 'OK') {
error "Pipeline aborted due to quality gate failure: ${qg.status}"
}
}
=====================================
Logs from Jenkins Server - Job Running Sonarscanner and qualitygate
SonarQube Scanner 4.2.0.1873
Java 1.8.0_242 Oracle Corporation (64-bit)
Linux 2.6.32-754.27.1.el6.x86_64 amd64
SonarQube server 7.9.1 - Community 7.9.1.27448
[CloudBees Jenkins Enterprise 2.204.3.7-rolling]
09:40:13.671 DEBUG: Upload report
09:40:13.931 DEBUG: POST 200 https://sonarqube_server_fqdn/api/ce/submit?projectKey=ProjKey | time=256ms
09:40:13.935 INFO: Analysis report uploaded in 264ms
09:40:13.938 INFO: ANALYSIS SUCCESSFUL, you can browse https://sonarqube_server_fqdn/dashboard?id=ProjKey
09:40:13.938 INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
09:40:13.938 INFO: More about the report processing at https://sonarqube_server_fqdn/api/ce/task?id=AXDt34Wae-uSoUyAgrS-
[Pipeline] waitForQualityGate
Checking status of SonarQube task 'AXDt34Wae-uSoUyAgrS-' on server 'Sonar-Prod'
org.sonarqube.ws.client.HttpException: Error 401 on https://sonarqube_server_fqdn/api/ce/task?id=AXDt34Wae-uSoUyAgrS-
It was a firewall issue. Communication from Jenkins to SonarQube server was opened but not the other way round. This issue can be closed.
When my build failed - test report is missing in Jenkins. What do I do wrong?
This is part of my jenkinsfile:
node {
stage('Integration tests') {
git url: "https://$autotestsGitRepo", branch: 'develop',credentialsId: gitlabCredentialsId
sh 'chmod +x gradlew && ./gradlew clean test
step([$class: 'JUnitResultArchiver', testResults:'build/test-results/test/*.xml'])
}
}
This is my jenkins output:
5 tests completed, 5 failed
Task :test FAILED
FAILURE: Build failed with an exception.
What went wrong: Execution failed for task ':test'.
There were failing tests. See the report at: file:///var/jenkins_home/workspace/test2/build/reports/tests/test/index.html
Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org
BUILD FAILED in 1m 7s 4 actionable tasks: 4 executed [Pipeline] }
[Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of
Pipeline ERROR: script returned exit code 1 Finished: FAILURE
I'm using gatling version 2.1.6 with jenkins plugins annd without maven, my scenario are launch with execute shell
#!/bin/bash
if [ ! -f "novapostme/jenkins/${JOB_NAME}.scala" ]; then
exit 1
fi
${GATLING_HOME}/bin/gatling.sh -df ${WORKSPACE}/novapostme/datas -rf ${WORKSPACE} -sf ${WORKSPACE} -s ${JOB_NAME}
Is each job is run alone on onlly one executor (useing blocking jobs options) all is working well, but when I run multiple job on executors I have sometimes this type of error
[workspace] $ /bin/bash /tmp/hudson6410661655551326741.sh
GATLING_HOME is set to /opt/gatling-charts-highcharts-2.1.6
Exception in thread "main" java.lang.NoClassDefFoundError: MeDocumentDownload
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at io.gatling.app.classloader.FileSystemBackedClassLoader.findClass(FileSystemBackedClassLoader.scala:71)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at io.gatling.app.classloader.SimulationClassLoader$$anonfun$simulationClasses$1.applyOrElse(SimulationClassLoader.scala:57)
at io.gatling.app.classloader.SimulationClassLoader$$anonfun$simulationClasses$1.applyOrElse(SimulationClassLoader.scala:57)
at scala.collection.immutable.List.collect(List.scala:303)
at io.gatling.app.classloader.SimulationClassLoader.simulationClasses(SimulationClassLoader.scala:57)
at io.gatling.app.Gatling.loadSimulations(Gatling.scala:92)
at io.gatling.app.Gatling.start(Gatling.scala:70)
at io.gatling.app.Gatling$.fromArgs(Gatling.scala:59)
at io.gatling.app.Gatling$.main(Gatling.scala:44)
at io.gatling.app.Gatling.main(Gatling.scala)
Caused by: java.lang.ClassNotFoundException: MeDocumentDownload
at io.gatling.app.classloader.FileSystemBackedClassLoader.findClass(FileSystemBackedClassLoader.scala:70)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more
Build step 'Execute shell' marked build as failure
Build step 'Groovy Postbuild' marked build as failure
Archiving Gatling reports...
No newer Gatling reports to archive.
Finished: FAILURE
The question is : is it possible to run simultaneous jobs on mutiple executor ?