I am trying to execute my tests in parallel but it is not working.
I have this config in jeniks file .sh
docker run --rm -i --name integration-tests-$p --network=host $DOCKER_VOLUME -e CYPRESS_INCLUDE_TAGS=$TAG_TO_INCLUDES -e CYPRESS_EXCLUDE_TAGS=disabled -w $WORKING_DIR cypress/included:10.10.0 --record --group 4x-electron --key keyId --parallel --ci-build-id $BUILD_NUMBER
and the other config in the .jenkins file
stage('Integration Tests') {
when {
expression { return RUN_E2E != ''}
}
steps {
sh "$RUSH hc -t web" // Check if web is running
sh "$RUSH integration-tests --tags=smoke --to manager"
}
}
I need to run these tests in parallel, can anyone help?
Related
I am writing a declarative pipeline in a Jenkinsfile in order to build and deploy an app.
The deployment is usually done by sshing to the docker host and running:
cd myDirectory
docker stack deploy --compose-file docker-compose.yml foo
I managed to run a single shell command via ssh, but don't know how to run multiple commands after eacht other.
This is what I have now:
pipeline {
agent { label 'onlyMyHost' }
stages {
stage("checkout"){...}
stage("build"){...}
stage("deploy") {
steps {
sshagent(credentials: ['my-sshKey']){
sh 'ssh -o StrictHostKeyChecking=no myUser#foo.bar.com hostname'
sh ("""
ssh -o StrictHostKeyChecking=no myUser#foo.bar.com 'bash -s' < "cd MyDirectory && docker stack deploy --composefile docker-compose.yml foo"
""")
}
}
}
}
}
This fails. What is a good way of running a script on a remote from my specific jenkins-worker
Not sure why 'bash -s' is added here. Removing that from your SSH command should allow you to execute docker deploy remotely.
Moreover, you may run any number of commands in the same line by appending with ; after each command. For example:
ssh -o StrictHostKeyChecking=no myUser#foo.bar.com < "cd MyDirectory && docker stack deploy --composefile docker-compose.yml foo ; docker ps"
i am trying to ssh into a remote host and then execute certain commands on the remote host's shell. Following is my pipeline code.
pipeline {
agent any
environment {
// comment added
APPLICATION = 'app'
ENVIRONMENT = 'dev'
MAINTAINER_NAME = 'jenkins'
MAINTAINER_EMAIL = 'jenkins#email.com'
}
stages {
stage('clone repository') {
steps {
// cloning repo
checkout scm
}
}
stage('Build Image') {
steps {
script {
sshagent(credentials : ['jenkins-pem']) {
sh "echo pwd"
sh 'ssh -t -t ubuntu#xx.xxx.xx.xx -o StrictHostKeyChecking=no'
sh "echo pwd"
sh 'sudo -i -u root'
sh 'cd /opt/docker/web'
sh 'echo pwd'
}
}
}
}
}
}
But upon running this job it executes sh 'ssh -t -t ubuntu#xx.xxx.xx.xx -o StrictHostKeyChecking=no' successfully but it stops there and does not execute any further commands. I want to execute the commands that are written after ssh command inside the remote host's shell. any help is appreciated.
I would try something like this:
sshagent(credentials : ['jenkins-pem']) {
sh "echo pwd"
sh 'ssh -t -t ubuntu#xx.xxx.xx.xx -o StrictHostKeyChecking=no "echo pwd && sudo -i -u root && cd /opt/docker/web && echo pwd"'
}
I resolve this issue
script
{
sh """ssh -tt login#host << EOF
your command
exit
EOF"""
}
stage("DEPLOY CONTAINER"){
steps {
script {
sh """
#!/bin/bash
sudo ssh -i /path/path/keyname.pem username#serverip << EOF
sudo bash /opt/filename.sh
exit 0
<< EOF
"""
}
}
}
There is a better way to run commands on remote using SSH. I know this is late answer but I just explored this thing so would like to share and this will help others to resolve this problem easily.
I just found this link helpful on how to run multiple commands on remote using SSH. Also we can run multiple commands conditionally as mentioned in above blog.
By going through it, I found the syntax:
ssh username#hostname "command1; command2;commandN"
Now, how to run command inside remote hots using SSH in Jenkins pipeline?
Here is the solution:
pipeline {
agent any
environment {
/*
define your command in variable
*/
remoteCommands =
"""java --version;
java --version;
java --version """
}
stages {
stage('Login to remote host') {
steps {
sshagent(['ubnt-creds']) {
/*
Provide variable as argument in ssh command
*/
sh 'ssh -tt username#hostanem $remoteCommands'
}
}
}
}
}
Firstly and optionally, you can define a variable that holds all commands separated by ;(semicolon) and then pass it as parameter in command.
Another way, you can also pass your commands directly to ssh command as
sh "ssh -tt username#hostanem 'command1;command2;commandN'"
I have used it in my code and it's working great!
see the output here
Happy Learning :)
I have the JenkinsFile as followed
pipeline {
agent any
environment {
DISABLE_AUTH = 'true'
DB_ENGINE = 'sqlite'
}
stages {
stage('Run unit/integration tests') {
steps {
sh 'make test'
}
}
stage('Build application artefacts') {
steps {
sh 'make build'
}
}
stage('Create release environment and run acceptance tests') {
steps {
sh 'make release'
}
}
stage('Example') {
steps {
echo 'Reached example'
}
}
}
and Makefile as follows
test:
${INFO} "Pulling latest images..."
# docker-compose -p $(DEV_PROJECT) -f $(DEV_COMPOSE_FILE) pull
${INFO} "Building images..."
# docker-compose -p $(DEV_PROJECT) -f $(DEV_COMPOSE_FILE) build --pull test
# docker-compose -p $(DEV_PROJECT) -f $(DEV_COMPOSE_FILE) build cache
${INFO} "Ensuring database is ready..."
# docker-compose -p $(DEV_PROJECT) -f $(DEV_COMPOSE_FILE) up agent
${INFO} "Running tests..."
# docker-compose -p $(DEV_PROJECT) -f $(DEV_COMPOSE_FILE) up test
# # cd $$(docker-compose -p $(DEV_PROJECT) -f $(DEV_COMPOSE_FILE) ps -q test):/reports
# # cp *.xml ./reports
# echo $$(docker-compose -p $(DEV_PROJECT) -f $(DEV_COMPOSE_FILE) ps -q test)
# docker cp $$(docker-compose -p $(DEV_PROJECT) -f $(DEV_COMPOSE_FILE) ps -q test):**/reports/*.xml
${CHECK} $(DEV_PROJECT) $(DEV_COMPOSE_FILE) test
${INFO} "Testing complete"
And when i run it on Jenkins it gives the error that 'make command not found' on the console of the Jenkins.I think this is because of the environment variable.How to fix this?
I want to make a Jenkinsfile that will do tests and build my Spring boot Java application. The problem is that my tests require Postgres and RabbitMQ.
What I'm trying to do:
1) Setup Jenkins in docker
## Run Jenkins Docker :
sudo docker run -d -p 8080:8080 -p 50000:50000 -v /home/jenkins:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock -u root jenkins
Bash into docker container
## Bash into new docker container
docker exec -it {{ontainer_ID}} bash
## Download an install docker as root
curl -sSL https://get.docker.com/ | sh
exit
2) Make pipeline to do it:
pipeline {
agent {
docker {
image 'maven:3-alpine'
args '-v /root/.m2:/root/.m2'
}
}
stages {
stage('Build') {
steps {
sh 'mvn -B -DskipTests clean package'
}
}
stage('Test') {
steps {
/* Run some tests which require PostgreSQL */
sh 'mvn test'
}
post {
always {
junit 'target/surefire-reports/*.xml'
}
}
}
}
}
My goal to add postgres and rabbit to be launched on the phase right before tests. I found this https://jenkins.io/doc/book/pipeline/docker/
There is an example how to run additional docker images:
checkout scm
/*
* In order to communicate with the MySQL server, this Pipeline explicitly
* maps the port (`3306`) to a known port on the host machine.
*/
docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw" -p 3306:3306') { c ->
/* Wait until mysql service is up */
sh 'while ! mysqladmin ping -h0.0.0.0 --silent; do sleep 1; done'
/* Run some tests which require MySQL */
sh 'make check'
}
Looking for some expirienced devops who can help with my setup. Thanks.
At the time of writing, declarative pipeline doesn't support such sidecar containers (as described in the docs. So what you found is correct for your problem.
The snippet you found is, however, for scripted pipeline. To use this within your declarative pipeline, you need to wrap it in a script step:
stage('Test') {
steps {
docker.image('postgres:9').withRun('<whatever perameters you need>') { c ->
sh 'mvn test'
}
}
}
Of course, replace this with the postgres
I'm trying to execute an SSH command from inside a Docker container in a Jenkins pipeline. I'm using the CloudBees Docker Pipeline Plugin to spin up the container and execute commands, and the SSH Agent Plugin to manage my SSH keys. Here's a basic version of my Jenkinsfile:
node {
step([$class: 'WsCleanup'])
docker.image('node').inside {
stage('SSH') {
sshagent (credentials: [ 'MY_KEY_UUID' ]) {
sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu#example.org uname -a"
}
}
}
}
When the SSH command runs, I get this error:
+ ssh -vvv -o StrictHostKeyChecking=no ubuntu#example.org uname -a
No user exists for uid 1005
I combed through the logs and realized the Docker Pipeline Plugin is automatically telling the container to run with the same user that is logged in on the host by passing a UID as a command line argument:
$ docker run -t -d -u 1005:1005 [...]
I decided to check what users existed in the host and the container by running cat /etc/passwd in each environment. Sure enough, the list of users was different in each. 1005 was the jenkins user on the host machine, but that UID didn't exist in the container. To solve the issue, I mounted /etc/passwd from the host to the container when spinning it up:
node {
step([$class: 'WsCleanup'])
docker.image('node').inside('-v /etc/passwd:/etc/passwd') {
stage('SSH') {
sshagent (credentials: [ 'MY_KEY_UUID' ]) {
sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu#example.org uname -a"
}
}
}
}
The solution provided by #nathan-thompson is awesome, but in my case I was unable to find the user even in the /etc/passwd of the host machine! It means mounting the passwd file did not fix the problem. This question https://superuser.com/questions/580148/users-not-found-in-etc-passwd suggested some users are logged in the host using an identity provider like LDAP.
The solution was finding a way to add the proper line to the passwd file on the container. Calling getent passwd $USER on the host will provide the passwd line for the Jenkins user running the container.
I added a step running on the node (and not the docker agent) to get the line and save it in a file. Then in the next step I mounted the generated passwd to the container:
stages {
stage('Create passwd') {
steps {
sh """echo \$(getent passwd \$USER) > /tmp/tmp_passwd
"""
}
}
stage('Test') {
agent {
docker {
image '*******'
args '***** -v /tmp/tmp_passwd:/etc/passwd'
reuseNode true
registryUrl '*****'
registryCredentialsId '*****'
}
}
steps {
sh """ssh -i ********
"""
}
}
}
I just found another solution to this problem, that I want to share. It differentiates from the existing solutions in that it allows to run the complete pipeline in one agent, instead of per stage.
The trick is to, instead of directly using an image, refer to a Dockerfile (which may be build FROM the original) and then add the user:
# Dockerfile
FROM node
ARG jenkinsUserId=
RUN if ! id $jenkinsUserId; then \
usermod -u ${jenkinsUserId} jenkins; \
groupmod -g ${nodeId} jenkins; \
fi
// Jenkinsfile
pipeline {
agent {
dockerfile {
additionalBuildArgs "--build-arg jenkinsUserId=\$(id -u jenkins)"
}
}
}
agent {
docker {
image 'node:14.10.1-buster-slim'
args '-u root:root'
}
}
environment {
SSH_deploy = credentials('e99988ea-6bdc-45fc-b9e1-536b875bcac7')
}
stage('build') {
steps {
sh '''#!/bin/bash
eval $(ssh-agent -s)
cat $SSH_deploy | tr -d '\r' | ssh-add -
touch .env
echo 'REACT_APP_BASE_API = "//172.22.132.115:8080"' >> .env
echo 'REACT_APP_ADMIN_PANEL_URL = "//172.22.132.115"' >> .env
yarn install
CI=false npm run build
ssh -t -o StrictHostKeyChecking=no root#172.22.132.115 'rm -rf /usr/local/src/build'
scp -r -o StrictHostKeyChecking=no build root#172.22.132.115:/usr/local/src/
ssh -t -o StrictHostKeyChecking=no root#172.22.132.115 'systemctl restart nginx'
'''
}
From the solution provided by Nathan Thompson, I modified it this way for Jenkins DOCKER build container which runs inside a Jenkins DOCKER-slave. #docker in docker
if (validated_parameters.custom_gradle_image){
docker.image(validated_parameters.custom_gradle_image).inside(" -v /etc/passwd:/etc/passwd -v /var/lib/jenkins/.ssh/:/var/lib/jenkins/.ssh/ "){
sshagent(['jenkins-git-io']){
sh "${gradleCommand}"
}