Run docker build inside Jenkins Docker Slave - docker

Currently I've a CI pipeline with the following stages:
Build
Unit Tests
Static Code Analysis
This is how my Jenkinsfile looks like:
pipeline {
agent any
stages {
stage("Install") {
steps {
sh "npm install"
}
}
stage("Build") {
steps {
sh "npm run build"
}
}
stage("Format") {
steps {
sh "npm run format"
}
}
stage("Lint") {
steps {
sh "npm run lint"
}
}
stage("Test") {
steps {
sh "npm run test"
}
}
stage("Code Coverage") {
steps {
sh "npm run test:cov"
publishHTML(target: [
reportDir: "./coverage/lcov-report",
reportFiles: "index.html",
reportName: "Jest Coverage Report"
])
}
}
stage("End-To-End Testing") {
steps {
sh "npm run test:e2e"
}
}
}
}
I want to add more stages to my pipeline:
Build and tag Docker Image from Dockerfile
Push the image to the Docker Hub
Some more steps which would need Docker CLI
Example:
pipeline {
.
.
.
stage("Docker Build") {
steps {
sh "docker build -t [user_name]/[image_name]:[tag] .
}
}
}
I'm quite new to this, and I have tried multiple ways to install docker and it was unsuccessful and it is a bad practice too.
We can run docker run -v /var/run/docker.sock:/var/run/docker.sock ... but I can't use bind mounting while using docker build command.
Can someone please suggest me a way where I can use docker commands inside Jenkins SSH Agents?

Solution
Install Docker CLI without the Daemon in Jenkins Docker Slave. I have used this Docker Agent and installed Docker CLI inside it using this method
Then as a docker daemon I've used my remote docker host. (Also, you can configure the local docker host as remote using these steps). You can use docker remote host using --host flag. E.g. docker --host x.x.x.x:2375 build -t johndoe:calculator .
Syntax: docker --host [Docker_Host]:[Port] build -t [Image_Name]:[Image_Tag] .
My New Jenkinsfile is as follows:
pipeline {
agent any
stages {
stage("Install") {
steps {
sh "npm install"
}
}
stage("Build") {
steps {
sh "npm run build"
}
}
stage("Format") {
steps {
sh "npm run format"
}
}
stage("Lint") {
steps {
sh "npm run lint"
}
}
stage("Test") {
steps {
sh "npm run test"
}
}
stage("Code Coverage") {
steps {
sh "npm run test:cov"
publishHTML(target: [
reportDir: "./coverage/lcov-report",
reportFiles: "index.html",
reportName: "Jest Coverage Report"
])
}
}
stage("End-To-End Testing") {
steps {
sh "npm run test:e2e"
}
}
stage("Docker Build") {
steps {
withCredentials([string(credentialsId: 'Docker_Host', variable: 'DOCKER_HOST')]) {
sh 'docker --host $DOCKER_HOST build -t xxx/xxx .'
}
}
}
}
}
Note: I have stored Docker host URL on Jenkins as a credential and used it using withCredentials function.

Related

How do we install npm pdf-parse library in jenkins docker container

While running the Cypress tests on jenkins, I am getting the below error. Our jenkins is integrated with Docker container and devs asked me to install the pdf-parse library in docker container which will solve the issue. How do I install pdf-parse in docker container, which file does that ? Could some one please advise ?
Note: I am unable to see a docker file in my project root directory
11:38:29 Or you might have renamed the extension of your `pluginsFile`. If that's the case, restart the test runner.
11:38:29
11:38:29 Please fix this, or set `pluginsFile` to `false` if a plugins file is not necessary for your project.
11:38:29
11:38:29 Error: Cannot find module 'pdf-parse'
docker file:
FROM cypress/browsers:node12.14.1-chrome85-ff81
COPY package.json .
COPY package-lock.json .
RUN npm install --save-dev cypress
RUN $(npm bin)/cypress verify
# there is a built-in user "node" that comes from the very base Docker Node image
# we are going to recreate this user and give it _same id_ as external user
# that is going to run this container.
ARG USER_ID=501
ARG GROUP_ID=999
# if you want to see all existing groups uncomment the next command
# RUN cat /etc/group
RUN groupadd -g ${GROUP_ID} appuser
# do not log creating new user, otherwise there could be a lot of messages
RUN useradd -r --no-log-init -u ${USER_ID} -g appuser appuser
RUN install -d -m 0755 -o appuser -g appuser /home/appuser
# move test runner binary folder to the non-root's user home directory
RUN mv /root/.cache /home/appuser/.cache
USER appuser
jenkins file:
pipeline {
agent {
docker {
image 'abcdtest'
args '--link postgres:postgres -v /.composer:/.composer'
}
}
options {
ansiColor('xterm')
}
stages {
stage("print env variables") {
steps {
script {
echo sh(script: 'env|sort', returnStdout: true)
}
}
}
stage("composer install") {
steps {
script {
withCredentials([usernamePassword(credentialsId: 'bitbucket-api', passwordVariable: 'bitbucketPassword', usernameVariable: 'bitbucketUsername')]) {
def authProperties = readJSON file: 'auth.json.dist'
authProperties['http-basic']['bitbucket.sometest.com']['username'] = bitbucketUsername
authProperties['http-basic']['bitbucket.sometest.com']['password'] = bitbucketPassword
writeJSON file: 'auth.json', json: authProperties
}
}
sh 'php composer.phar install --prefer-dist --no-progress'
}
}
stage('unit tests') {
steps {
lock('ABCD Unit Tests') {
script {
try {
sh 'mv codeception.yml.dist codeception.yml'
sh 'mv tests/unit.suite.yml.jenkins tests/unit.suite.yml'
sh 'php vendor/bin/codecept run tests/unit --html'
}
catch (err) {
echo "unit tests step failed"
currentBuild.result = 'FAILURE'
}
finally {
publishHTML (target: [
allowMissing: false,
alwaysLinkToLastBuild: false,
keepAll: true,
reportDir: 'tests/_output/',
reportFiles: 'report.html',
reportName: "Unit Tests Report"
])
}
}
}
}
}
}
post {
success {
slackSend color: 'good', channel: '#jenkins-abcdtest-ci', message: "*SUCCESSED* - CI passed successfully for *${env.BRANCH_NAME}* (<${env.BUILD_URL}|build ${env.BUILD_NUMBER}>)"
}
failure {
slackSend color: 'danger', channel: '#jenkins-abcdtest-ci', message: "*FAILED* - CI failed for *${env.BRANCH_NAME}* (<${env.BUILD_URL}|build ${env.BUILD_NUMBER}> - <${env.BUILD_URL}console|click here to see the console output>)"
}
}
}
I suppose you use cypress/base:10 as the image to new a container in jenkins. If you don't have dockerfile, you may have to write your own dockerfile extends from cypress/base:10.
Dockerfile:
FROM cypress/base:10
RUN npm install pdf-parse
Then, docker build -t mycypress ., docker push mycypress to push the image to dockerhub(You may need an account) to let your jenkins use your new image to setup container.
NOTE: you will have to find how your project choose image to start your container, with this, you can find suitable way to install pdf-parse. One possible maybe next:
pipeline {
agent {
docker { image 'cypress/base:10' }
}
stages {
stage('Test') {
steps {
sh 'node --version'
}
}
}
}
Then, you may change docker { image 'cypress/base:10' } to docker { image 'mycypress' }.

How to run Jenkins parallel cypress on different agents?

I'm running parallel cypress in Jenkins on the same slave, and it's working,
I want to change the parallel stages so each stage will run on a different slave, how can I do it?
for example:
run "cypress tester A" on slave-1.
run "cypress tester B" on slave-2.
run "cypress tester C" on slave-3.
this is my current Jenkinsfile:
pipeline {
options {
timeout(time: 15, unit: 'MINUTES')
}
agent {
docker {
image 'cypress/base:12.18.2'
label 'slave-1'
}
}
parameters {
string(defaultValue: 'master', description: 'Branch name', name: 'branchName')
}
stages {
stage('build') {
steps {
echo 'Running build...'
sh 'npm ci'
sh 'npm run cy:verify'
}
}
stage('cypress parallel tests') {
environment {
CYPRESS_RECORD_KEY = 'MY_CYPRESS_RECORD_KEY'
CYPRESS_trashAssetsBeforeRuns = 'false'
}
parallel {
stage('cypress tester A') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run cypress:run"
}
}
stage('cypress tester B') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run cypress:run"
}
}
stage('cypress tester C') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run cypress:run"
}
}
}
}
}
post {
always {
cleanWs(deleteDirs: true)
echo 'Tests are finished'
}
}
}
The cypress:run command is:
cypress run --record --parallel --config videoUploadOnPasses=false --ci-build-id $BUILD_TAG
I was able to get this to work by explicityly defining the agent within each parallel stage:
parallel {
stage('cypress tester A') {
agent {
node: {
label "slave-1"
}
}
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run cypress:run"
}
}
stage('cypress tester B') {
agent {
node: {
label "slave-2"
}
}
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run cypress:run"
}
}
stage('cypress tester C') {
agent {
node: {
label "slave-3"
}
}
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run cypress:run"
}
}
}
However, one disadvantage I found is now that you're running cypress in each individual node/virtual machine, cypress needs to know where to find the running instance of your application. Cypress looks into cypress.json at baseUrl to see where to find your app. Its common to use a localhost address for development, which means cypress runnning on slave-1 will look for an app running on localhost of slave-1 - but there isn't one, so it will fail.
For simplicity's sake, I just did an npm install and npm start & npx wait-on http://localhost:3000 in each node:
stage('cypress tester A') {
agent {
node: {
label "slave-1"
}
}
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm install --silent"
sh "npm start & npx wait-on http://localhost:3000"
sh "npm run cypress:run"
}
}
This is obviously not very efficient because you have to install and run the app on each node. However, you could potentially set up a previous stage on a dedicated node (say, slave-0) to install and serve your project, and use that. Within your Jenkinsfile, you'll need to know the IP of that slave-0, or you could get it dynamically within your Jenkinsfile. Then instead of installing and running your project on slave-1, 2 and 3, you would install and run it just on slave-0, and use the CYPRESS_BASE_URL env variable to tell cypress where to find the running instance of your app. If the IP of slave-0 is 2222.2222.2222.2222, you might try something like this:
pipeline {
stage ('Serve your project'){
agent {
label 'slave-0'
}
steps {
sh 'npm install --silent'
sh 'npm start & npx wait-on http://localhost:3000'
}
}
stage('Cypress'){
environment {
CYPRESS_BASE_URL=2222.2222.2222.2222:3000
// other env vars
}
parallel {
stage {
agent {
label 'slave-1'
}
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run cypress:run"
}
}
// more parallel stages
}
}
}
There's a lot of variations you can do, but hopefully that will get you started.

Getting error Jenkin pipeline docker: command not found

Dockerfile:
pipeline {
agent any
stages {
stage ('Compile') {
steps {
withMaven(maven: 'maven_3_6_3') {
sh 'mvn clean compile'
}
}
}
stage ('unit test and Package') {
steps {
withMaven(maven: 'maven_3_6_3') {
sh 'mvn package'
}
}
}
stage ('Docker build') {
steps {
sh 'docker build -t dockerId/cakemanager .'
}
}
}
}
docker build -t dockerId/cakemanager .
/Users/Shared/Jenkins/Home/workspace/CDCI-Cake-Manager_master#tmp/durable-e630df16/script.sh:
line 1: docker: command not found
First install docker plugin from Manage Jenkins >> Manage Plugins >> Click on available and search for Docker and install it.
and then configure it on Manage Jenkins >> Global tool configuration.
You need to manually install docker on your Jenkins master or on agents if you're running builds on them.
Here's the doc to install docker on OS X https://docs.docker.com/docker-for-mac/install/

Npm test in Jenkins build takes 8 hours

My Jenkins build is still not finished after 8hrs. I have a simple React project I want to implement Continuous Integration with.
My Jenkinsfile looks like this:
pipeline {
agent {
docker {
image 'node'
args '-u root'
}
}
stages {
stage('Build') {
steps {
echo 'Building...'
sh 'npm install'
sh 'npm install node'
}
}
stage('Test') {
steps {
echo 'Testing...'
sh 'npm test'
}
}
}
}
I think what is happening is npm test is testing ALL the node modules. The build itself takes 44s.
Also, I have not been able to get npm install to install the node modules? So far as I understand it should install node automatically?
How can I stop it taking so long?
Override docker entrypoint with command --entrypoint \'\'
agent will therefore look like
agent {
docker {
image 'node'
args '-u root --entrypoint \'\''
}
}
This is a wild guess, all I can do with so little information

How to change the Agent label in Jenkins depending on Branch Name

I am creating a Jenkin Pipeline for below task.
Pull the latest code from vsts
Build the code and create .jar file out of it
creating a Docker image on the basis of the jar
tag the image
push the image into Docker registry
for this, I have written below Jenkinsfile
pipeline {
agent {
label "master"
}
stages {
stage('Build') {
steps {
echo '..........................Building Jar..........................'
sh 'npm install'
}
}
stage('Build-Image') {
steps {
echo '..........................Building Image..........................'
sh 'sudo docker build -t some-org/admin-portal:v0.1 --build-arg PORT=9007 --build-arg ENVIRONMENT=develop .'
}
}
stage('Tag-Image') {
steps {
echo '..........................Taging Image..........................'
sh 'sudo docker login some-repo -u username001 -p password'
sh 'sudo docker tag some-org/admin-portal:v0.1 some.dtr.io/some-org/admin-portal:v0.1'
}
}
stage('Push-Image') {
steps {
echo '..........................Pushing Image..........................'
sh 'sudo docker push some.dtr.io/some-org/admin-portal:v0.1'
}
}
}
}
Below is Jenkins job configuration snapshot for Pipeline
My Question is how can I change the agent label depending upon branch name or some conditions.
e.g if the branch is develop I want to use slave1 node and if the branch is production I want to use master
Any Help will be appreciable.
Thanks in Advance.
You can assign the agent labels inside the stage, so that you can execute the stages with required agents.
eg:
pipeline {
agent none
stages {
stage('Build') {
agent {
label "master"
}
steps {
echo '..........................Building Jar..........................'
sh 'npm install'
}
}
stage('Build-Image') {
agent {
label "master"
}
steps {
echo '..........................Building Image..........................'
sh 'sudo docker build -t some-org/admin-portal:v0.1 --build-arg PORT=9007 --build-arg ENVIRONMENT=develop .'
}
}
stage('Tag-Image') {
agent {
label "slave1"
}
steps {
echo '..........................Taging Image..........................'
sh 'sudo docker login some-repo -u username001 -p password'
sh 'sudo docker tag some-org/admin-portal:v0.1 some.dtr.io/some-org/admin-portal:v0.1'
}
}
stage('Push-Image') {
agent {
label "slave1"
}
steps {
echo '..........................Pushing Image..........................'
sh 'sudo docker push some.dtr.io/some-org/admin-portal:v0.1'
}
}
}
}

Resources