The following is an excerpt from a Jenkinsfile used in a multi-branch pipeline:
def GetNextFreePort() {
lock ('portProvider') {
def port = powershell(returnStdout: true, script: '((Get-NetTCPConnection | Sort-Object -Property LocalPort | Select-Object -Last 1).LocalPort) + 1')
}
return port.trim()
}
I'd like the line that gets the port number (on windows) to return a different port for each branch. However, despite using the lockable resources plugin, I cannot serialize access to the powershell callout that gets the next available port.
In the end I managed to achieve what I wanted to do via the Jenkins lock resource plugin. Here is my method to obtain an external port number and start the container:
def StartContainer() {
PORT_NUMBER = GetNextFreePort()
bat "docker run -e \"ACCEPT_EULA=Y\" -e \"SA_PASSWORD=P#ssword1\" --name ${CONTAINER_NAME} -d -i -p ${PORT_NUMBER}:1433 microsoft/mssql-server-linux:2017-GA"
powershell "While (\$((docker logs ${CONTAINER_NAME} | select-string ready | select-string client).Length) -eq 0) { Start-Sleep -s 1 }"
}
and here is the call to this which is wrapped by a call to lock:
stage('start container') {
steps {
RemoveContainer()
timeout(time: 20, unit: 'SECONDS') {
lock ('create SQL Server container') {
StartContainer()
}
}
}
}
Related
I have 2 servers on AWS EC2. I want to deploy our node JS application into both the instances.
My below code is working fine if both the instances are available.
node (label: 'test') {
def sshConn = 'ssh -i /home/ec2-user/pem/ourpemfile.pem ec2-user#IP for server1'
def sshConn1 = 'ssh -i /home/ec2-user/pem/ourpemfile.pem ec2-user#IP for server2'
stage('Checkout from Github')
{
checkout([
$class: 'GitSCM',
*
*
])
}
stage('Build for Node1')
{
echo "Starting to Build..."
sh "$sshConn pm2 stop application || true"
}
stage('Deploy to Node1')
{
echo "Starting Deployment..."
"
}
stage('Build for Node2')
{
echo "Starting to Build..."
sh "$sshConn1 pm2 stop application || true"
}
stage('Deploy to Node2')
{
echo "Starting Deployment..."
}
}
But my use cases is .
if one of the server will stopped then build job must be successful and application should deploy on available instance.
Currently, I am facing timeout error if we stop server1 and run the jenkins job.
Depends on your setup.
1) you can connect your nodes to jenkins as slaves vi ssh-slaves plugin.
And then you can run on your servers via
node('node_label') {
sh('any command here')
}
2) you can use ssh-agent plugin. You can put your private key into Jenkins credentials
3) use retry
retry(3) {
// your code
}
You can check ec2 instances states via aws-cli commands, and depending on theirs states do or not you deployment :
If you want to give it a shot, you'll have to declare your AWS credentials in jenkins using 'CloudBees AWS Credentials' plugin.
and add to your pipeline something like that:
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
accessKeyVariable: 'aV',
secretKeyVariable: 'sV',
credentialsId: 'id_of_your_credentials',]]) {
sh '''
AWS_ACCESS_KEY_ID=${aV}\
AWS_SECRET_ACCESS_KEY=${sV}\
AWS_DEFAULT_REGION=us-east-1\
aws ec2 describe-instances --instance-id --filters Name=instance-state-name,Values=running --query "Reservations[*].Instances[?Tags[?Key == 'Name' && contains(Value, 'server1')]].[Tags[3].Value,NetworkInterfaces[0].PrivateIpAddress,InstanceId,State.Name]" --output text
'''
}
Regardless to the AWS cli cmd :
I don't know how you manage your servers, I've assumed that you use a tag 'Name' to identify your servers.
Also, I think you should consider max suggestion and use ssh plugin for managing the configuration, credentials ...etc...
Another option can be using ssh-agent. You have to store private keys in credentials plugin (also possible to configure AWS secrets for that)
and then in your pipeline
https://www.jenkins.io/doc/pipeline/steps/ssh-agent/
node {
sshagent (credentials: ['deploy-dev']) {
sh 'ssh -o StrictHostKeyChecking=no -l cloudbees 192.168.1.106 uname -a'
}
}
I want to run two docker container in declarative Jenkins pipeline, because I have container with backend which utilises Selenium server container for test. I know that there is a scripted example but I wonder if there is a declarative option.
Scripted looks like this:
node {
checkout scm
docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw"') { c ->
docker.image('mysql:5').inside("--link ${c.id}:db") {
/* Wait until mysql service is up */
sh 'while ! mysqladmin ping -hdb --silent; do sleep 1; done'
}
docker.image('centos:7').inside("--link ${c.id}:db") {
/*
* Run some tests which require MySQL, and assume that it is
* available on the host name `db`
*/
sh 'make check'
}
}
}
In the end I used description from here.
withRun - executes commands on the host
inside - inside the container
stage ('Test') {
steps {
// Create network where I will connect all containers
sh 'docker network create test'
script {
//withRun command starts the container and doesn't stop it untill all inside is executed.
//Commands inside are executed on HOST machine
docker.image('selenium/standalone-chrome').withRun("-p 4444:4444 --name=selenium -itd --network=test") {
docker.image("$CONTAINER_NAME:front").withRun("-p 3001:80 --name=front -itd --network=test") {
//We start backend container...
docker.image("$CONTAINER_NAME:back").withRun("-p 8001:80 --name=back -itd --network=test") {
//...and with inside command execute commands *surprise* inside the container
docker.image("$CONTAINER_NAME:back").inside("-itd --network=test") {
//execute commands inside the container
}
}
}
}
}
}
}
I've created my Jenkinsfile for building my project in production and the pipeline looks like this:
pipeline {
agent any
stages {
stage('Pull') {
steps {
sh '''ssh ${SSH_USER}#${SERVER_ADDRESS} <<EOF
cd ${SOURCE_FOLDER}/project
git pull
git status
EOF'''
}
}
stage('Composer') {
parallel {
stage('Composer') {
steps {
sh '''ssh ${SSH_USER}#${SERVER_ADDRESS} <<EOF
docker run --rm -v ${SOURCE_FOLDER}/project:/app composer/composer:latest install
EOF'''
}
}
stage('Composer 2') {
steps {
sh '''ssh ${SSH_USER}#${SERVER_ADDRESS} <<EOF
docker run --rm -v ${SOURCE_FOLDER}/project/sub:/app
composer/composer:latest install
EOF'''
}
}
}
}
}
}
Is there a way to have all the stages all in one single SSH connection in order to minimise the overhead and the connection number?
I've done all the SSL stuff manually by creating the keys and pasting the public key on the production machine.
You can create a function for the connection and pass the SSH_USER & SERVER_ADDRESS as input parameters to that function. Call this function from all your stages.
I am using docker to simulate postgres database for my app. I was testing it in Cypress for some time and it works fine. I want to set up Jenkins for further testing, but I seem stuck.
On my device, I would use commands
docker create -e POSTGRES_DB=myDB -p 127.0.0.1:5432:5432 --name myDB postgres
docker start myDB
to create it. How can I simulate this in Jenkins pipeline? I need the DB for the app to work.
I use Dockerfile as my agent, and I have tried putting the ENV variables there, but it does not work. Docker is not installed on the pipeline.
The way I see it is either:
Create an image by using a
Somehow install docker inside the pipeline and use the same commands
Maybe with master/slave nodes? I don't understand them well yet.
This might be a use case for sidecar pattern one of Jenkins Pipeline's advanced features.
For example (from the above site):
node {
checkout scm
docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw"') { c ->
docker.image('mysql:5').inside("--link ${c.id}:db") {
/* Wait until mysql service is up */
sh 'while ! mysqladmin ping -hdb --silent; do sleep 1; done'
}
docker.image('centos:7').inside("--link ${c.id}:db") {
/*
* Run some tests which require MySQL, and assume that it is
* available on the host name `db`
*/
sh 'make check'
}
}
}
The above example uses the object exposed by withRun, which has the
running container’s ID available via the id property. Using the
container’s ID, the Pipeline can create a link by passing custom
Docker arguments to the inside() method.
Best thing is that the containers should be automatically stopped and removed when the work is done.
EDIT:
To use docker network instead you can do the following (open Jira to support this OOTB). Following helper function
def withDockerNetwork(Closure inner) {
try {
networkId = UUID.randomUUID().toString()
sh "docker network create ${networkId}"
inner.call(networkId)
} finally {
sh "docker network rm ${networkId}"
}
}
Actual usage
withDockerNetwork{ n ->
docker.image('sidecar').withRun("--network ${n} --name sidecar") { c->
docker.image('main').inside("--network ${n}") {
// do something with host "sidecar"
}
}
}
For declarative pipelines:
pipeline {
agent any
environment {
POSTGRES_HOST = 'localhost'
POSTGRES_USER = myuser'
}
stages {
stage('run!') {
steps {
script {
docker.image('postgres:9.6').withRun(
"-h ${env.POSTGRES_HOST} -e POSTGRES_USER=${env.POSTGRES_USER}"
) { db ->
// You can your image here but you need psql to be installed inside
docker.image('postgres:9.6').inside("--link ${db.id}:db") {
sh '''
psql --version
until psql -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -c "select 1" > /dev/null 2>&1 || [ $RETRIES -eq 0 ]; do
echo "Waiting for postgres server, $((RETRIES-=1)) remaining attempts..."
sleep 1
done
'''
sh 'echo "your commands here"'
}
}
}
}
}
}
}
Related to Docker wait for postgresql to be running
I'm in a process of migrating from freestyle jobs chained into pipeline to have the pipeline in a Jenkinsfile.
My current pipeline will execute 2 jobs in parallel, one will create a tunnel to database (with a randomly generated port) and the next job needs to get this port number, so I'm performing a curl command and reading the console of the create-db-tunnel job and storing the port number. The create-db-tunnel needs to keep running as the follow up job is connecting to the database and is taking DB dump. This is the curl command which I run on the second job and which is returning the randomly generated port number from the established DB tunnel:
Port=$(curl -u ${USERNAME}:${TOKEN} http://myjenkinsurl.com/job/create-db-tunnel/lastBuild/consoleText | grep Port | grep -Eo '[0-9]{3,5}')
I wonder if there is anything similar I can use in Jenkinsfile? I currently have the 2 jobs triggered in parallel, but since the create-db-tunnel is no longer a freestyle job, I'm not sure if I can get the port number still? I can confirm that the console logs for the db_tunnel stage has the port number in there, just not sure how can I query that console. Here is my jenkinsfile:
pipeline {
agent any
environment {
APTIBLE_LOGIN = credentials('aptible')
}
stages {
stage('Setup') {
parallel {
// run db_tunnel and get_port in parralel
stage ('db_tunnel') {
steps {
sh """
export PATH=$PATH:/usr/local/bin
aptible login --email=$APTIBLE_LOGIN_USR --password=$APTIBLE_LOGIN_PSW
aptible db:tunnel postgres-prod & sleep 30s
"""
}
}
stage('get_port') {
steps {
sh """
sleep 15s
//this will not work
Port=$(curl -u ${USERNAME}:${TOKEN} http://myjenkinsurl.com/job/db_tunnel/lastBuild/consoleText | grep Port | grep -Eo '[0-9]{3,5}')
echo "Port=$Port" > port.txt
"""
}
}
}
}
}
}
Actually, I found a solution to my question - it was a very similar curl command I had to run and I'm now getting the desired port number I needed. Here is the jenkinsfile if someone is interested:
pipeline {
agent any
environment {
APTIBLE_LOGIN = credentials('aptible')
JENKINS_TOKEN = credentials('jenkins')
}
stages {
stage('Setup') {
parallel {
// run db_tunnel and get_port in parralel
stage ('db_tunnel') {
steps {
sh """
export PATH=$PATH:/usr/local/bin
aptible login --email=$APTIBLE_LOGIN_USR --password=$APTIBLE_LOGIN_PSW
aptible db:tunnel postgres-prod & sleep 30s
"""
}
}
stage('get_port') {
steps {
sh """
sleep 20
Port=\$(curl -u $JENKINS_TOKEN_USR:$JENKINS_TOKEN_PSW http://myjenkinsurl.com/job/schema-archive-jenkinsfile/lastBuild/consoleText | grep Port | grep -Eo '[0-9]{3,5}')
echo "Port=\$Port" > port.txt
"""
}
}
}
}
}
}