I have 2 servers on AWS EC2. I want to deploy our node JS application into both the instances.
My below code is working fine if both the instances are available.
node (label: 'test') {
def sshConn = 'ssh -i /home/ec2-user/pem/ourpemfile.pem ec2-user#IP for server1'
def sshConn1 = 'ssh -i /home/ec2-user/pem/ourpemfile.pem ec2-user#IP for server2'
stage('Checkout from Github')
{
checkout([
$class: 'GitSCM',
*
*
])
}
stage('Build for Node1')
{
echo "Starting to Build..."
sh "$sshConn pm2 stop application || true"
}
stage('Deploy to Node1')
{
echo "Starting Deployment..."
"
}
stage('Build for Node2')
{
echo "Starting to Build..."
sh "$sshConn1 pm2 stop application || true"
}
stage('Deploy to Node2')
{
echo "Starting Deployment..."
}
}
But my use cases is .
if one of the server will stopped then build job must be successful and application should deploy on available instance.
Currently, I am facing timeout error if we stop server1 and run the jenkins job.
Depends on your setup.
1) you can connect your nodes to jenkins as slaves vi ssh-slaves plugin.
And then you can run on your servers via
node('node_label') {
sh('any command here')
}
2) you can use ssh-agent plugin. You can put your private key into Jenkins credentials
3) use retry
retry(3) {
// your code
}
You can check ec2 instances states via aws-cli commands, and depending on theirs states do or not you deployment :
If you want to give it a shot, you'll have to declare your AWS credentials in jenkins using 'CloudBees AWS Credentials' plugin.
and add to your pipeline something like that:
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
accessKeyVariable: 'aV',
secretKeyVariable: 'sV',
credentialsId: 'id_of_your_credentials',]]) {
sh '''
AWS_ACCESS_KEY_ID=${aV}\
AWS_SECRET_ACCESS_KEY=${sV}\
AWS_DEFAULT_REGION=us-east-1\
aws ec2 describe-instances --instance-id --filters Name=instance-state-name,Values=running --query "Reservations[*].Instances[?Tags[?Key == 'Name' && contains(Value, 'server1')]].[Tags[3].Value,NetworkInterfaces[0].PrivateIpAddress,InstanceId,State.Name]" --output text
'''
}
Regardless to the AWS cli cmd :
I don't know how you manage your servers, I've assumed that you use a tag 'Name' to identify your servers.
Also, I think you should consider max suggestion and use ssh plugin for managing the configuration, credentials ...etc...
Another option can be using ssh-agent. You have to store private keys in credentials plugin (also possible to configure AWS secrets for that)
and then in your pipeline
https://www.jenkins.io/doc/pipeline/steps/ssh-agent/
node {
sshagent (credentials: ['deploy-dev']) {
sh 'ssh -o StrictHostKeyChecking=no -l cloudbees 192.168.1.106 uname -a'
}
}
Related
Below is the Jenkins DSL groovy for setting the Terraform path and retrieving the service principal credentials to run Terraform init and Terraform plan.
When ran against Terraform 12.0 version I get the error below even though I tested using the same Azure service principal credentials mentioned in the pipeline as below using a Jenkins free style job and az login worked fine.
+ terraform init -input=false
[0m[1mInitializing modules...[0m
[0m[1mInitializing the backend...[0m
[31m
[1m[31mError: [0m[0m[1mError building ARM Config: Error populating Client ID from the Azure CLI: No Authorization Tokens were found - please re-authenticate using `az login`.[0m
[0m[0m[0m
pipeline{
agent any
stages{
stage('Set Terraform path') {
steps {
script {
def tfHome = tool name: 'Terraform'
env.PATH = "${tfHome}:${env.PATH}"
}
sh 'terraform version'
}
}
stage('Provision infrastructure') {
steps {
dir('environments/dev')
{
withCredentials([azureServicePrincipal('xx-xxx-subscription-azure-sp')]) {
sh 'az login --service-principal -u $AZURE_CLIENT_ID -p $AZURE_CLIENT_SECRET -t $AZURE_TENANT_ID'
sh 'terraform init -input=false'
sh 'terraform plan -out=tfplan -input=false'
}
// sh ‘terraform destroy -auto-approve’
}
}
}
}
}
I've created my Jenkinsfile for building my project in production and the pipeline looks like this:
pipeline {
agent any
stages {
stage('Pull') {
steps {
sh '''ssh ${SSH_USER}#${SERVER_ADDRESS} <<EOF
cd ${SOURCE_FOLDER}/project
git pull
git status
EOF'''
}
}
stage('Composer') {
parallel {
stage('Composer') {
steps {
sh '''ssh ${SSH_USER}#${SERVER_ADDRESS} <<EOF
docker run --rm -v ${SOURCE_FOLDER}/project:/app composer/composer:latest install
EOF'''
}
}
stage('Composer 2') {
steps {
sh '''ssh ${SSH_USER}#${SERVER_ADDRESS} <<EOF
docker run --rm -v ${SOURCE_FOLDER}/project/sub:/app
composer/composer:latest install
EOF'''
}
}
}
}
}
}
Is there a way to have all the stages all in one single SSH connection in order to minimise the overhead and the connection number?
I've done all the SSL stuff manually by creating the keys and pasting the public key on the production machine.
You can create a function for the connection and pass the SSH_USER & SERVER_ADDRESS as input parameters to that function. Call this function from all your stages.
Our env: Jenkins version: 2.138.3
Kubernetes plugin: 1.13.5
Sshagent plugin: 1.17
I have a job that runs OK on an AWS machine (use sshagent works as it should) but when I run the same job on our Kubernetes cluster it failed on ssh error.
Attached the working pipeline:
pipeline {
agent {
label 'deploy-test'
}
stages {
stage('sshagent') {
steps {
script {
sshagent(['deploy_user']) {
sh 'ssh -o StrictHostKeyChecking=no 99.99.999.99 ls'
}
}
}
}
}
}
If I change the label to label 'k8s-slave', it fails on:
+ ssh -o StrictHostKeyChecking=no 99.99.999.99 ls
Warning: Permanently added '99.99.999.99' (ECDSA) to the list of known hosts.
Permission denied (publickey).
Any idea?
just added my kubernetes configuration in Jenkins
I have a simple jenkins pipeline build, this is my jenkinsfile:
pipeline {
agent any
stages {
stage('deploy-staging') {
when {
branch 'staging'
}
steps {
sshagent(['my-credentials-id']) {
sh('git push joe#repo:project')
}
}
}
}
}
I am using sshagent to push to a git repo on a remote server. I have created credentials that point to a private key file in Jenkins master ~/.ssh.
When I run the build, I get this output (I replaced some sensitive info with *'s):
[ssh-agent] Using credentials *** (***#*** ssh key)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-cjbm7oVQaJYk/agent.11558
SSH_AGENT_PID=11560
$ ssh-add ***
Identity added: ***
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 11560 killed;
[ssh-agent] Stopped.
[TDBNSSBFW6JYM3BW6AAVMUV4GVSRLNALY7TWHH6LCUAVI7J3NHJQ] Running shell script
+ git push joe#repo:project
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
As you can see, the ssh-agent starts, stops immediately after and then runs the git push command. The weird thing is it did work correctly once but that seemed completely random.
I'm still fairly new to Jenkins - am I missing something obvious? Any help appreciated, thanks.
edit: I'm running a multibranch pipeline, in case that helps.
I recently had a similar issue though it was inside a docker container.
The logs gave the impression that ssh-agent exits too early but actually the problem was that I had forgotten to add the git server to known hosts.
I suggest ssh-ing onto your jenkins master and trying to do the same steps as the pipeline does with ssh-agent (the cli). Then you'll see where the problem is.
E.g:
eval $(ssh-agent -s)
ssh-add ~/yourKey
git clone
As explained on help.github.com
Update:
Here a util to add knownHosts if not yet added:
/**
* Add hostUrl to knownhosts on the system (or container) if necessary so that ssh commands will go through even if the certificate was not previously seen.
* #param hostUrl
*/
void tryAddKnownHost(String hostUrl){
// ssh-keygen -F ${hostUrl} will fail (in bash that means status code != 0) if ${hostUrl} is not yet a known host
def statusCode = sh script:"ssh-keygen -F ${hostUrl}", returnStatus:true
if(statusCode != 0){
sh "mkdir -p ~/.ssh"
sh "ssh-keyscan ${hostUrl} >> ~/.ssh/known_hosts"
}
}
I was using this inside docker, and adding it to my Jenkins master's known_hosts felt a bit messy, so I opted for something like this:
In Jenkins, create a new credential of type "Secret text" (let's call it GITHUB_HOST_KEY), and set its value to be the host key, e.g.:
# gets the host for github and copies it. You can run this from
# any computer that has access to github.com (or whatever your
# git server is)
ssh-keyscan github.com | clip
In your Jenkinsfile, save the string to known_hosts
pipeline {
agent { docker { image 'node:12' } }
stages {
stage('deploy-staging') {
when { branch 'staging' }
steps {
withCredentials([string(credentialsId: 'GITHUB_HOST_KEY', variable: 'GITHUB_HOST_KEY')]) {
sh 'mkdir ~/.ssh && echo "$GITHUB_HOST_KEY" >> ~/.ssh/known_hosts'
}
sshagent(['my-credentials-id']) {
sh 'git push joe#repo:project'
}
}
}
}
}
This ensures you're using a "trusted" host key.
TLDR: how to ssh a different machine and where to store ssh credentials on Jenkins pipeline (using ssh / SSHAgent plugin /etc...) ?
The Problem: In Jenkins pipeline I need a remote ssh to target machine.
My old approach was to use "Execute shell scripts on remote host using ssh". I would like to specify both username and password.
I've read that the groovy approach shoud be something like
sshagent(['RemoteCredentials']) {
sh 'ssh -o StrictHostKeyChecking=no -l remoteusername remotetarget uname -a'
}
RemoteCredentials: it is the private key with passphrase
Is there a way to make ssh with username/password remote credentials? The sshagent does not support username/password auth
Riccardo
So unfortunately, you're right.
It looks like the ssh-agent-plugin only supports stored user,passphrase,public key credentials added through the Credentials Management area in Jenkins. See this unit test that verifies that ssh-agent is working correctly based around a public key. It's unlikely that there is untested functionality in the plugin to support user+password auth.
If you can, make the switch to Public Key based authentication. If for some reason you can't switch ... you COULD install sshpass on your Jenkins box, but this is generally considered bad practice.
node {
stage 'Does sshpass work?'
sh 'sshpass -p \'password\' ssh user#host "ls; hostname; whois google.com;"'
}
Stepping up your game running ssh tasks on Jenkins agents, will make your servers more secure. Jenkins is an attack-vector when you run ssh tasks like Ansible, and being in control over releases is desirable.
Improving /etc/sshd_config will stop a lot of probes by hackers.
PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin no
Move over from DSA or RSA key-pairs to the more secure ed25519 elliptic curve
cryptography with brute-force protection on the private key-file.
ssh-keygen -t ed25519 -o -a 300 -C Jenkins
Because the private key needs hundreds of rounds, each use will take tens of seconds,
that is prefect for a build agent where developers might snoop around.
I rather not store the private key as a credential in Jenkins, because Jenkins is a pluggable code execution engine, instead I store only the passphrase as a secret text credential (named mySecretText in the example). I have dedicated agents for environments, each with they own ~/.ssh/id_ed25519 key file with limited blast radius. The passphrase is used to start an ssh-agent, more specifically to load the key per session.
The Jenkinsfile below allows the use of the key-pair to push a tag to git, but there is no plaintext on disk. This implementation was chosen because the ssh-agent plugin did not allow ssh-add.
node {
println("Checkout repository...")
checkout scm
}
pipeline {
agent {
label "linux"
}
options {
disableConcurrentBuilds()
}
stages {
stage('PushTheTag') {
steps {
script {
withCredentials([string(credentialsId: 'mySecretText', variable: 'SSH_PASSPHRASE')]) {
sh "echo 'echo \$SSH_PASSPHRASE' > ~/.ssh/tmp && chmod 700 ~/.ssh/tmp"
sh "eval `ssh-agent -s` && cat ~/.ssh/id_ed25519|DISPLAY=None SSH_ASKPASS=~/.ssh/tmp ssh-add - && git tag ${env.BUILD_NUMBER} && git push --tags; kill -9 \$SSH_AGENT_PID"
}
}
}
}
}
}
Solution: You want to ssh some machine in your pipeline. I want to offer a different approach, that is more secured (I want to be able managing my ssh credentials on the machine itself) . Make sure that your ssh keys is pre built in your infra under ~/.ssh/id_rsa ( can be preconfigured through ansible/chef/puppet or just use some image snapshot with your aws/gcp/azure cloud environment ). your pipeline should load the ssh private key credentials in the machine and connect to the node (with the public key inside).
_node(with private key) ---> testing/staging/production_node(with public key)
When you use it? Use case for example - you want to run/stop some process in another node, or deploy application on instances x and y in your pipeline
Simple example - Point to Point ( node -> destination_node ) would be:
def ip-address=<some-ip-address>
sh """#!/bin/bash
eval "\$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
ssh -o StrictHostKeyChecking=no ${ip-address} << 'EOF'
echo 'im in ...'
"""
Complex example - Bastion using bastion cloud architecture ( node -> vpc-endpoint -> destination-node) would be:
def vpc-endpoint-gw-ip=<some-ip-address>
def subnet-ip=<some-subnet-address>
sh """#!/bin/bash
eval "\$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
ssh -At -o StrictHostKeyChecking=no ${vpc-endpoint-gw-ip} << 'EOF'
ssh -o StrictHostKeyChecking=no ${subnet-ip} << 'EOF2'
echo 'im in ...'
"""