Running ssh-agent within docker on jenkins doesnt work - docker

I am trying to use a container within my jenkins pipeline, however I cant get ssh-agent to work inside it. I am on v1.19 of the plugin, when I run the below code I get
Host key verification failed. fatal: Could not read from remote
repository.
Please make sure you have the correct access rights and the repository
exists.
However if I run the code from outside the image it works perfect, proving that the user has the correct permissions.
node('nodeName'){
cleanWs()
ws("short"){
withDockerRegistry([credentialsId: 'token', url: "https://private.repo.com"]) {
docker.image("img:1.0.0").inside("-u root:root --network=host") {
sshagent(credentials: ["bitbucket_token"]) {
sh "mkdir ~/.ssh"
sh 'ssh-keyscan bitbucket.company.com >> ~/.ssh/known_hosts'
sh 'git clone ssh://git#bitbucket.company.com:PORT/repo.git'
}
}
}
}
}
Here is the output:
[Pipeline] sshagent
[ssh-agent] Using credentials jenkins (bitbucket_token)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ docker exec abcdef123456 ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-qwertyu/agent.15
SSH_AGENT_PID=22
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/short#tmp/private_key_8675309.key (/home/jenkins/short#tmp/private_key_8675309.key)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ mkdir /root/.ssh
[Pipeline] sh
+ ssh-keyscan bitbucket.company.com
# bitbucket.company.com:22 SSH-2.0-OpenSSH_6.6.1
# bitbucket.company.com:22 SSH-2.0-OpenSSH_6.6.1
# bitbucket.company.com:22 SSH-2.0-OpenSSH_6.6.1
[Pipeline] sh
+ git clone ssh://git#bitbucket.company.com:PORT/repo.git
Cloning into 'repo'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
[Pipeline] }
$ docker exec --env ******** --env ******** abcdef123456 ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 22 killed;
[ssh-agent] Stopped.
[Pipeline] // sshagent
Im completely stumped by this

Related

Jenkins ssh agent plugin is getting stacked, No error, no timeout issue

I installed jenkins ssh agent plugin. I created ssh private key on the linux server(using ssh-keygen -t rsa command) I am trying to connect. Then under jenkins credintials added SSH Username with private key with all required fields. In jenkinsfile added simple command to run over ssh:
pipeline {
agent any
stages {
stage('---doingsomething---') {
steps {
sshagent (credentials: ['jbowner-195']) {
sh 'ssh -o StrictHostKeyChecking=no -l jbowner 10.10.23.195 uname -a'
}
}
}
}
}
When I press build button process is starting and never ending. No error, no timeout issue.
Here is piece of output on which jenkins stacks
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (---echoing---)
[Pipeline] sshagent
[ssh-agent] Using credentials jbowner (jbowner 10.10.23.195)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-XSQPEUHOqZQR/agent.10226
SSH_AGENT_PID=10229
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/.jenkins/workspace/Eformgenerator-Prod#tmp/private_key_5151715321960722060.key (/home/jenkins/.jenkins/workspace/Eformgenerator-Prod#tmp/private_key_5151715321960722060.key)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ ssh -o StrictHostKeyChecking=no -l jbowner 10.10.23.195 uname -a
any ideas?
Jenkins ssh plugin didn't work for me. My solution is
generate rsa keys on source machine using ssh-keygen -t rsa
ssh-copy-id username#destination_ip. Then enter destination password. This will add as destination ip as a known host and adds source key on the destination machine as a authorized key.
then instead of using jenkins ssh agent I used standard ssh command like this.
pipeline {
agent any
stages {
stage('---echoing---') {
steps {
sh 'ssh -o StrictHostKeyChecking=no jbowner#10.10.23.195 uptime'
}
}
}
}
This is working because servers have been trusting each other using ssh key

Jenkins Kubernetes Plugin - sshagent to git clone terraform module

I am attempting to use sshagent in Jenkins to pass my private key into the terraform container to allow terraform to source a module in a private repo.
stage('TF Plan') {
steps {
container('terraform') {
sshagent (credentials: ['6c92998a-bbc4-4f27-b925-b50c861ef113']){
sh 'ssh-add -L'
sh 'terraform init'
sh 'terraform plan -out myplan'
}
}
}
}
When running the job it fails with the following output:
[ssh-agent] Using credentials (id_rsa_jenkins)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
Executing shell script inside container [terraform] of pod [gcp-tf-builder-h79rb-h5f3m]
Executing command: "ssh-agent"
exit
SSH_AUTH_SOCK=/tmp/ssh-2xAa2W04uQV6/agent.20; export SSH_AUTH_SOCK;
SSH_AGENT_PID=21; export SSH_AGENT_PID;
echo Agent pid 21;
SSH_AUTH_SOCK=/tmp/ssh-2xAa2W04uQV6/agent.20
SSH_AGENT_PID=21
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/agent/workspace/demo#tmp/private_key_2729797926.key (user#workstation.local)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ ssh-add -L
ssh-rsa REDACTED user#workstation.local
[Pipeline] sh
+ terraform init
[0m[1mInitializing modules...[0m
- module.demo_proj
Getting source "git::ssh://git#bitbucket.org/company/terraform-module"
[31mError downloading modules: Error loading modules: error downloading 'ssh://git#bitbucket.org/company/deploy-kickstart-project': /usr/bin/git exited with 128: Cloning into '.terraform/modules/e11a22f40c64344133a98e564940d3e4'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
[0m[0m
[Pipeline] }
Executing shell script inside container [terraform] of pod [gcp-tf-builder-h79rb-h5f3m]
Executing command: "ssh-agent" "-k"
exit
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 21 killed;
[ssh-agent] Stopped.
I've triple checked and I am for sure using the correct key pair. I am able to git clone locally from my mac to the repo with no issues.
An important note is that this Jenkins deployment is running within Kubernetes. The Master stays up and uses the Kubernetes plugin to spawn agents.
What does the Host key verification failed. error mean? From my research it can be due to known_hosts not properly being set. Is ssh-agent responsible for that?
Turns out it was an issue with known_hosts not being set. As a workaround we added this to our jenkinsfile
environment {
GIT_SSH_COMMAND = "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
}

Jenkins pipeline not correctly using sshagent credentials

I have this code snippet that has to use a custom private key from the Jenkins credentials using the ssh-agent-plugin.
This doesn't seem to work, but it also doesn't print a very useful output.
Any ideas how to debug this?
stage('Test Git') {
steps {
sshagent(credentials : ['denpal']) {
sh 'git commit --allow-empty -m "test withCredentials"'
sh 'git push origin feature/Jenkinsfile'
}
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Test Git)
[Pipeline] sshagent
[ssh-agent] Using credentials git (denpal)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-WEsIsQvX4CFc/agent.12163
SSH_AGENT_PID=12166
Running ssh-add (command line suppressed)
[Pipeline] // sshagent
[Pipeline] }
I had the same problem trying to push code to my repo from Jenkins.
I found the solution here: https://www.reddit.com/r/docker/comments/b8lmc4/jenkins_pipeline_not_correctly_using_sshagent/
I replaced the sshagent code block with:
withCredentials([sshUserPrivateKey(credentialsId: 'myCredentials', keyFileVariable: 'KEY_FILE')]) {
sh "eval `ssh-agent -s` && ssh-add ${KEY_FILE} && ssh-add -L && git push -u origin develop"
}
It worked for me.

SSH from jenkins to same host

I have a jenkins instance running on my raspberry pi 3 and i also have my (simple) apache webserver running on the same raspberry pi.
I've got a pipeline from jenkins to fetch a git repo, build it and put (via scp) the build files on my webserver.
I have a ssh private/public key setup, but it's a bit stupid (?) to have an ssh key when the jenkins is hosted on the same 'machine' with the same ip address no?
Anyway, on my raspberry pi i have setup the autorized keys file and the known host file with the public key on it, and i've added the private key to jenkins via the ssh-agent plugin.
Here you have my jenkinsfile thats being used by jenkins to define my pipeline:
node{
stage('Checkout') {
checkout scm
}
stage('install') {
nodejs(nodeJSInstallationName: 'nodeJS10.5.0') {
sh "npm install"
}
}
stage('build'){
nodejs(nodeJSInstallationName: 'nodeJS10.5.0') {
sh "npm run build"
}
}
stage('connect ssh and remove files') {
sshagent (credentials: ["0527982f-7794-45d0-99b0-135c868c5b36"]) {
sh "ssh pi#123.456.789.123 -p 330 rm -rf /var/www/html/*"
}
}
stage('upload new files'){
sshagent (credentials: ["0527982f-7794-45d0-99b0-135c868c5b36"]) {
sh "scp -P 330 -r ./build/* pi#123.456.789.123:/var/www/html"
}
}
}
Here is the output from the second to last job that is failing:
[Pipeline] }
[Pipeline] // nodejs
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (connect ssh and remove files)
[Pipeline] sh
[Deploy_To_Production] Running shell script
+ ssh pi#123.456.789.123 -p 330 rm -rf /var/www/html/asset-manifest.json /var/www/html/css /var/www/html/favicon.ico /var/www/html/fonts /var/www/html/images /var/www/html/index.html /var/www/html/manifest.json /var/www/html/service-worker.js /var/www/html/static /var/www/html/vendor
Host key verification failed.
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 255
Finished: FAILURE
Note: I've changed my IP address and my ssh port for security reasons.
Manually i can ssh to my raspberry pi and i can execute the commands manually from my laptop (both from same and other domain works).
I've also port forwarded the local ip so that i connect to it via SSH when i'm not home.
I guess I'm doing something wrong with the SSH keys etc, but i'm no expert whatsoever!
Can anyone help?
I need 4 more reputation point to comment, so I must write answer:)
Try use -v to debug ssh connection:
stage('connect ssh and remove files') {
sshagent (credentials: ["0527982f-7794-45d0-99b0-135c868c5b36"]) {
sh "ssh -v pi#123.456.789.123 -p 330 rm -rf /var/www/html/*"
}
}
In another hand
Host key verification failed means that the host key of the remote host was changed or you don't have the host key of the remote host. So at first try just ssh -v pi#123.456.789.123 as Jenkins user, from Jenkins host.
The issue was indeed that the host key verification was failing. I think this was due to not trusting the host.
But the real issue was pointed out by #3sky (see other answer). I needed to login as the jenkins user and try to ssh to my raspberry pi (which are both on the same machine).
So these are the steps i did:
Login via ssh to my raspberry pi
ssh -v pi#123.456.789.123 -p 330
Then I switched user to the jenkins user. After some google search i've found out how
sudo su -s /bin/bash jenkins
Then i ssh again to my own machine (where i already was ssh'ed in), so that i get the pop-up for thrusting this host once and for all!
ssh -v pi#123.456.789.123 -p 330
This solved my issue! Big thanks to 3sky for helping out!

Jenkins ansible plugin can't find executable

I'm running a Jenkins on Amazon EC2--the master in a Docker container and an agent on a separate box. My playbook executes an Ansible script, using the Jenkins Ansible plugin.
I had to install a new version of Ansible on the agent. I installed Ansible from git using the Running from Source instructions, and installed to /home/ec2-user/ansible. If I ssh to the agent and run which ansible I get ~/ansible/bin/ansible as expected. I entered /home/ec2-user/ansible/bin in the 'Ansible executables directory' for my new install, at the Manage Jenkins > Global Tool Configuration page.
When I run my Jenkins pipeline, however, I get this:
Running on docker-agent-1 in /home/ec2-user/jenkins/workspace/planet-healthcare-pipeline
[Pipeline] {
[Pipeline] pwd
[Pipeline] stage
[Pipeline] { (Download source and capture commit ID)
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ which ansible
which: no ansible in (/usr/local/bin:/bin:/usr/bin)
It says it's running on docker-agent-1 (which is the name of my agent), and I can see Ansible if I ssh there. Why can't Jenkins find the ansible executable?
UPDATE: After adding PATH as an environment variable, it can find Ansible, but now something else breaks. Here's the new output:
Running on docker-agent-1 in /home/ec2-user/jenkins/workspace/planet-healthcare-pipeline
[Pipeline] {
[Pipeline] pwd
[Pipeline] stage
[Pipeline] { (Download source and capture commit ID)
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ which ansible
/home/ec2-user/ansible/bin/ansible
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ ansible --version
ansible 2.2.0 (devel 1975a545bd) last updated 2016/09/20 16:19:06 (GMT +000)
lib/ansible/modules/core: (detached HEAD 70d4ff8e38) last updated 2016/09/20 16:19:08 (GMT +000)
lib/ansible/modules/extras: (detached HEAD db7a3f48e1) last updated 2016/09/20 16:19:09 (GMT +000)
config file = /home/ec2-user/jenkins/workspace/planet-healthcare-pipeline/ansible.cfg
configured module search path = Default w/o overrides
[Pipeline] git
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url git#bitbucket.org:planetgroup/planethealthcareportal.git # timeout=10
Fetching upstream changes from git#bitbucket.org:planetgroup/planethealthcareportal.git
> git --version # timeout=10
using GIT_SSH to set credentials Deployment key for Planet Healthcare Portal
> git fetch --tags --progress git#bitbucket.org:planetgroup/planethealthcareportal.git +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/develop^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/develop^{commit} # timeout=10
Checking out Revision e69608a15c9d433e2a22824c7e607048332a4160 (refs/remotes/origin/develop)
> git config core.sparsecheckout # timeout=10
> git checkout -f e69608a15c9d433e2a22824c7e607048332a4160
> git branch -a -v --no-abbrev # timeout=10
> git branch -D develop # timeout=10
> git checkout -b develop e69608a15c9d433e2a22824c7e607048332a4160
> git rev-list e69608a15c9d433e2a22824c7e607048332a4160 # timeout=10
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ git rev-parse --verify HEAD
[Pipeline] readFile
[Pipeline] echo
Current commit ID: e69608a
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Copy application.yml to environment)
[Pipeline] withCredentials
[Pipeline] {
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ sudo cp **** config/application.yml
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build image)
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ docker build -t planethealthcare/portal_app .
Sending build context to Docker daemon 557.1 kB
Sending build context to Docker daemon 1.114 MB
Sending build context to Docker daemon 1.671 MB
Sending build context to Docker daemon 2.228 MB
Sending build context to Docker daemon 2.785 MB
Sending build context to Docker daemon 3.342 MB
Sending build context to Docker daemon 3.398 MB
Step 1 : FROM ruby:2.3
---> 7b66156f376c
Step 2 : MAINTAINER David Ham <dham#uxfactory.com>
---> Using cache
---> 47f6f577f049
Step 3 : RUN apt-get update && apt-get install -y build-essential curl gstreamer1.0-plugins-base gstreamer1.0-tools gstreamer1.0-x libqt5webkit5-dev qt5-default xvfb && apt-get clean && rm -rf /var/lib/apt/lists/* && mkdir -p /app
---> Using cache
---> 38c1313e574d
Step 4 : WORKDIR /app
---> Using cache
---> 75a023d99fce
Step 5 : COPY Gemfile Gemfile.lock ./
---> Using cache
---> c39c81496a6b
Step 6 : ENV QMAKE /usr/bin/qmake
---> Using cache
---> 3226bf5f4e63
Step 7 : RUN bundle install --retry 20
---> Using cache
---> 91cb9908d53a
Step 8 : COPY . ./
---> 7330a8f5ba7c
Removing intermediate container bd55b7deddaf
Step 9 : EXPOSE 3000
---> Running in 76e6418e2b3f
---> 81427ffb31f5
Removing intermediate container 76e6418e2b3f
Step 10 : CMD bundle exec rails server
---> Running in c2a90c3c59f6
---> 15ab02b3ab8d
Removing intermediate container c2a90c3c59f6
Successfully built 15ab02b3ab8d
[Pipeline] dockerFingerprintFrom
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Run test suite)
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=phc_portal_test postgres:9.5
[Pipeline] dockerFingerprintRun
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ docker inspect -f . planethealthcare/portal_app
.
[Pipeline] withDockerContainer
$ docker run -t -d -u 500:500 --link 85511ce90ce11c24818ae63bbbf7ab47745be7d96807d450b4adebd4c3196c5e:postgres -p 3000:3000 -e RAILS_ENV=test -w /home/ec2-user/jenkins/workspace/planet-healthcare-pipeline -v /home/ec2-user/jenkins/workspace/planet-healthcare-pipeline:/home/ec2-user/jenkins/workspace/planet-healthcare-pipeline:rw -v /home/ec2-user/jenkins/workspace/planet-healthcare-pipeline#tmp:/home/ec2-user/jenkins/workspace/planet-healthcare-pipeline#tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat planethealthcare/portal_app
[Pipeline] {
[Pipeline] echo
running tests...
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ rails db:migrate
/home/ec2-user/jenkins/workspace/planet-healthcare-pipeline#tmp/durable-32785ba4/script.sh: 2: /home/ec2-user/jenkins/workspace/planet-healthcare-pipeline#tmp/durable-32785ba4/script.sh: rails: not found
[Pipeline] }
$ docker stop 3acf37726ce1061d2e0f6e8d0cec882c707b42e710916636b17aaece4f516f2d
$ docker rm -f 3acf37726ce1061d2e0f6e8d0cec882c707b42e710916636b17aaece4f516f2d
[Pipeline] // withDockerContainer
[Pipeline] sh
[planet-healthcare-pipeline] Running shell script
+ docker stop 85511ce90ce11c24818ae63bbbf7ab47745be7d96807d450b4adebd4c3196c5e
85511ce90ce11c24818ae63bbbf7ab47745be7d96807d450b4adebd4c3196c5e
+ docker rm -f 85511ce90ce11c24818ae63bbbf7ab47745be7d96807d450b4adebd4c3196c5e
85511ce90ce11c24818ae63bbbf7ab47745be7d96807d450b4adebd4c3196c5e
[Pipeline] }
[Pipeline] // stage
[Pipeline] mail
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 127
Finished: FAILURE
And here's the pipeline:
node('docker') {
currentBuild.result = "SUCCESS"
try{
def git_commit = ""
def workspace = pwd()
def APPLICATION_YML
def image
stage("Download source and capture commit ID") {
sh "which ansible"
sh "ansible --version"
// Download source
git branch: 'develop', credentialsId: 'b96345a1-543c-4ccd-9a86-deca7203625c', url: 'git#bitbucket.org:planetgroup/planethealthcareportal.git'
// Get the commit ID
sh 'git rev-parse --verify HEAD > GIT_COMMIT'
git_commit = readFile('GIT_COMMIT').take(7)
echo "Current commit ID: ${git_commit}"
}
stage("Copy application.yml to environment"){
// write the application.yml to a file
withCredentials([[$class: 'FileBinding', credentialsId: '67dbd2e7-008f-4463-89a6-9645060e8ec8', variable: 'APPLICATION_YML']]) {
sh "sudo cp ${env.APPLICATION_YML} config/application.yml"
}
}
stage("Build image"){
image = docker.build "planethealthcare/portal_app"
}
stage("Run test suite"){
// start postgres
def postgres95 = docker.image('postgres:9.5')
postgres95.withRun("-p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=phc_portal_test"){ postgres ->
image.inside("--link ${postgres.id}:postgres -p 3000:3000 -e RAILS_ENV=test") {
echo "running tests..."
sh "rails db:migrate"
sh "rspec --tag ~pending"
sh "cucumber"
}
}
}
stage("Push to ECR registry"){
docker.withRegistry('https://0000000000.dkr.ecr.us-east-1.amazonaws.com', 'ecr:dham'){
image.push "${git_commit}"
image.push 'latest'
}
}
stage("Deploy app"){
// run the playbook
ansiblePlaybook([
colorized: true,
credentialsId: 'planet-healthcare',
installation: 'ansible-2-2-0',
inventory: 'staging',
playbook: 'deploy.yml',
extras: "--extra-vars 'app_build_id=${git_commit}''"
])
}
}
catch(err) {
currentBuild.result = "FAILURE"
mail body: "project build error: ${err}\n\n\n ${currentBuild.description}" ,
subject: 'project build failed',
to: 'me#example.com'
throw err
}
}
It's failing in the "Run test suite" stage--it can't find rails to run rails db:migrate, even though I know it's in the container.
Why would setting PATH on the agent affect a script that happens inside a Docker container?
Do you execute which ansible in your script? It searches only defined PATHs.
And it seems /home/ec2-user/ansible/bin is not in /usr/local/bin:/bin:/usr/bin (from your output).
You may go to agent-node's settings in Jenkins and add PATH environment variable with $PATH:/home/ec2-user/ansible/bin value.

Resources