SSH from jenkins to same host - jenkins

I have a jenkins instance running on my raspberry pi 3 and i also have my (simple) apache webserver running on the same raspberry pi.
I've got a pipeline from jenkins to fetch a git repo, build it and put (via scp) the build files on my webserver.
I have a ssh private/public key setup, but it's a bit stupid (?) to have an ssh key when the jenkins is hosted on the same 'machine' with the same ip address no?
Anyway, on my raspberry pi i have setup the autorized keys file and the known host file with the public key on it, and i've added the private key to jenkins via the ssh-agent plugin.
Here you have my jenkinsfile thats being used by jenkins to define my pipeline:
node{
stage('Checkout') {
checkout scm
}
stage('install') {
nodejs(nodeJSInstallationName: 'nodeJS10.5.0') {
sh "npm install"
}
}
stage('build'){
nodejs(nodeJSInstallationName: 'nodeJS10.5.0') {
sh "npm run build"
}
}
stage('connect ssh and remove files') {
sshagent (credentials: ["0527982f-7794-45d0-99b0-135c868c5b36"]) {
sh "ssh pi#123.456.789.123 -p 330 rm -rf /var/www/html/*"
}
}
stage('upload new files'){
sshagent (credentials: ["0527982f-7794-45d0-99b0-135c868c5b36"]) {
sh "scp -P 330 -r ./build/* pi#123.456.789.123:/var/www/html"
}
}
}
Here is the output from the second to last job that is failing:
[Pipeline] }
[Pipeline] // nodejs
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (connect ssh and remove files)
[Pipeline] sh
[Deploy_To_Production] Running shell script
+ ssh pi#123.456.789.123 -p 330 rm -rf /var/www/html/asset-manifest.json /var/www/html/css /var/www/html/favicon.ico /var/www/html/fonts /var/www/html/images /var/www/html/index.html /var/www/html/manifest.json /var/www/html/service-worker.js /var/www/html/static /var/www/html/vendor
Host key verification failed.
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 255
Finished: FAILURE
Note: I've changed my IP address and my ssh port for security reasons.
Manually i can ssh to my raspberry pi and i can execute the commands manually from my laptop (both from same and other domain works).
I've also port forwarded the local ip so that i connect to it via SSH when i'm not home.
I guess I'm doing something wrong with the SSH keys etc, but i'm no expert whatsoever!
Can anyone help?

I need 4 more reputation point to comment, so I must write answer:)
Try use -v to debug ssh connection:
stage('connect ssh and remove files') {
sshagent (credentials: ["0527982f-7794-45d0-99b0-135c868c5b36"]) {
sh "ssh -v pi#123.456.789.123 -p 330 rm -rf /var/www/html/*"
}
}
In another hand
Host key verification failed means that the host key of the remote host was changed or you don't have the host key of the remote host. So at first try just ssh -v pi#123.456.789.123 as Jenkins user, from Jenkins host.

The issue was indeed that the host key verification was failing. I think this was due to not trusting the host.
But the real issue was pointed out by #3sky (see other answer). I needed to login as the jenkins user and try to ssh to my raspberry pi (which are both on the same machine).
So these are the steps i did:
Login via ssh to my raspberry pi
ssh -v pi#123.456.789.123 -p 330
Then I switched user to the jenkins user. After some google search i've found out how
sudo su -s /bin/bash jenkins
Then i ssh again to my own machine (where i already was ssh'ed in), so that i get the pop-up for thrusting this host once and for all!
ssh -v pi#123.456.789.123 -p 330
This solved my issue! Big thanks to 3sky for helping out!

Related

Running ssh-agent within docker on jenkins doesnt work

I am trying to use a container within my jenkins pipeline, however I cant get ssh-agent to work inside it. I am on v1.19 of the plugin, when I run the below code I get
Host key verification failed. fatal: Could not read from remote
repository.
Please make sure you have the correct access rights and the repository
exists.
However if I run the code from outside the image it works perfect, proving that the user has the correct permissions.
node('nodeName'){
cleanWs()
ws("short"){
withDockerRegistry([credentialsId: 'token', url: "https://private.repo.com"]) {
docker.image("img:1.0.0").inside("-u root:root --network=host") {
sshagent(credentials: ["bitbucket_token"]) {
sh "mkdir ~/.ssh"
sh 'ssh-keyscan bitbucket.company.com >> ~/.ssh/known_hosts'
sh 'git clone ssh://git#bitbucket.company.com:PORT/repo.git'
}
}
}
}
}
Here is the output:
[Pipeline] sshagent
[ssh-agent] Using credentials jenkins (bitbucket_token)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ docker exec abcdef123456 ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-qwertyu/agent.15
SSH_AGENT_PID=22
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/short#tmp/private_key_8675309.key (/home/jenkins/short#tmp/private_key_8675309.key)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ mkdir /root/.ssh
[Pipeline] sh
+ ssh-keyscan bitbucket.company.com
# bitbucket.company.com:22 SSH-2.0-OpenSSH_6.6.1
# bitbucket.company.com:22 SSH-2.0-OpenSSH_6.6.1
# bitbucket.company.com:22 SSH-2.0-OpenSSH_6.6.1
[Pipeline] sh
+ git clone ssh://git#bitbucket.company.com:PORT/repo.git
Cloning into 'repo'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
[Pipeline] }
$ docker exec --env ******** --env ******** abcdef123456 ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 22 killed;
[ssh-agent] Stopped.
[Pipeline] // sshagent
Im completely stumped by this

Jenkins ssh agent plugin is getting stacked, No error, no timeout issue

I installed jenkins ssh agent plugin. I created ssh private key on the linux server(using ssh-keygen -t rsa command) I am trying to connect. Then under jenkins credintials added SSH Username with private key with all required fields. In jenkinsfile added simple command to run over ssh:
pipeline {
agent any
stages {
stage('---doingsomething---') {
steps {
sshagent (credentials: ['jbowner-195']) {
sh 'ssh -o StrictHostKeyChecking=no -l jbowner 10.10.23.195 uname -a'
}
}
}
}
}
When I press build button process is starting and never ending. No error, no timeout issue.
Here is piece of output on which jenkins stacks
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (---echoing---)
[Pipeline] sshagent
[ssh-agent] Using credentials jbowner (jbowner 10.10.23.195)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-XSQPEUHOqZQR/agent.10226
SSH_AGENT_PID=10229
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/.jenkins/workspace/Eformgenerator-Prod#tmp/private_key_5151715321960722060.key (/home/jenkins/.jenkins/workspace/Eformgenerator-Prod#tmp/private_key_5151715321960722060.key)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ ssh -o StrictHostKeyChecking=no -l jbowner 10.10.23.195 uname -a
any ideas?
Jenkins ssh plugin didn't work for me. My solution is
generate rsa keys on source machine using ssh-keygen -t rsa
ssh-copy-id username#destination_ip. Then enter destination password. This will add as destination ip as a known host and adds source key on the destination machine as a authorized key.
then instead of using jenkins ssh agent I used standard ssh command like this.
pipeline {
agent any
stages {
stage('---echoing---') {
steps {
sh 'ssh -o StrictHostKeyChecking=no jbowner#10.10.23.195 uptime'
}
}
}
}
This is working because servers have been trusting each other using ssh key

Jenkins Kubernetes Plugin - sshagent to git clone terraform module

I am attempting to use sshagent in Jenkins to pass my private key into the terraform container to allow terraform to source a module in a private repo.
stage('TF Plan') {
steps {
container('terraform') {
sshagent (credentials: ['6c92998a-bbc4-4f27-b925-b50c861ef113']){
sh 'ssh-add -L'
sh 'terraform init'
sh 'terraform plan -out myplan'
}
}
}
}
When running the job it fails with the following output:
[ssh-agent] Using credentials (id_rsa_jenkins)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
Executing shell script inside container [terraform] of pod [gcp-tf-builder-h79rb-h5f3m]
Executing command: "ssh-agent"
exit
SSH_AUTH_SOCK=/tmp/ssh-2xAa2W04uQV6/agent.20; export SSH_AUTH_SOCK;
SSH_AGENT_PID=21; export SSH_AGENT_PID;
echo Agent pid 21;
SSH_AUTH_SOCK=/tmp/ssh-2xAa2W04uQV6/agent.20
SSH_AGENT_PID=21
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/agent/workspace/demo#tmp/private_key_2729797926.key (user#workstation.local)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ ssh-add -L
ssh-rsa REDACTED user#workstation.local
[Pipeline] sh
+ terraform init
[0m[1mInitializing modules...[0m
- module.demo_proj
Getting source "git::ssh://git#bitbucket.org/company/terraform-module"
[31mError downloading modules: Error loading modules: error downloading 'ssh://git#bitbucket.org/company/deploy-kickstart-project': /usr/bin/git exited with 128: Cloning into '.terraform/modules/e11a22f40c64344133a98e564940d3e4'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
[0m[0m
[Pipeline] }
Executing shell script inside container [terraform] of pod [gcp-tf-builder-h79rb-h5f3m]
Executing command: "ssh-agent" "-k"
exit
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 21 killed;
[ssh-agent] Stopped.
I've triple checked and I am for sure using the correct key pair. I am able to git clone locally from my mac to the repo with no issues.
An important note is that this Jenkins deployment is running within Kubernetes. The Master stays up and uses the Kubernetes plugin to spawn agents.
What does the Host key verification failed. error mean? From my research it can be due to known_hosts not properly being set. Is ssh-agent responsible for that?
Turns out it was an issue with known_hosts not being set. As a workaround we added this to our jenkinsfile
environment {
GIT_SSH_COMMAND = "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
}

Bind Volumes to Docker container in a pipeline job

So, I have this pipeline job that builds completely inside a Docker container. The Docker image used is pulled from a local repository before build and has almost all the dependencies required to run my project.
The problem is I need an option to define volumes to bind mound from Host to container so that I can perform some analysis using a tool available on my Host system but not in the container.
Is there a way to do this from inside a Jenkinsfile (Pipeline script)?
I'm not fully clear if this is what you mean. But if it isn't. Let me know and I'll try to figure out.
What I understand of mounting from host to container is mounting the content of the Jenkins Workspace inside the container.
For example in this pipeline:
pipeline {
agent { node { label 'xxx' } }
options {
buildDiscarder(logRotator(numToKeepStr: '3', artifactNumToKeepStr: '1'))
}
stages {
stage('add file') {
steps {
sh 'touch myfile.txt'
sh 'ls'
}
}
stage('Deploy') {
agent {
docker {
image 'lvthillo/aws-cli'
args '-v $WORKSPACE:/project'
reuseNode true
}
}
steps {
sh 'ls'
sh 'aws --version'
}
}
}
post {
always {
cleanWs()
}
}
}
In the first stage I just add a file to the workspace. just in Jenkins. Nothing with Docker.
In the second stage I start a docker container which contains the aws CLI (this is not installed on our jenkins slaves). We will start the container and mount the workspace inside the /project folder of my container. Now I can execute AWS CLI command's and I have access to the text file. In a next stage (not in the pipeline) you can use the file again in a different container or jenkins slave itself.
Output:
[Pipeline] {
[Pipeline] stage
[Pipeline] { (add file)
[Pipeline] sh
[test] Running shell script
+ touch myfile.txt
[Pipeline] sh
[test] Running shell script
+ ls
myfile.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Deploy)
[Pipeline] getContext
[Pipeline] sh
[test] Running shell script
+ docker inspect -f . lvthillo/aws-cli
.
[Pipeline] withDockerContainer
FJ Arch Slave 7 does not seem to be running inside a container
$ docker run -t -d -u 201:201 -v $WORKSPACE:/project -w ... lvthillo/aws-cli cat
$ docker top xx -eo pid,comm
[Pipeline] {
[Pipeline] sh
[test] Running shell script
+ ls
myfile.txt
[Pipeline] sh
[test] Running shell script
+ aws --version
aws-cli/1.14.57 Python/2.7.14 Linux/4.9.78-1-lts botocore/1.9.10
[Pipeline] }
$ docker stop --time=1 3652bf94e933cbc888def1eeaf89e1cf24554408f9e4421fabfd660012a53365
$ docker rm -f 3652bf94e933cbc888def1eeaf89e1cf24554408f9e4421fabfd660012a53365
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
In your case you can mount your data in the container. Perform the stuff and in a later stage you can do your analysis on your code on your jenkins slave itself (without docker)
Suppose you are under Linux, run the following code
docker run -it --rm -v /local_dir:/image_root_dir/mount_dir image_name
Here is some detail:
-it: interactive terminal
--rm: remove container after exit the container
-v: volume or say mount your local directory to a volume.
Since the mount function will 'cover' the directory in your image, your should alway make a new directory under your images root directory.
Visit Use bind mounts to get more information.
ps:
run
sudo -s
and tpye the password before you run docker, that saves you a lot of time, since you don't have to type sudo in front of docker every time you run docker.
ps2:
suppose you have an image with a long name and the image ID is 5ed6274db6ce, you can simply run at least the first three digits, or more
docker run [options] 5ed
if you have more image have the same first three digits, you can use four or more.
For example, you have following two images
REPOSITORY IMAGE ID
My_Image_with_very_long_name 5ed6274db6ce
My_Image_with_very_long_name2 5edc819f315e
you can simply run
docker run [options] 5ed6
to run the image My_Image_with_very_long_name.

Jenkins Docker Pipelining inside Docker

I'm following along with this tutorial:
https://www.linkedin.com/pulse/building-docker-pipeline-cloudbees-jenkins-jay-johnson
I'm running Jenkins on Docker 17:
docker run -d -p 8080:8080 -p 50000:50000 --name jenkins jenkins
I followed the instructions and replaced Jay's credentials with my own. I added my creds to Global and then renamed the creds in the pipeline script. When I attempt the build, though I'm getting the following error:
Proceeding
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withDockerRegistry
Wrote authentication to /var/jenkins_home/.dockercfg
[Pipeline] {
[Pipeline] stage (Building)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Building
Proceeding
[Pipeline] sh
[alfred-master] Running shell script
+ docker build -t jayjohnson/django-slack-sphinx:testing django
/var/jenkins_home/workspace/alfred-master#tmp/durable-713ce0d7/script.sh: 2: /var/jenkins_home/workspace/alfred-master#tmp/durable-713ce0d7/script.sh: docker: not found
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 127
Finished: FAILURE
I'm assuming this is looking for the docker binary.
How can I build a docker image from a repo from inside a Docker container?
The issue is here:
/var/jenkins_home/workspace/alfred-master#tmp/durable-713ce0d7/script.sh: 2: /var/jenkins_home/workspace/alfred-master#tmp/durable-713ce0d7/script.sh: docker: not found
I'm assuming your build is running on the master instance, which is just a basic installation of Jenkins - no extra tools.
You'll want to run an agent slave and connect it to your master - this agent should ensure it has Docker installed, and then you will be able to run those commands.
You can either set this up yourself; or use an open source option - Currently in my own setup I'm using this image which has everything I need (Well, personally - I've forked it and added some of my own tools as well).

Resources