Jenkins Pipeline - connect to Docker host using SSH credentials - docker

Question

We would like to use Jenkins Pipelines to work with AWS ECR images on a remote host that has Docker installed, but does not (and will not) expose the Docker socket over port 2376.
A couple of simpler options include:
using the existing Jenkins SSH/scripts
using the pipeline ssh-agent and running commands in-line there
However, because the declarative docker plugin seems to have everything needed it would be cleaner to use this since tags, etc., will all align with other parts of the pipeline.
All examples on the internet show
docker.withServer("tcp://X:2376","credentialsId") {...}
However from configuring the Jenkins Cloud Config -> Docker Templates it seems ssh is provided so we tried the following:
stages {
stage('Deploy to Remote Host') {
steps {
script {
docker.withServer("ssh://ec2-x-x-x-x.mars-1.compute.amazonaws.com:22", "ssh-credentials-id") {
docker.withRegistry("https://1234567890.dkr.ecr.mars-1.amazonaws.com", "ecr:mars-1:ecr-credentials-id") {
docker.pull('my-image:latest')
}
}
}
}
}
}
Unfortunately, we get the following connection error:
error during connect: Post http://docker/v1.40/auth: command [ssh -p 22 ec2-x-x-x-x.mars-1.compute.amazonaws.com -- docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Host key verification failed.
We have Docker v19 on the server, and the ssh key is fine using ssh-agent.
Any ideas about what we need to do to get this working?

Answer 1

I solved this by adding SSH commands (invoked on the agent) to the remote/target host, then invoking 'docker pull...', 'docker run...' etc using the shell.... however doing this in Jenkins Pipeline is fragile - the ssh commands turn into scripts, the credentials lookups are fairly complex & messy to look at and the overall outcome, while presenting a nice pipeline in Blue Ocean, is going to become difficult to code/maintain.
So following the advice that 'doing everything in Jenkins Pipeline is an anti-pattern' I broke the process into two pieces - building/pushing the image using Jenkins Pipeline + Docker Plugin which is documented and supported as a first class concern, and then invoking a Ansible 'build job' from the Pipeline passing dev/stage/live parameters, and letting Ansible take care of all of the other variables and secrets needed. This means the end result is similar in Blue Ocean, but the complex deployment logic is coded more cleanly in an appropriate tool designed to handle my use-case, and looping through environments/hosts is much cleaner in Ansible [or substitute your preferred deployment tool here].

Answer 2

To get rid of the "Host key verification failed" error I used this in my pipeline:
sh '''
[ -d ~/.ssh ] || mkdir ~/.ssh && chmod 0700 ~/.ssh && ssh-keyscan HOSTNAME >> ~/.ssh/known_hosts
'''