SSH from inside container docker to gcp instance in subnetwork - docker

I already add public key to authorized keys to target instance. But I still cannot ssh from inside container to the instance. I'm doing this because I use jenkins in container. But I can ssh from outside container to target instance using public key from host instance. Key from container and host instance is different.
[jenkins#4f5159051056 /]$ ssh jenkins#10.64.0.5 -v
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to 10.64.0.5 [10.64.0.5] port 22.
debug1: Connection established.
debug1: identity file /var/jenkins_home/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u3
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.64.0.5:22 as 'jenkins'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:2QOdKiyn5cehOBPIfQFaEo8WvZz6BJYq4fc7gNIIJ1g
The authenticity of host '10.64.0.5 (10.64.0.5)' can't be established.
ECDSA key fingerprint is SHA256:2QOdKiyn5cehOBPIfQFaEo8WvZz6BJYq4fc7gNIIJ1g.
ECDSA key fingerprint is MD5:13:03:df:e8:e5:4b:f7:4a:9b:65:82:c3:9d:80:11:f4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.64.0.5' (ECDSA) to the list of known hosts.
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/jenkins_home/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /var/jenkins_home/.ssh/id_dsa
debug1: Trying private key: /var/jenkins_home/.ssh/id_ecdsa
debug1: Trying private key: /var/jenkins_home/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

Did you check the following
Setting proper permissions:
chmod 700 /var/jenkins_home/.ssh
chmod 644 /var/jenkins_home/.ssh/authorized_keys
Owner should be Jenkins in your case
chown -R jenkins:jenkins /var/jenkins_home/*
Also you are checking on the client side using ssh -v, you need to check on the server side why the server is not accepting the key Permission denied (publickey).Try checking in the below location
sudo vi /var/log/secure
EDITED
If faced with the below issue
localhost sshd[36601]: User jenkins from xxx.xxx.xxx.xxx not allowed because not listed in AllowUsers
and with this
localhost sshd[36601]: input_userauth_request: invalid
user myuser [preauth]
add the user jenkins to the below file
vim /etc/ssh/sshd_config
and restart service
systemctl restart sshd.service
It should workk now :)

Related

How to use tensorboard within a docker container across a ssh tunnel?

I'm running a deep learning network on a remote GPU cluster and I'd like to use tensorboard to visualize my training and performance. The problem is I'm really confused on how to expose the port tensorboard uses to my local computer. Due to security and account permission reasons, I have to connect to this server through another server. I also have the complication of training my network within a docker container. I'm not super familiar with advanced ssh stuff or just networking stuff in general so I'm wondering how to accomplish this.
Here's a roadmap of how I connect:
local computer -> intermediate network -> GPU cluster -> docker container
I normally connect to the GPU cluster using
ssh -X -J 2fly#intermediate.server 2try#XX.XXX.XX.XXX.
My docker command looks like
NV_GPU=all nvidia-docker run --runtime=nvidia --rm -p 5001:6006 -v path/to/data:/UserData/data -v path/to/code:/UserData/code -v path/to/logs:/UserData/logs -it container_name
I've tried adding the -L option to my ssh command but I get an error saying
stty: standard input: Inappropriate ioctl for device
ssh_exchange_identification: Connection closed
EDIT:
Here's the output of ssh -f -v -N -J 2fly#intermediate.server 2try#XX.XXX.XX.XXX –L 5001:XX.XXX.XX.XXX:5001 (with identifying info removed)
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 13: Deprecated option "useroaming"
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l 2fly -v -W '[%h]:%p' intermediate.server
debug1: Executing proxy command: exec ssh -l 2fly -v -W '[XX.XXX.XX.XXX]:22' intermediate.server
debug1: permanently_drop_suid: 16000
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
stty: standard input: Inappropriate ioctl for device
OpenSSH_7.5p1, OpenSSL 1.0.2o 27 Mar 2018
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 13: Deprecated option "useroaming"
debug1: Connecting to intermediate.server [address] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to intermediate.server:22 as '2fly'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: zlib#openssh.com
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: zlib#openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:key
debug1: Host 'intermediate.server' is known and matches the RSA host key.
debug1: Found key in /home/mobaxterm/.ssh/known_hosts:4
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: pubkey_prepare: ssh_fetch_identitylist: agent refused operation
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/mobaxterm/.ssh/id_rsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_dsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ecdsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ed25519
debug1: Next authentication method: password
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (password).
Authenticated to intermediate.server ([address]:22).
debug1: channel_connect_stdio_fwd XX.XXX.XX.XXX:22
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to XX.XXX.XX.XXX:22 as '2try'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: zlib#openssh.com
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: zlib#openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:key
debug1: Host 'XX.XXX.XX.XXX' is known and matches the RSA host key.
debug1: Found key in /home/mobaxterm/.ssh/known_hosts:5
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: pubkey_prepare: ssh_fetch_identitylist: agent refused operation
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256#openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/mobaxterm/.ssh/id_rsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_dsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ecdsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ed25519
debug1: Next authentication method: password
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (password).
Authenticated to XX.XXX.XX.XXX (via proxy).
debug1: Local connections to LOCALHOST:5001 forwarded to remote address XX.XXX.XX.XXX:5001
debug1: Local forwarding listening on ::1 port 5001.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 5001.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions#openssh.com
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0

I try to connect by ssh from Home Assistant (docker) in Debian and there is no way

Good companions. I have a PC with Openmediavault and Home Assistant on a docker container.
I want to ssh the home assistant to the openmediavault system (debian). I try to do it from the SSH & Web Terminal addons, but there is no humane way to do it.
I try to do it like this:
cd /config
mkdir .ssh
ssh-keygen -b 4096 -t rsa
He asks me for the destination route and I give it to him:
/config/.ssh/id_rsa
This copies the private key to the openmediavault server (Debian)
ssh-copy-id -i /config/.ssh/id_rsa peter#192.168.1.37
With this I try to connect to the server to avoid being asked for the password. But he keeps asking me. With the -v it debugs and shows errors supposedly that it does not find the private key.
ssh -v -i /root/.ssh/id_rsa peter#192.168.1.37
OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 192.168.1.37 [192.168.1.37] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: compat_banner: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.1.37:22 as 'peter'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:JEfCsgteJLvd11uZUNVx/DKemQZ1tg5xpRKTF0RggUqBzsY
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '192.168.1.37' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /root/.ssh/id_rsa RSA SHA256:lxlph5P16mJj6H1WhyGku6zaDlQo3ATHWgyphk374pTp8
debug1: Will attempt key: /root/.ssh/id_dsa
debug1: Will attempt key: /root/.ssh/id_ecdsa
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk
debug1: Will attempt key: /root/.ssh/id_ed25519
debug1: Will attempt key: /root/.ssh/id_ed25519_sk
debug1: Will attempt key: /root/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa RSA SHA256:lxlph5P16mJj6H1WhyGku6zaDlQo3ATHWgyphk374pTp8
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ecdsa_sk
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Trying private key: /root/.ssh/id_ed25519_sk
debug1: Trying private key: /root/.ssh/id_xmss
debug1: Next authentication method: password
peter#192.168.1.37's password:

Jenkins Pipeline: SCP commnd not working when executed through pipeline, same works fine through GIT Bash

I have two ubuntu aws instance and below are following details
DevServer: 172.31.29.201
QAServer: 172.31.30.52
I establish a passwordless SSH between two machine bu generating ssh-keygen in DevServer and copied it to QAServer using ssh-copy-id ubuntu#172.31.30.52 [ubuntu is username and 172.31.30.52 is private ip address of the QAServer)
I'm able to successfully make a SSH connection to the QAServer from DevServer without any issue
Even scp command to transfer the .war file to the QAServer tomcat8/webapps is successful. File qaapp.war is transferred to the tomcat8/webapps/ folder
ubuntu#DevServer:~$ scp -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
webapp.war
100% 3125 3.8MB/s 00:00
ubuntu#DevServer:~$
Pipeline code:
node('built-in')
{
stage('ContinousDownload')
{
git 'https://github.com/rasivana/maven.git'
}
stage('ContinousBuild')
{
sh 'mvn package'
}
stage('ContinousDeployment')
{
sh 'scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war'
}
}
We get following error
Complete logs:
+ scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
Executing: program /usr/bin/ssh host 172.31.30.52, user ubuntu, command scp -v -t /var/lib/tomcat8/webapps/qaapp.war
OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.31.30.52 [172.31.30.52] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.31.30.52:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:428mcfQF4kAq6DMmrP68kF6nz18rUZlKA3DjtsTcrpk
debug1: Host '172.31.30.52' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
Load key "/var/lib/jenkins/.ssh/id_rsa": Permission denied
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
debug1: No more authentication methods to try.
ubuntu#172.31.30.52: Permission denied (publickey,password).
lost connection
Same with cmd prompt scp works file and file gets transferred. Thought first time it asked for ubuntu user passowrd. Pasting below for analysis
ubuntu#DevServer:~$ scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
Executing: program /usr/bin/ssh host 172.31.30.52, user ubuntu, command scp -v -t /var/lib/tomcat8/webapps/qaapp.war
OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.31.30.52 [172.31.30.52] port 22.
debug1: Connection established.
debug1: identity file /home/ubuntu/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.31.30.52:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:428mcfQF4kAq6DMmrP68kF6nz18rUZlKA3DjtsTcrpk
debug1: Host '172.31.30.52' is known and matches the ECDSA host key.
debug1: Found key in /home/ubuntu/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:UHBTYkIj/Zm87NrPLFNOwz8/1bzOtM9ahxtdwcCfKIA /home/ubuntu/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 172.31.30.52 ([172.31.30.52]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = C.UTF-8
debug1: Sending command: scp -v -t /var/lib/tomcat8/webapps/qaapp.war
Sending file modes: C0644 3125 webapp.war
Sink: C0644 3125 webapp.war
webapp.war 100% 3125 4.2MB/s 00:00
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 5972, received 2648 bytes, in 0.5 seconds
Bytes per second: sent 12812.5, received 5681.1
debug1: Exit status 0
ubuntu#DevServer:~$
Why same command when executed through GIT Bash works without any error, and command executed through jenkins pipe line execution is not working?
Attaching a text comparison between the Jenkins execution and GIT bash execution image file.
When triggered through Jenkins, default user is 'jenkins' and not ubuntu when i tried manually. So I switched the user to 'jenkins', created a new SSH file using 'ssh-keygen'. Transferred the newly created SSH file to QA server.
Now executing the scp command uses the SSH file and able to transfer the file.

Jenkins job failed, cannot do scp to remote, Permission denied (publickey,password)

I need little help.
In my Jenkinsfile I have stage where I want to execute shell:
sh "sudo scp /home/boris2/webgoat-server-v8.1.0.jar boris3#10.113.80.109:/home/boris3"
but jenkins job failed with output: Permission denied (publickey,password)
I can connect via ssh to remote and I can do a scp command via terminal.
I have try many solutions but cannot find the right one.
Can anyone help?
When I try scp -v I have this output:
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 10.113.80.109 [10.113.80.109] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH compat 0x04000000
debug1: Authenticating to 10.113.80.109:22 as 'boris3'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Sr+ta5yB6J7Kz9FlJoZBy/W3jb0hBeZv4BHcJuDvsE8
debug1: Host '10.113.80.109' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /root/.ssh/id_rsa
debug1: Will attempt key: /root/.ssh/id_dsa
debug1: Will attempt key: /root/.ssh/id_ecdsa
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk
debug1: Will attempt key: /root/.ssh/id_ed25519
debug1: Will attempt key: /root/.ssh/id_ed25519_sk
debug1: Will attempt key: /root/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256#openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ecdsa_sk
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Trying private key: /root/.ssh/id_ed25519_sk
debug1: Trying private key: /root/.ssh/id_xmss
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
debug1: No more authentication methods to try.
boris3#10.113.80.109: Permission denied (publickey,password).
You can try scp -v to see which ssh keys are being used for the communication.
If you were able to connect to the remote machine - as you say - but not as the jenkins user, make sure that $JENKINS_HOME directory contains the .ssh/ directory with the specific ssh keys you need to communicate with the remote machine.
Also i don't think that sudo is necessary in order for scp to work.

how to sshfs/sftp on boot2docker or docker-toolbox

I work with MacOSX (El Capitan) on a Core2duo processor. I work with Docker, but I can't use native docker and I work with Boot2Docker over VirtualBox.
As filesystem is not share between host and container, I can't work with watch program to auto reload on modification file.
I thought to a solution in editing file over SSHFS. But I can't find out how to do that.
ssh docker#192.168.99.100
Work perfectly with password or with id_rsa key
scp -vvv docker#192.168.99.100:/var/log/*.log ./log
Work also correctly
But I can't succeed with sshsf. Is anybody could help me ?
sshfs -odebug,sshfs_debug,loglevel=debug docker#192.168.99.100:/var/log ./log
SSHFS version 2.5
FUSE library version: 2.9.7
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-ologlevel=debug> <-2> <docker#192.168.99.100> <-s> <sftp>
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to 192.168.99.100 [192.168.99.100] port 22.
debug1: Connection established.
debug1: identity file /Users/metzger9/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.4
debug1: match: OpenSSH_6.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.99.100:22 as 'docker'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm#openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm#openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:sY/F5Id4SuRpbOsKcQIGeanTvbibZtsPVV2qGBlSi8A
debug1: Host '192.168.99.100' is known and matches the ECDSA host key.
debug1: Found key in /Users/metzger9/.ssh/known_hosts:7
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/metzger9/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.99.100 ([192.168.99.100]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = fr_FR.UTF-8
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug1: Sending subsystem: sftp
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 3472, received 2388 bytes, in 0.0 seconds
Bytes per second: sent 122807.4, received 84465.5
debug1: Exit status 127
remote host has disconnected
SOME MORE INFORMATION
It seems that the problem come from sftp program. So trying the command
sftp -vvv docker#192.168.99.100
leads to the same error. Some more googling give me an answer for sftp :
sftp -s /usr/libexec/sftp-server docker#192.168.99.100
which work as expected. Now the question is "How configure the sftp server in the docker engine to be able to use it with sshfs"

Resources