openssh-server in a pod | ssh connection fails with error: Connection reset by <IP> port 30500 - docker

**Logs from sshd server
`debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 806
debug2: parse_server_config: config /etc/ssh/sshd_config len 806
debug3: /etc/ssh/sshd_config:17 setting Port 30500
debug3: /etc/ssh/sshd_config:19 setting ListenAddress 0.0.0.0
debug3: /etc/ssh/sshd_config:22 setting HostKey /etc/ssh/ssh_host_rsa_key
debug3: /etc/ssh/sshd_config:23 setting HostKey /etc/ssh/ssh_host_ecdsa_key
debug3: /etc/ssh/sshd_config:24 setting HostKey /etc/ssh/ssh_host_ed25519_key
debug3: /etc/ssh/sshd_config:36 setting SyslogFacility AUTH
debug3: /etc/ssh/sshd_config:38 setting LogLevel VERBOSE
debug3: /etc/ssh/sshd_config:43 setting PermitRootLogin yes
debug3: /etc/ssh/sshd_config:44 setting StrictModes no
debug3: /etc/ssh/sshd_config:52 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: /etc/ssh/sshd_config:60 setting HostbasedAuthentication no
debug3: /etc/ssh/sshd_config:68 setting PasswordAuthentication no
debug3: /etc/ssh/sshd_config:73 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:83 setting GSSAPIAuthentication no
debug3: /etc/ssh/sshd_config:84 setting GSSAPICleanupCredentials no
debug3: /etc/ssh/sshd_config:105 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:131 setting AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
debug3: /etc/ssh/sshd_config:132 setting AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
debug3: /etc/ssh/sshd_config:133 setting AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
debug3: /etc/ssh/sshd_config:134 setting AcceptEnv XMODIFIERS
debug3: /etc/ssh/sshd_config:137 setting Subsystem sftp /usr/libexec/openssh/sftp-server
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:6XsUKJrlEzspiLw1H/e5qfrzga/n4Rgs
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:yJkcJ2AX3E4dOADjCRn9EWnut+z5nW3xKhGOc
debug1: private host key #2: ssh-ed25519 SHA256:GHvEepwimuJpanKOXJx8Aacpcs8MwXxlmaU7Q
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug3: oom_adjust_setup
debug1: Set /proc/self/oom_score_adj from 1000 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 30500 on 0.0.0.0.
Bind to port 30500 on 0.0.0.0 failed: Address already in use.
Cannot bind any address.`
Logs from ssh command:
`ssh -vvv localhost -p 30500`
`OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "localhost" port 30500
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to localhost [::1] port 30500.
debug1: connect to address ::1 port 30500: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 30500.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to localhost:30500 as 'root'
debug3: put_host_port: [localhost]:30500
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
Connection reset by 127.0.0.1 port 30500`
Server seems running fine
`netstat -anp | grep 30500`
`tcp 0 0 0.0.0.0:30500 0.0.0.0:* LISTEN 1/sshd`
Observations
The same image works fine when run as a docker container locally (with the same sshd_config temporarily allowing password auth)
The service picks the correct endpoint and we can telnet to the custom ssh port > 30500
The SSH process is running in the pod
Telnet to the ssh port works fine (from another pod, from outside the cluster)
Another container within the same pod with nginx on port 80 works just fine
At present, we get the error Connection reset by port 30500
a) NodePort or LoadBalancer(OCI) does not work
b) In fact ssh to localhost from within the pod fails as well (or from another pod, or from the worker node)
Any guidance here please?

Related

jenkins pipeline scp through bastion host resulting in ssh_exchange_identification: Connection closed by remote host

I'm Trying to scp file to remote server though ssh using Jenkins pipeline.
~/.ssh/config
Host bastion-fasterdev
Hostname bastion.fasterdev.foo.works
User jenkins
IdentityFile /var/lib/jenkins/.ssh/id_rsa
Host app-server-fasterdev
Hostname ip-11-0-31-76.ap-northeast-1.compute.internal
User root
ForwardAgent yes
ProxyCommand ssh bastion-fasterdev -W %h:%p
Jenkins pipeline code
scp -vvv ./app-server app-server-fasterdev:/data/match/conf/config.properties
Jenkins logs
scp -vvv ./app-server app-server-fasterdev:/data/match/conf/config.properties
Executing: program /usr/bin/ssh host app-server-fasterdev, user root, command scp -v -t /data/match/conf/config.properties
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /var/lib/jenkins/.ssh/config
debug1: /var/lib/jenkins/.ssh/config line 5: Applying options for app-server-fasterdev
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Executing proxy command: exec ssh bastion-fasterdev -W ip-11-0-31-76.ap-northeast-1.compute.internal:22
debug1: permanently_drop_suid: 995
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
ssh_exchange_identification: Connection closed by remote host
Use jenkins user to run the same command with terminal succeeds?
-bash-4.2$ scp -vvv ./app-server app-server-fasterdev:/data/match/conf/config.properties
Executing: program /usr/bin/ssh host app-server-fasterdev, user (unspecified), command scp -v -t /data/match/conf/config.properties
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /var/lib/jenkins/.ssh/config
debug1: /var/lib/jenkins/.ssh/config line 1: Applying options for *
debug1: /var/lib/jenkins/.ssh/config line 28: Applying options for app-server-fasterdev
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -vvv -W %h:%p bastion-fasterdev
debug1: Executing proxy command: exec ssh -vvv -W ip-11-0-31-76.ap-northeast-1.compute.internal:22 bastion-fasterdev
debug1: permanently_drop_suid: 995
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /var/lib/jenkins/.ssh/config
debug1: /var/lib/jenkins/.ssh/config line 1: Applying options for *
debug1: /var/lib/jenkins/.ssh/config line 9: Applying options for bastion-fasterdev
...
...
...
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 2476, received 2452 bytes, in 0.1 seconds
Bytes per second: sent 29435.9, received 29150.6
debug1: Exit status 0
debug1: compress outgoing: raw data 770, compressed 504, factor 0.65
debug1: compress incoming: raw data 504, compressed 770, factor 1.53
debug3: send packet: type 1
debug1: channel 0: free: direct-tcpip: listening port 0 for ip-11-0-31-76.ap-northeast-1.compute.internal port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 direct-tcpip: listening port 0 for ip-11-0-31-76.ap-northeast-1.compute.internal port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536 (t4 r0 i0/0 o0/0 fd 4/5 cc -1)
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Killed by signal 1.
I've also tried with same result
scp -i /var/lib/jenkins/.ssh/id_rsa ./app-server app-server-fasterdev:/data/match/conf/config.properties.test
or
ssh -i /var/lib/jenkins/.ssh/id_rsa app-server-fasterdev

Jenkins Pipeline: SCP commnd not working when executed through pipeline, same works fine through GIT Bash

I have two ubuntu aws instance and below are following details
DevServer: 172.31.29.201
QAServer: 172.31.30.52
I establish a passwordless SSH between two machine bu generating ssh-keygen in DevServer and copied it to QAServer using ssh-copy-id ubuntu#172.31.30.52 [ubuntu is username and 172.31.30.52 is private ip address of the QAServer)
I'm able to successfully make a SSH connection to the QAServer from DevServer without any issue
Even scp command to transfer the .war file to the QAServer tomcat8/webapps is successful. File qaapp.war is transferred to the tomcat8/webapps/ folder
ubuntu#DevServer:~$ scp -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
webapp.war
100% 3125 3.8MB/s 00:00
ubuntu#DevServer:~$
Pipeline code:
node('built-in')
{
stage('ContinousDownload')
{
git 'https://github.com/rasivana/maven.git'
}
stage('ContinousBuild')
{
sh 'mvn package'
}
stage('ContinousDeployment')
{
sh 'scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war'
}
}
We get following error
Complete logs:
+ scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
Executing: program /usr/bin/ssh host 172.31.30.52, user ubuntu, command scp -v -t /var/lib/tomcat8/webapps/qaapp.war
OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.31.30.52 [172.31.30.52] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.31.30.52:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:428mcfQF4kAq6DMmrP68kF6nz18rUZlKA3DjtsTcrpk
debug1: Host '172.31.30.52' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
Load key "/var/lib/jenkins/.ssh/id_rsa": Permission denied
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
debug1: No more authentication methods to try.
ubuntu#172.31.30.52: Permission denied (publickey,password).
lost connection
Same with cmd prompt scp works file and file gets transferred. Thought first time it asked for ubuntu user passowrd. Pasting below for analysis
ubuntu#DevServer:~$ scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
Executing: program /usr/bin/ssh host 172.31.30.52, user ubuntu, command scp -v -t /var/lib/tomcat8/webapps/qaapp.war
OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.31.30.52 [172.31.30.52] port 22.
debug1: Connection established.
debug1: identity file /home/ubuntu/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.31.30.52:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:428mcfQF4kAq6DMmrP68kF6nz18rUZlKA3DjtsTcrpk
debug1: Host '172.31.30.52' is known and matches the ECDSA host key.
debug1: Found key in /home/ubuntu/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:UHBTYkIj/Zm87NrPLFNOwz8/1bzOtM9ahxtdwcCfKIA /home/ubuntu/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 172.31.30.52 ([172.31.30.52]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = C.UTF-8
debug1: Sending command: scp -v -t /var/lib/tomcat8/webapps/qaapp.war
Sending file modes: C0644 3125 webapp.war
Sink: C0644 3125 webapp.war
webapp.war 100% 3125 4.2MB/s 00:00
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 5972, received 2648 bytes, in 0.5 seconds
Bytes per second: sent 12812.5, received 5681.1
debug1: Exit status 0
ubuntu#DevServer:~$
Why same command when executed through GIT Bash works without any error, and command executed through jenkins pipe line execution is not working?
Attaching a text comparison between the Jenkins execution and GIT bash execution image file.
When triggered through Jenkins, default user is 'jenkins' and not ubuntu when i tried manually. So I switched the user to 'jenkins', created a new SSH file using 'ssh-keygen'. Transferred the newly created SSH file to QA server.
Now executing the scp command uses the SSH file and able to transfer the file.

SSH from inside container docker to gcp instance in subnetwork

I already add public key to authorized keys to target instance. But I still cannot ssh from inside container to the instance. I'm doing this because I use jenkins in container. But I can ssh from outside container to target instance using public key from host instance. Key from container and host instance is different.
[jenkins#4f5159051056 /]$ ssh jenkins#10.64.0.5 -v
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to 10.64.0.5 [10.64.0.5] port 22.
debug1: Connection established.
debug1: identity file /var/jenkins_home/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u3
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.64.0.5:22 as 'jenkins'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:2QOdKiyn5cehOBPIfQFaEo8WvZz6BJYq4fc7gNIIJ1g
The authenticity of host '10.64.0.5 (10.64.0.5)' can't be established.
ECDSA key fingerprint is SHA256:2QOdKiyn5cehOBPIfQFaEo8WvZz6BJYq4fc7gNIIJ1g.
ECDSA key fingerprint is MD5:13:03:df:e8:e5:4b:f7:4a:9b:65:82:c3:9d:80:11:f4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.64.0.5' (ECDSA) to the list of known hosts.
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/jenkins_home/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /var/jenkins_home/.ssh/id_dsa
debug1: Trying private key: /var/jenkins_home/.ssh/id_ecdsa
debug1: Trying private key: /var/jenkins_home/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).
Did you check the following
Setting proper permissions:
chmod 700 /var/jenkins_home/.ssh
chmod 644 /var/jenkins_home/.ssh/authorized_keys
Owner should be Jenkins in your case
chown -R jenkins:jenkins /var/jenkins_home/*
Also you are checking on the client side using ssh -v, you need to check on the server side why the server is not accepting the key Permission denied (publickey).Try checking in the below location
sudo vi /var/log/secure
EDITED
If faced with the below issue
localhost sshd[36601]: User jenkins from xxx.xxx.xxx.xxx not allowed because not listed in AllowUsers
and with this
localhost sshd[36601]: input_userauth_request: invalid
user myuser [preauth]
add the user jenkins to the below file
vim /etc/ssh/sshd_config
and restart service
systemctl restart sshd.service
It should workk now :)

ssh connection reset docker (gitlab)

$ ssh -v ce.gitro.net -p 2222
OpenSSH_7.3p1, OpenSSL 1.0.2j 26 Sep 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to ce.gitro.net [175.200.135.69] port 2222.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to ce.gitro.net:2222 as 'dreamboy'
debug1: SSH2_MSG_KEXINIT sent
Connection closed by 175.200.135.69 port 2222
I had the same problem, looking at the log files from sshd in /var/log/gitlab/sshd I could see that I had set the wrong permissions for the key files.
In my case running the following solved the problem:
chmod -R 700 /etc/gitlab

how to sshfs/sftp on boot2docker or docker-toolbox

I work with MacOSX (El Capitan) on a Core2duo processor. I work with Docker, but I can't use native docker and I work with Boot2Docker over VirtualBox.
As filesystem is not share between host and container, I can't work with watch program to auto reload on modification file.
I thought to a solution in editing file over SSHFS. But I can't find out how to do that.
ssh docker#192.168.99.100
Work perfectly with password or with id_rsa key
scp -vvv docker#192.168.99.100:/var/log/*.log ./log
Work also correctly
But I can't succeed with sshsf. Is anybody could help me ?
sshfs -odebug,sshfs_debug,loglevel=debug docker#192.168.99.100:/var/log ./log
SSHFS version 2.5
FUSE library version: 2.9.7
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-ologlevel=debug> <-2> <docker#192.168.99.100> <-s> <sftp>
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to 192.168.99.100 [192.168.99.100] port 22.
debug1: Connection established.
debug1: identity file /Users/metzger9/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.4
debug1: match: OpenSSH_6.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.99.100:22 as 'docker'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm#openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm#openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:sY/F5Id4SuRpbOsKcQIGeanTvbibZtsPVV2qGBlSi8A
debug1: Host '192.168.99.100' is known and matches the ECDSA host key.
debug1: Found key in /Users/metzger9/.ssh/known_hosts:7
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/metzger9/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.99.100 ([192.168.99.100]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = fr_FR.UTF-8
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug1: Sending subsystem: sftp
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 3472, received 2388 bytes, in 0.0 seconds
Bytes per second: sent 122807.4, received 84465.5
debug1: Exit status 127
remote host has disconnected
SOME MORE INFORMATION
It seems that the problem come from sftp program. So trying the command
sftp -vvv docker#192.168.99.100
leads to the same error. Some more googling give me an answer for sftp :
sftp -s /usr/libexec/sftp-server docker#192.168.99.100
which work as expected. Now the question is "How configure the sftp server in the docker engine to be able to use it with sshfs"

Resources