ssh connection reset docker (gitlab) - docker

$ ssh -v ce.gitro.net -p 2222
OpenSSH_7.3p1, OpenSSL 1.0.2j 26 Sep 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to ce.gitro.net [175.200.135.69] port 2222.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/dreamboy/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to ce.gitro.net:2222 as 'dreamboy'
debug1: SSH2_MSG_KEXINIT sent
Connection closed by 175.200.135.69 port 2222

I had the same problem, looking at the log files from sshd in /var/log/gitlab/sshd I could see that I had set the wrong permissions for the key files.
In my case running the following solved the problem:
chmod -R 700 /etc/gitlab

Related

How to use tensorboard within a docker container across a ssh tunnel?

I'm running a deep learning network on a remote GPU cluster and I'd like to use tensorboard to visualize my training and performance. The problem is I'm really confused on how to expose the port tensorboard uses to my local computer. Due to security and account permission reasons, I have to connect to this server through another server. I also have the complication of training my network within a docker container. I'm not super familiar with advanced ssh stuff or just networking stuff in general so I'm wondering how to accomplish this.
Here's a roadmap of how I connect:
local computer -> intermediate network -> GPU cluster -> docker container
I normally connect to the GPU cluster using
ssh -X -J 2fly#intermediate.server 2try#XX.XXX.XX.XXX.
My docker command looks like
NV_GPU=all nvidia-docker run --runtime=nvidia --rm -p 5001:6006 -v path/to/data:/UserData/data -v path/to/code:/UserData/code -v path/to/logs:/UserData/logs -it container_name
I've tried adding the -L option to my ssh command but I get an error saying
stty: standard input: Inappropriate ioctl for device
ssh_exchange_identification: Connection closed
EDIT:
Here's the output of ssh -f -v -N -J 2fly#intermediate.server 2try#XX.XXX.XX.XXX –L 5001:XX.XXX.XX.XXX:5001 (with identifying info removed)
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 13: Deprecated option "useroaming"
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l 2fly -v -W '[%h]:%p' intermediate.server
debug1: Executing proxy command: exec ssh -l 2fly -v -W '[XX.XXX.XX.XXX]:22' intermediate.server
debug1: permanently_drop_suid: 16000
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
stty: standard input: Inappropriate ioctl for device
OpenSSH_7.5p1, OpenSSL 1.0.2o 27 Mar 2018
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 13: Deprecated option "useroaming"
debug1: Connecting to intermediate.server [address] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to intermediate.server:22 as '2fly'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: zlib#openssh.com
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: zlib#openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:key
debug1: Host 'intermediate.server' is known and matches the RSA host key.
debug1: Found key in /home/mobaxterm/.ssh/known_hosts:4
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: pubkey_prepare: ssh_fetch_identitylist: agent refused operation
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/mobaxterm/.ssh/id_rsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_dsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ecdsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ed25519
debug1: Next authentication method: password
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (password).
Authenticated to intermediate.server ([address]:22).
debug1: channel_connect_stdio_fwd XX.XXX.XX.XXX:22
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to XX.XXX.XX.XXX:22 as '2try'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: zlib#openssh.com
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: zlib#openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:key
debug1: Host 'XX.XXX.XX.XXX' is known and matches the RSA host key.
debug1: Found key in /home/mobaxterm/.ssh/known_hosts:5
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: pubkey_prepare: ssh_fetch_identitylist: agent refused operation
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256#openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/mobaxterm/.ssh/id_rsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_dsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ecdsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ed25519
debug1: Next authentication method: password
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (password).
Authenticated to XX.XXX.XX.XXX (via proxy).
debug1: Local connections to LOCALHOST:5001 forwarded to remote address XX.XXX.XX.XXX:5001
debug1: Local forwarding listening on ::1 port 5001.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 5001.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions#openssh.com
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0

jenkins pipeline scp through bastion host resulting in ssh_exchange_identification: Connection closed by remote host

I'm Trying to scp file to remote server though ssh using Jenkins pipeline.
~/.ssh/config
Host bastion-fasterdev
Hostname bastion.fasterdev.foo.works
User jenkins
IdentityFile /var/lib/jenkins/.ssh/id_rsa
Host app-server-fasterdev
Hostname ip-11-0-31-76.ap-northeast-1.compute.internal
User root
ForwardAgent yes
ProxyCommand ssh bastion-fasterdev -W %h:%p
Jenkins pipeline code
scp -vvv ./app-server app-server-fasterdev:/data/match/conf/config.properties
Jenkins logs
scp -vvv ./app-server app-server-fasterdev:/data/match/conf/config.properties
Executing: program /usr/bin/ssh host app-server-fasterdev, user root, command scp -v -t /data/match/conf/config.properties
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /var/lib/jenkins/.ssh/config
debug1: /var/lib/jenkins/.ssh/config line 5: Applying options for app-server-fasterdev
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Executing proxy command: exec ssh bastion-fasterdev -W ip-11-0-31-76.ap-northeast-1.compute.internal:22
debug1: permanently_drop_suid: 995
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
ssh_exchange_identification: Connection closed by remote host
Use jenkins user to run the same command with terminal succeeds?
-bash-4.2$ scp -vvv ./app-server app-server-fasterdev:/data/match/conf/config.properties
Executing: program /usr/bin/ssh host app-server-fasterdev, user (unspecified), command scp -v -t /data/match/conf/config.properties
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /var/lib/jenkins/.ssh/config
debug1: /var/lib/jenkins/.ssh/config line 1: Applying options for *
debug1: /var/lib/jenkins/.ssh/config line 28: Applying options for app-server-fasterdev
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -vvv -W %h:%p bastion-fasterdev
debug1: Executing proxy command: exec ssh -vvv -W ip-11-0-31-76.ap-northeast-1.compute.internal:22 bastion-fasterdev
debug1: permanently_drop_suid: 995
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /var/lib/jenkins/.ssh/config
debug1: /var/lib/jenkins/.ssh/config line 1: Applying options for *
debug1: /var/lib/jenkins/.ssh/config line 9: Applying options for bastion-fasterdev
...
...
...
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 2476, received 2452 bytes, in 0.1 seconds
Bytes per second: sent 29435.9, received 29150.6
debug1: Exit status 0
debug1: compress outgoing: raw data 770, compressed 504, factor 0.65
debug1: compress incoming: raw data 504, compressed 770, factor 1.53
debug3: send packet: type 1
debug1: channel 0: free: direct-tcpip: listening port 0 for ip-11-0-31-76.ap-northeast-1.compute.internal port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 direct-tcpip: listening port 0 for ip-11-0-31-76.ap-northeast-1.compute.internal port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536 (t4 r0 i0/0 o0/0 fd 4/5 cc -1)
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Killed by signal 1.
I've also tried with same result
scp -i /var/lib/jenkins/.ssh/id_rsa ./app-server app-server-fasterdev:/data/match/conf/config.properties.test
or
ssh -i /var/lib/jenkins/.ssh/id_rsa app-server-fasterdev

Jenkins Pipeline: SCP commnd not working when executed through pipeline, same works fine through GIT Bash

I have two ubuntu aws instance and below are following details
DevServer: 172.31.29.201
QAServer: 172.31.30.52
I establish a passwordless SSH between two machine bu generating ssh-keygen in DevServer and copied it to QAServer using ssh-copy-id ubuntu#172.31.30.52 [ubuntu is username and 172.31.30.52 is private ip address of the QAServer)
I'm able to successfully make a SSH connection to the QAServer from DevServer without any issue
Even scp command to transfer the .war file to the QAServer tomcat8/webapps is successful. File qaapp.war is transferred to the tomcat8/webapps/ folder
ubuntu#DevServer:~$ scp -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
webapp.war
100% 3125 3.8MB/s 00:00
ubuntu#DevServer:~$
Pipeline code:
node('built-in')
{
stage('ContinousDownload')
{
git 'https://github.com/rasivana/maven.git'
}
stage('ContinousBuild')
{
sh 'mvn package'
}
stage('ContinousDeployment')
{
sh 'scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war'
}
}
We get following error
Complete logs:
+ scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
Executing: program /usr/bin/ssh host 172.31.30.52, user ubuntu, command scp -v -t /var/lib/tomcat8/webapps/qaapp.war
OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.31.30.52 [172.31.30.52] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/lib/jenkins/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.31.30.52:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:428mcfQF4kAq6DMmrP68kF6nz18rUZlKA3DjtsTcrpk
debug1: Host '172.31.30.52' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/jenkins/.ssh/id_rsa
Load key "/var/lib/jenkins/.ssh/id_rsa": Permission denied
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ed25519
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
debug1: No more authentication methods to try.
ubuntu#172.31.30.52: Permission denied (publickey,password).
lost connection
Same with cmd prompt scp works file and file gets transferred. Thought first time it asked for ubuntu user passowrd. Pasting below for analysis
ubuntu#DevServer:~$ scp -v -o StrictHostKeyChecking=no /var/lib/jenkins/workspace/Pipeline_dev_01/webapp/target/webapp.war ubuntu#172.31.30.52:/var/lib/tomcat8/webapps/qaapp.war
Executing: program /usr/bin/ssh host 172.31.30.52, user ubuntu, command scp -v -t /var/lib/tomcat8/webapps/qaapp.war
OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.31.30.52 [172.31.30.52] port 22.
debug1: Connection established.
debug1: identity file /home/ubuntu/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.31.30.52:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:428mcfQF4kAq6DMmrP68kF6nz18rUZlKA3DjtsTcrpk
debug1: Host '172.31.30.52' is known and matches the ECDSA host key.
debug1: Found key in /home/ubuntu/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:UHBTYkIj/Zm87NrPLFNOwz8/1bzOtM9ahxtdwcCfKIA /home/ubuntu/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 172.31.30.52 ([172.31.30.52]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = C.UTF-8
debug1: Sending command: scp -v -t /var/lib/tomcat8/webapps/qaapp.war
Sending file modes: C0644 3125 webapp.war
Sink: C0644 3125 webapp.war
webapp.war 100% 3125 4.2MB/s 00:00
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 5972, received 2648 bytes, in 0.5 seconds
Bytes per second: sent 12812.5, received 5681.1
debug1: Exit status 0
ubuntu#DevServer:~$
Why same command when executed through GIT Bash works without any error, and command executed through jenkins pipe line execution is not working?
Attaching a text comparison between the Jenkins execution and GIT bash execution image file.
When triggered through Jenkins, default user is 'jenkins' and not ubuntu when i tried manually. So I switched the user to 'jenkins', created a new SSH file using 'ssh-keygen'. Transferred the newly created SSH file to QA server.
Now executing the scp command uses the SSH file and able to transfer the file.

SSH from inside container docker to gcp instance in subnetwork

I already add public key to authorized keys to target instance. But I still cannot ssh from inside container to the instance. I'm doing this because I use jenkins in container. But I can ssh from outside container to target instance using public key from host instance. Key from container and host instance is different.
[jenkins#4f5159051056 /]$ ssh jenkins#10.64.0.5 -v
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to 10.64.0.5 [10.64.0.5] port 22.
debug1: Connection established.
debug1: identity file /var/jenkins_home/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /var/jenkins_home/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u3
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.64.0.5:22 as 'jenkins'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:2QOdKiyn5cehOBPIfQFaEo8WvZz6BJYq4fc7gNIIJ1g
The authenticity of host '10.64.0.5 (10.64.0.5)' can't be established.
ECDSA key fingerprint is SHA256:2QOdKiyn5cehOBPIfQFaEo8WvZz6BJYq4fc7gNIIJ1g.
ECDSA key fingerprint is MD5:13:03:df:e8:e5:4b:f7:4a:9b:65:82:c3:9d:80:11:f4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.64.0.5' (ECDSA) to the list of known hosts.
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/jenkins_home/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /var/jenkins_home/.ssh/id_dsa
debug1: Trying private key: /var/jenkins_home/.ssh/id_ecdsa
debug1: Trying private key: /var/jenkins_home/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).
Did you check the following
Setting proper permissions:
chmod 700 /var/jenkins_home/.ssh
chmod 644 /var/jenkins_home/.ssh/authorized_keys
Owner should be Jenkins in your case
chown -R jenkins:jenkins /var/jenkins_home/*
Also you are checking on the client side using ssh -v, you need to check on the server side why the server is not accepting the key Permission denied (publickey).Try checking in the below location
sudo vi /var/log/secure
EDITED
If faced with the below issue
localhost sshd[36601]: User jenkins from xxx.xxx.xxx.xxx not allowed because not listed in AllowUsers
and with this
localhost sshd[36601]: input_userauth_request: invalid
user myuser [preauth]
add the user jenkins to the below file
vim /etc/ssh/sshd_config
and restart service
systemctl restart sshd.service
It should workk now :)

how to sshfs/sftp on boot2docker or docker-toolbox

I work with MacOSX (El Capitan) on a Core2duo processor. I work with Docker, but I can't use native docker and I work with Boot2Docker over VirtualBox.
As filesystem is not share between host and container, I can't work with watch program to auto reload on modification file.
I thought to a solution in editing file over SSHFS. But I can't find out how to do that.
ssh docker#192.168.99.100
Work perfectly with password or with id_rsa key
scp -vvv docker#192.168.99.100:/var/log/*.log ./log
Work also correctly
But I can't succeed with sshsf. Is anybody could help me ?
sshfs -odebug,sshfs_debug,loglevel=debug docker#192.168.99.100:/var/log ./log
SSHFS version 2.5
FUSE library version: 2.9.7
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-ologlevel=debug> <-2> <docker#192.168.99.100> <-s> <sftp>
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to 192.168.99.100 [192.168.99.100] port 22.
debug1: Connection established.
debug1: identity file /Users/metzger9/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/metzger9/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.4
debug1: match: OpenSSH_6.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.99.100:22 as 'docker'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm#openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm#openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:sY/F5Id4SuRpbOsKcQIGeanTvbibZtsPVV2qGBlSi8A
debug1: Host '192.168.99.100' is known and matches the ECDSA host key.
debug1: Found key in /Users/metzger9/.ssh/known_hosts:7
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/metzger9/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.99.100 ([192.168.99.100]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = fr_FR.UTF-8
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug1: Sending subsystem: sftp
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 3472, received 2388 bytes, in 0.0 seconds
Bytes per second: sent 122807.4, received 84465.5
debug1: Exit status 127
remote host has disconnected
SOME MORE INFORMATION
It seems that the problem come from sftp program. So trying the command
sftp -vvv docker#192.168.99.100
leads to the same error. Some more googling give me an answer for sftp :
sftp -s /usr/libexec/sftp-server docker#192.168.99.100
which work as expected. Now the question is "How configure the sftp server in the docker engine to be able to use it with sshfs"

Resources