Getting 'Host key verification failed' when using ssh in docker context - docker

i am setting up docker context like described here and cofigured the ssh key and the context. Unfortunately I keep getting an error from docker while i'm in the new context:
docker context use myhostcontext
docker ps
error during connect: Get "http://docker.example.com/v1.24/containers/json": command [ssh -l user -- myhost docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Host key verification failed.
Suprisingly when i ssh into user#myhost the connection is established as it should be.
ssh -vv user#myhost shows that it uses the given key in ~/.ssh/config
Additional Info:
Platform: Ubuntu 20.04
Docker: 20.10.23
OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f 31 Mar 2020
Here is what i've done:
I've created a docker context with
docker context create myhostcontext --docker "host=ssh://user#myhost"
I also created a new ssh keypair with ssh-keygen (tried with rsa and ecdsa),
executed ssh-add /path/to/key and ssh-copy-id -i /path/to/key user#myhost
I tried with using "id_rsa" as keyname as well as "myhost" to make sure its not just a default naming problem.
Looking at several instructions (e.g. This question) unfortunately did not help. I also checked the authorized_keys on the remote host and the public key on my local machine, they match.
My ~/.ssh/config looks like this
Host myhost
HostName myhost
User user
StrictHostKeyChecking no
IdentityFile ~/.ssh/myhost
Also removing entries from known_host did not help.
Using the remote hosts IP instead of its name did not help either.
Installing ssh-askpass just shows me, that the authenticity could not be established (default message when using ssh on a host for the first time). Since I later want to use docker context in a CI/CD environment i don't want to have any non-cli stuff.
The only other possible "issue" that comes to my mind is that the user of the remote host is different that the one i am using on the client. But - if understood correctly - that should not be an issue and also i would not know how to manage that.
Any help or suggestion is highly appreciated, since I am struggling with this for days.
Thanks in advance :)

Related

Docker certificate has expired or is not yet valid

I just installed docker for the first time and when the default virtualbox or 'docker-machine'(not sure what it is called I run into this error:
This is an excerpt
Checking connection to Docker...
Error creating machine: Error checking the host: Error checking and/or regenerat
ing the certs: There was an error validating certificates for host "192.168.99.1
01:2376": x509: certificate has expired or is not yet valid
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]
'.
Be advised that this will trigger a Docker daemon restart which might stop runni
ng containers.
I tried using the docker-machine regenerate-certs [name] command and it seemed to work no errors were raised at least. Only when I attempted to run docker's hello world command this is the result I got.
docker: error during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.39/c
ontainers/create: open //./pipe/docker_engine: The system cannot find the file s
pecified. In the default daemon configuration on Windows, the docker client must
be run elevated to connect. This error may also indicate that the docker daemon
is not running.
See 'docker run --help'.
I searched up similar errors and found a few case where creating a new virtualbox/docker-machine(still don't know what its called) solved it but the same error that appeared with the default box showed up.
Error creating machine: Error checking the host: Error checking and/or regenerat
ing the certs: There was an error validating certificates for host "192.168.99.1
01:2376": x509: certificate has expired or is not yet valid
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]
Afterwards I gave the docker-machine ls command a try in order to see if I could find something helpful. This is the output.
NAME ACTIVE DRIVER STATE URL SWARM DO
CKER ERRORS
default - virtualbox Running tcp://192.168.99.100:2376 Un
known Unable to query docker version: Get https://192.168.99.100:2376/v1.15/ve
rsion: x509: certificate has expired or is not yet valid
first - virtualbox Running tcp://192.168.99.101:2376 Un
known Unable to query docker version: Get https://192.168.99.101:2376/v1.15/ve
rsion: x509: certificate has expired or is not yet valid
How can I fix it?
Try docker-machine regenerate-certs --client-certs.
Try this:
In your command prompt run:
docker-machine ls
it will return
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
<name> <status> <driver> <state> <url> <swarm> <docker> <errors>
get the <name> of the machine you want to run (if it's the first time you are running docker, then you have only one machine running and its name is default, in this example I will call it <name>)
then run:
docker-machine kill <name>
docker-machine create <name>
docker-machine env <name>
The last command will output more lines, copy the last one, in my case it was like this:
# eval $("C:\my\path\to\docker-machine.exe" env <name>)
then paste it in the command prompt without the # and run it, the problem should be fixed now.
You can check it by running docker-machine ls, if the problem is fixed, you should see no more messages under the ERROR column.
Furthermore, it is possible that the lines
docker-machine kill <name>
docker-machine create <name>
are not required if you have only one docker-machine existing and its <name> is default.
Maybe you can vi /etc/docker/daemon.json
use this setting:
{
"insecure-registries":[
"0.0.0.0/0"
]
}
then restart service:
service docker restart
and try again.

Docker connect to remote daemon via ssh - Permission denied (publickey)

I have a problem with connecting to my remote(DigitalOcean) docker engine. What I've done is
Made a droplet with Docker 19.03.12 on Ubuntu 20.04.
Made a new user myuser and add to docker group on the remote host.
Made a .ssh/authorized_keys for the new user it's home and set the permissions, owner etc.
Restarted both ssh and docker services.
Result
I can ssh from my Mac notebook to my remote host with myuser. (when I run ssh keychain asks for the passphrase for the id_rsa.key.)
After I logged in to remote host via ssh I can run docker ps, docker info without any problem.
Problem
Before I make a new context for the remote engine, I tried to run some docker command from my local client on my Mac laptop. Interesting part for me is none of the commands below asks for the id_rsa passphrase)
docker -H ssh://myuser#droplet_ip ps -> Error
DOCKER_HOST=ssh://myuser#droplet_ip docker ps -> Error
Error
docker -H ssh://myuser#droplet_ip ps
error during connect: Get http://docker/v1.40/containers/json: command [ssh -l myuser -- droplet_ip docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=myuser#droplet_ip: Permission denied (publickey).
What step I missed? How can I connect to a remote docker engine?
It sounds like Docker may not allow ssh to prompt for a key passphrase when connecting. The easiest solution is probably to load your key into an ssh-agent, so that Docker will be able to use the key without requesting a password.
If you want to add your default key (~/.ssh/id_rsa) you can just run:
ssh-add
You can add specific keys by providing a path to the key:
ssh-add ~/.ssh/id_rsa_special
Most modern desktop environments run an ssh-agent process by default.

SSH keys keep getting deleted from Google Compute Engine VM

Background:
I am running a Google Compute Engine VM, called host.
There is a Docker container running on the machine called container.
I connect to the VM using an account called user#gmail.com.
I need to connect through ssh from the container to the host, without being prompted for the user password.
Problem:
Minutes after successfully connecting from the container to the host, the user/.ssh/authorized_keys gets "modified" by some process from Google itself. As far as I understood this process appends some ssh keys needed to connect to the VM. In my case though, the process seems to overwrite the key that I generated from the container.
Setup:
I connect to host using Google Compute Engine GUI, pressing on the SSH button.
Then I follow the steps described in this answer on AskUbuntu.
I set the password for user on host:
user#host:~$ sudo passwd user
I set PasswordAuthentication to yes in sshd_config, and I restart sshd:
user#host:~$ sudo nano /etc/ssh/sshd_config
user#host:~$ sudo systemctl restart sshd
I enter in the Docker container using bash, I generate the key, and I copy it on the host:
user#host:~$ docker exec -it container /bin/bash
(base) root#container-id:# ssh-keygen
(base) root#container-id:# ssh-copy-id user#host
The key is successfully copied to the host, the host is added to the known_hosts file, and I am able to connect from the container to the host without being prompted for the password (as I gave it during the ssh-copy-id execution).
Now, if I detach from the host, let some time pass, and attach again, I find that the user/.ssh/authorized_keys file contains some keys generated by Google, but there is no trace of my key (the one that allows the container to connect to the host).
What puzzles me more than everything is that we consistently used this process before and we never had such problem. Some accounts on this same host have still keys from containers that no longer exist!
Does anyone has any idea about this behavior? Do you know about any solutions that let me keep the key for as long as it is needed?
It looks like the accounts daemon is doing this task. You could refer this discussion thread for more details about this.
You might find the OS Login API a easier management option. Once enabled, you can use a single gcloud command or API call to add SSH keys.
In case anyone has trouble with this even AFTER adding SSH keys to the GCE metadata:
Make sure your username is in the SSH key description section!
For example, if your SSH key is
ssh-rsa AAAA...zzzz
and your login is ubuntu, make sure you actually enter
ssh-rsa AAAA...zzzz ubuntu
since it appears Google copies the key to the authorized_keys of the user specified inside the key.
In case anyone is still looking for solution for this, I solved this issue by storing the SSH Keys in Compute Engine Metadata https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys

Docker unable to start

When I try to do docker run I get this:
docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.
So I looked here https://github.com/docker/kitematic/issues/1010 and I tried this:
docker-machine env default
But I'm getting:
Error checking TLS connection: exit status 126
So I looked here https://github.com/docker/toolbox/issues/453 and I tried this:
docker-machine rm default
Now I'm getting:
Error removing host "default": exit status 126
So what is the issue and how can I solve it?
This issue could be caused by a few things:
Permissions - your user does not have the correct access rights to talk to the socket. Runner sudo usermod -aG docker YOUR-USER replacing YOUR-USER to the correct value. Note you will need to completely log out and log back in again for the changes to take effect
Your shell env is not set correctly with docker-machine - Each tab you open in terminal needs to connect to the correct machine if you are to use it correctly try running eval $(docker-machine env default) and then running some docker commands to see if that resolves your issues
Try regenerating the TLS certs for the machine and repeating step 2 - I noticed there was a TLS error. Sometimes the certs for connecting to the daemon can become invalid. Regenerate the certs by running docker-machine regenerate-certs default.
Update me with your progress and I'll be happy to help troubleshoot further.
Hope this helps
Dylan
Edit
Try creating a fresh docker machine with docker-machine create -d YOUR-PROVIDER YOUR-NAME and seeing if it is a machine specific issue

How to add private registry certs to Docker Machine

I upgraded my Mac (OS X) from an older Docker installation to Docker Toolbox, meaning that I'm now working with Docker Machine, and in the process discovered that certs I had working for push/pull with a private registry are not there, and I can't for the life of me figure out how to get them in place. At the moment when I try a test pull I get the dreaded x509: certificate signed by unknown authority error. I've searched around, looked at issues in Github, but nothing has worked for me. I even tried ssh'ing into the machine VM and manually copying them into /etc/ssl/certs, and various other things, with no luck. And I certainly don't want to get into the "insecure-registry" stuff. This used to work with boot2docker prior to moving to docker-machine.
This seems like a very simple question: I have a couple of .crt files that I need put in the right place so that I can do a push/pull. How does one do this? And secondarily, how can this not be documented anywhere? Can we wish for a docker-machine add-cert command someday?
Thanks for any help, and I hope a good answer here can stick around to assist others who run into this.
Okay so let's imagine I have a registry running at the address: 192.168.188.190:5000 and I have a proper certificate for this address.
I would now run the following commands to install the root certificate into my machine:
docker-machine scp ./dockerCA.crt $MACHINE_NAME:dockerCA.crt
docker-machine ssh $MACHINE_NAME sudo mkdir -p /etc/docker/certs.d/192.168.188.190:5000
docker-machine ssh $MACHINE_NAME sudo mv dockerCA.crt /etc/docker/certs.d/192.168.188.190:5000/dockerCA.crt
Set the variable MACHINE_NAME to whatever the name of your machine is. The machine will now trust your root certificate.
Having the same issue I read the Documentation in Docker on how to add a certificate to my computer.
As you mentioned that you are on a updated Mac OS X, proceed by doing the following:
Copy the cert file from your docker registry to your hard drive, e.g.
scp user#docker.reg.ip:/path/to/crt/domain.crt /tmp/domain.crt
Add the certificate to your trusted certificates using the following command
sudo security add-trusted-cert -d -r trustRoot \
-k /Library/Keychains/System.keychain /tmp/domain.crt
Restart your local docker handler and now you should be able to upload your local Docker images to the Docker registry.
If you are running on any other operating systems please check this site on how to add trusted root certificates.

Resources