Can not pull docker image from private repo when using Minikube - docker

I am attempting to use Minikube for local kubernetes development. I have set up my docker environment to use the docker daemon running in the provided Minikube VM (boot2docker) as suggested:
eval $(minikube docker-env)
It sets up these environment variables:
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/home/jasonwhite/.minikube/certs"
When I attempt to pull an image from our private docker repository:
docker pull oururl.com:5000/myimage:v1
I get this error:
Error response from daemon: Get https://oururl.com:5000/v1/_ping: x509: certificate signed by unknown authority
It appears I need to add a trusted ca root certificate somehow, but have been unsuccessful so far in my attempts.
I can hit the repository fine with curl using our ca root cert:
curl --cacert /etc/ssl/ca/ca.pem https://oururl.com:5000/v1/_ping

I've been unable to find anyway to get the cert into the minikube vm. But, minikube has a command line parameter to pass in an insecure-registry.
minikube start --insecure-registry=<HOST>:5000
Then to configure authentication on the registry, create a secret.
kubectl create secret docker-registry tp-registry --docker-server=<REGISTRY>:5000 --docker-username=<USERNAME> --docker-password=<PASSWORD> --docker-email=<EMAIL> --insecure-skip-tls-verify=true
Add secret to the default service account as described in the kubernetes docs.

I came up with a work-around for the situation with suggestions from these sources:
https://github.com/docker/machine/issues/1799
https://github.com/docker/machine/issues/1872
I logged into the Minikube VM (minikube ssh), and edited the /usr/local/etc/ssl/certs/ca-certificates.crt file by appending my own ca cert.
I then restarted the docker daemon while still within the VM: sudo /etc/init.d/docker restart
This is not very elegant in that if I restart the Minikube VM, I need to repeat these manual steps each time.
As an alternative, I also attempted to set the --insecure-registry myurl.com:5000 option in the DOCKER_OPTS environment variable (restarted docker), but this didn't work for me.

An addon was recently added to Minikube that makes access to private container registries much easier:
minikube addons configure registry-creds
minikube addons enable registry-creds

For an http registry this steps works for me:
1) minikube ssh
2) edit /var/lib/boot2docker/profile and add to $EXTRA_ARGS --insecure-registry yourdomain.com:5000
3) restart the docker daemon sudo /etc/init.d/docker restart

The Kubernetes documentation on this is pretty good.
Depending on where your private docker repository is hosted, the solution will look a bit different. The documentation explains how to handle each type of repository.
If you want an automated approach to handle this authentication, you will want to use a Kubernetes secret and specify the imagePullSecrets for your Pod.

Sounds like your question has more to do with Docker than Kubernetes. The Docker CLI supports a number of TLS-related options. Since you already have the CA cert, something like this should work:
docker --tlsverify --tlscacert=/etc/ssl/ca/ca.pem pull oururl.com:5000/myimage:v1

You need to edit /etc/default/docker to look like so:
# Docker Upstart and SysVinit configuration file
#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
# Please see the documentation for "systemd drop-ins":
# https://docs.docker.com/engine/admin/systemd/
#
# Customize location of Docker binary (especially for development testing).
#DOCKERD="/usr/local/bin/dockerd"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--insecure-registry oururl.com:5000"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export DOCKER_TMPDIR="/mnt/bigdrive/docker-tmp"
Make sure to sudo service docker stop and sudo docker start to apply the changes. You should then be able to push/pull to your registry.

login account minikube
vi ~/.minikube/machines/<PROFILE_NAME>/config.json (in my case vi ~/.minikube/machines/minikube/config.json)
add private repo on InsecureRegistry attribute (json path: HostOptions.EngineOptions.InsecureRegistry)
minikube start again

Related

unable to resolve docker endpoint

Post my latest Docker Update installation i am getting below error
PS C:\Users\vpasumar> docker ps -a
unable to resolve docker endpoint: default orchestrator is kubernetes but unable to resolve kubernetes endpoint: Error loading config file "C:\Users\vpasumar\.kube\config": yaml: control characters are not allowed
I see docker as running and kubernetes as 'starting' only. I did not start after much of time also.
Any idea, how to access docker.
It looks like you have docker stack defaulted to be deployed to Kubernetes in other words Kubernetes is the default orchestrator in your machine.
You can disable this easily from Docker Desktop settings/preferences, uncheck Deploy Docker Stacks to Kubernetes by default
I am using docker desktop 3.6.0 and i don't have option to select or unselect Default Stack.
However, Removing following statement from C:\Users\your user\.docker\config.json
"stackOrchestrator": "kubernetes"
Run any docker command again, it solves the problem.
I have removed the C:\Users\vpasumar\.kube\config file; the issue is now solved. I'm able to use docker command correctly.
If you're using Linux and looking to undo and reset the local environment
Try :
eval $(docker-machine env --unset)
You can refere to this answer of Elton Stoneman enter link description here
Hope this helps ^^
This should fix the issue:
kubectl config set-context docker-desktop
or
kubectl config set-context minikube
Docker needs a default kubernetes endpoint.

Docker pull image without ssl in Kubernetes with docker private registry

When I try to deploy something with docker registry I every time view errors:
x509: cannot validate certificate for 10.2.10.7 because it doesn't contain any IP SANs
Question:
How I can disable ssl from deploy image in docker registry to Kubernetes ?
Assuming relaxed security is OK for your environment, a way to accomplish in Kubernetes what you want is to configure Docker to connect to the private registry as an insecure registry.
Per the doc here:
With insecure registries enabled, Docker goes through the following
steps:
First, try using HTTPS. If HTTPS is available but the certificate is invalid, ignore the error about the certificate.
If HTTPS is not available, fall back to HTTP.
Notice that the change to /etc/docker/daemon.json described in that doc - adding "insecure-registries" configuration - has to be applied to all nodes in the Kubernetes cluster on which pods/containers can be scheduled to run. Plus, Docker has to be restarted for the change to take effect.
It is also to note that the above assumes the cluster uses the Docker container runtime and not some other runtime (e.g. CRI-O) that supports the Docker image format and registry.
As you're using self signed TLS certificate, you need to add the certificate to the known certificates list.
Grab you .crt file and cope it to the client machine's ssl certificates directory.
For ubuntu:
$ sudo cp registry.crt /usr/local/share/ca-certificates/registry.crt
$ sudo update-ca-certificates
Now restart docker:
$ sudo systemctl restart docker
For CentOS 7:
copy the certificate inside /etc/pki/ca-trust/source/anchors/
Use update-ca-trust command
My problem was with certificates because I used self-signed TLS certificates. It is not good idea. You might encounter with known certificates list and you will need to add certificates each time and use command update-ca-certificates (if you are using Centos 7). However, you might encounter another issue with certificates with another error code.
To resolve this issue i've used 3rd party Certificate Authority called Let'sEncrypt.

Using minikube to pull image from local Docker registry (with self-signed CA certificate)

Using minikube to pull image from local Docker registry (with self-signed CA certificate)
I'd like to be able to run minikube so that it can access a local docker registry using a self signed CA certificate. Ideally the process should be automated so that I can use a *deployment.yaml file to pull the required image without intervention.
At the moment I'm using a workaroud as follows:
#ssh into the minikube instance
sudo minikube ssh
#create a folder for the certificate
sudo mkdir /etc/docker/certs.d/dave.local:5000
#copy the crt file from the registry computer to the minikube instance
sudo scp user#192.168.1.2:/home/dave/certs/domain.crt /etc/docker/certs.d/dave.local:5000
#then check login
docker login dave.local:5000
#then pull image so that it's already in minikube
docker pull dave.local:5000/davedockerimage
I then edit the *deployment.yaml with imagePullPolicy: Never . When I then run sudo kubectl create -f dave-deployment.yamlit finds dave.local:5000/davedockerimagelocally on minikube it uses the already pulled image.
If imagePullPolicy: Always . The image pull fails in minikube.
I've been through a range of tutorials/stack overflow answers and have been unable to crack this. Any help appreciated.
As a alternative for using self signed certificate in minikube you can start minikube with insecure-registry option like below:
minikube start --insecure-registry="dave.local:5000"

Can't push to remote docker registry

When trying to push an image to a remote docker registry i get the message:
vagrant#vagrant-ubuntu-trusty-64:~/helloworld$ docker push 11.22.33.44:5000/ltrojanowski/helloworld
The push refers to a repository [11.22.33.44:5000/ltrojanowski/helloworld]
unable to ping registry endpoint https://11.22.33.44:5000/v0/
v2 ping attempt failed with error: Get https://11.22.33.44:5000/v2/: EOF
v1 ping attempt failed with error: Get https://11.22.33.44:5000/v1/_ping: EOF
This is despite changing the etc/default/docker on the server to look like:
# Docker Upstart and SysVinit configuration file
#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
# Please see the documentation for "systemd drop-ins":
# https://docs.docker.com/engine/articles/systemd/
#
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
DOCKER_OPTS="--insecure-registry 11.22.33.44:5000"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
I only added the DOCKER_OPTS="--insecure-registry 11.22.33.44:5000" line, as stated in the documentation.
Many thanks for any help.
As suggested in the comments I misunderstood the instruction. One needs to add the line DOCKER_OPTS="--insecure-registry 11.22.33.44:5000" in the machine which is pushing the container, and not the registry machine.

Network timed out while trying to connect to https://index.docker.io

I installed Docker-Toolbox just now while following their webpage
I started with Docker QuickStart Terminal and see following
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com
bash-3.2$
But when I try to perform docker pull hello-world, this is what I see
bash-3.2$ docker run hello-world
Unable to find image 'hello-world:latest' locally
Pulling repository docker.io/library/hello-world
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/hello-world/images. You may want to check your internet connection or if you are behind a proxy.
bash-3.2$
What's wrong?
I had the same problem this morning and the following fixed it for me:
$ docker-machine restart default # Restart the environment
$ eval $(docker-machine env default) # Refresh your environment settings
It appears that this is due to the Docker virtual machine getting itself into a strange state. There is an open github issue here
I installed Docker without the Toolbox on Windows 10, so the version that requires Hyper-V to be enabled.
For Docker version 1.12 I had to go into the taskbar, right click the Docker Icon, select Settings -> Network and set the DNS Server to fixed, so that is uses Google's DNS server at 8.8.8.8.
Once that setting was changed, it finally worked.
The simpler solution is to add the following entry in /etc/default/docker file
export http_proxy="http://HOST:PORT/"
and restart the docker service
service docker restart
Update August 2016
Using Docker for Mac (version 1.12.0), was seeing issues of the form:
➜ docker pull node
Using default tag: latest
Pulling repository docker.io/library/node
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/node/images. You may want to check your internet connection or if you are behind a proxy.`enter code here`
This was resolved by updating my MacBook Pro wireless network settings to include the following DNS entry: 8.8.8.8
For further info, please see this (dated) issue which provided the answer given here.
I ran into this problem running Docker on my MAC(host) with Docker VM in VBOX 5.10. It is a networking issue. The simple fix is to add a bridged network to the VBOX image. You can use the included NAT config present with the VM, but you need to change the ssh port from 50375 to 2375.
sudo service docker stop
sudo service docker start
works for me..
somehow, sudo service docker restart didn't work
(RHEL7)
On Windows 7 and if you believe you are behind proxy
Logon to default machine
$ docker-machine ssh default
Update profile to update proxy settings
docker#default:~$ sudo vi /var/lib/boot2docker/profile
Append from the below as appropriate
# replace with your office's proxy environment
export"HTTP_PROXY=http://PROXY:PORT"
export"HTTPS_PROXY=http://PROXY:PORT"
# you can add more no_proxy with your environment.
export"NO_PROXY=192.168.99.*,*.local,169.254/16,*.example.com,192.168.59.*"
Exit
docker#default:~$ exit
Restart docker machine
docker-machine restart default
Update environment settings
eval $(docker-machine env default)
Above steps are slightly tweaked but as given in troubleshooting guide: https://docs.docker.com/toolbox/faqs/troubleshoot/#/update-varlibboot2dockerprofile-on-the-docker-machine
I ran into this exact same problem yesterday and none of the "popular" answers (like fixing DNS to 8.8.8.8) worked for me. I eventually happened across this link, and that did the trick ... https://github.com/docker/for-win/issues/16
Between Docker for Windows, Windows 10 and Hyper-V, there seems to be a problem during the virtual network adapter creation process. Specifically, you might end up with two "vEthernet (DockerNAT)" network adapters. Check this with Get-NetAdapter "vEthernet (DockerNAT)" (in an elevated PowerShell console). If the result shows more than one adapter, you can disable and rename it with:
$vmNetAdapter = Get-VMNetworkAdapter -ManagementOS -SwitchName DockerNAT
Get-NetAdapter "vEthernet (DockerNAT)" | ? { $_.DeviceID -ne $vmNetAdapter.DeviceID } | Disable-NetAdapter -Confirm:$False -PassThru | Rename-NetAdapter -NewName "OLD"
Then open up Device Manager and delete the disabled adapter (for some reason you can do this from here, but not from the Network and Sharing Center adapters view).
I assume that you have a network problem. Are you behind a proxy? Is it possible that it filters the connection to docker.io or blocks the docker user agent?
I installed the toolbox and ran your test. It works fine, here:
docker is configured to use the default machine with IP 192.168.99.101
For help getting started, check out the docs at https://docs.docker.com
bash-3.2$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
535020c3e8ad: Pull complete
af340544ed62: Already exists
library/hello-world:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:d5fbd996e6562438f7ea5389d7da867fe58e04d581810e230df4cc073271ea52
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker Hub account:
https://hub.docker.com
For more examples and ideas, visit:
https://docs.docker.com/userguide/
bash-3.2$
On Windows 10. Just right-click on the systray docker icon-> Settings... -> Rest -> Restrart Docker
I had this same problem with boot2docker and fixed it by restarting it with:
boot2docker restart
I just ran into this today with 1.10.1 and none of the existing solutions worked. I tried to restart, upgrade, regenerate certs, ...
I noticed that I had a lot of networks created on the machine. After removing them with:
docker network ls | grep bridge | awk '{print $1}' | xargs -n1 docker network rm
The DNS started working again.
Note: You may ignore errors about pre-defined networks
If you are behind proxy it is not enough to set HTTP_PROXY and HTTPS_PROXY env. You should set it while machine creation.
Paramer for this is --engine-env:
docker-machine create -d "virtualbox" --engine-env HTTP_PROXY=http://<PROXY>:<PORT> --engine-env HTTPS_PROXY=<PROXY>:<PORT> dev
In my case, installing docker on Alpine Linux I get the error:
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/........
Using the script here:
https://github.com/docker/docker/blob/master/contrib/download-frozen-image-v2.sh
Works. It downloads the image using curl and then shows you how to untar and 'docker load' it.
I tried the above methods of static DNS at 8.8.8.8 and disabling ipv6 (I didn't understand the proxy thing) and none of them worked for me.
EDIT 9/8/2016:
I was initially using dropbear instead of openssh. Reinstalled Alpine with openssh fixed the problem.
The next problem was 'ApplyLayer exit status 1 stdout: stderr: chmod /bin/mount: permission denied' error during pull.
From (nixaid.com/grsec-in-docker/):
To build the Docker image, I had to disable the following grsec
protections. Modify the /etc/sysctl.d/grsec.conf as follows:
kernel.grsecurity.chroot_deny_chmod = 0
kernel.grsecurity.chroot_deny_mknod = 0
kernel.grsecurity.chroot_caps = 0 # related to a systemd package/CAP_SETFCAP
in alpine's case though it's
/etc/sysctl.d/00-alpine.conf
reboot
Restarting Docker or recreating the image did not help. I rebooted Windows to no avail.
Astoundingly, when I ssh'ed into the running container and did curl https://index.docker.io/v1/repositories/library/hello-world/images I got a perfectly valid response.
I used the Docker Toolbox with VirtualBox on 64bit Windows 10 Pro.
The solution in my case was to uninstall the old Docker version and install the new one that uses Hyper-V instead of VirtualBox.
Now Docker works again.
If you are behind proxy kindly use below commands
sudo mkdir /etc/systemd/system/docker.service.d
sudo cd /etc/systemd/system/docker.service.d
sudo vi http-proxy.conf
[Service]
Environment=HTTP_PROXY=http://proxy-server-ip:port" "NO_PROXY=localhost,127.0.0.1"
sudo systemctl daemon-reload
sudo systemctl show --property=Environment docker
sudo systemctl restart docker
Try this if you can fetch latest ubuntu
sudo docker run -it ubuntu bash
Unable to find image ubuntu:latest locally
latest: Pulling from library/ubuntu b3e1c725a85f: Pull complete
4daad8bdde31: Pull complete
63fe8c0068a8: Pull complete
4a70713c436f: Pull complete
bd842a2105a8: Pull complete
Digest:
sha256:7a64bc9c8843b0a8c8b8a7e4715b7615e4e1b0d8ca3c7e7a76ec8250899c397a
Status: Downloaded newer image for ubuntu:latest
It worked for me finally :)
Another scenario: if your docker network adapter is disabled, it will fail with this error. The adapter is named "vEthernet (DockerNAT)" or similar. Apparently this adapter is involved somehow in the normal docker pull behavior. Enable it back to solve the problem.
Create a systemd drop-in directory for the docker service:
$ sudo mkdir -p /etc/systemd/system/docker.service.d
Create a file called /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"
Hope it helps
refer to https://docs.docker.com/network/proxy/
for me, proxy setting without http:// or https:// prefix works.
e.g:
PROXY:PORT
or with / suffix with http:// or https:// prefix
e.:
http://PROXY:PORT/
On Windows this happened when I moved from a work network to a home network.
To solve it, run:
docker-machine stop
docker-machine start
docker-env
"C:\Program Files\Docker Toolbox\docker-machine.exe" env | Invoke-Expression

Resources