Failed to pull image from private docker registry in Kubernetes Cluster [duplicate] - docker

Question

Trying to add insecure registry to containerd config as below:
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
max_conf_num = 1
conf_template = ""
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."test.http-registry.io"]
endpoint = ["http://v048011.dom600.lab:5000"]
Even after adding it to config.toml, when pulling image from the insecure registry, it fails:
sudo ctr image pull v048011.dom600.lab:5000:5000/myjenkins:latest
ctr: failed to resolve reference "v048011.dom600.lab:5000/myjenkins:latest": failed to do request: Head https://v048011.dom600.lab:5000:5000/v2/myjenkins/manifests/latest: http: server gave HTTP response to HTTPS client
In docker we could just add the insecure registry to daemon.json file and docker would pull images from it, how can i achieve the same in containerd ?
Replacing docker as runtime in k8s cluster.

Answer 1

ctr does not read the /etc/containerd/config.toml config file, this config is used by cri, which means kubectl or crictl would use it.
The error log http: server gave HTTP response to HTTPS client, shows that the registry is using http, but ctr is trying to connect it using https. So if you want to pull the image from http, you should add the param --plain-http with ctr like this:
$ ctr i pull --plain-http <image>
The registry config doc is here.
You should be able to pull the image with crictl, remember to restart containerd.
$ sudo crictl -r /run/containerd/containerd.sock pull <image>
# or config runntime once for all
$ sudo crictl config runtime-endpoint /run/containerd/containerd.sock
$ sudo crictl pull <image>
Config example:
# /etc/containerd/config.toml
# change <IP>:5000 to your registry url
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."<IP>:5000"]
endpoint = ["http://<IP>:5000"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."<IP>:5000".tls]
insecure_skip_verify = true
Restart the service after configuration modification.
$ sudo systemctl restart containerd

Answer 2

Adding the following config:
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
max_conf_num = 1
conf_template = ""
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."test.http-registry.io"]
endpoint = ["http://v048011.dom600.lab:5000"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."test.http-registry.io".tls]
insecure_skip_verify = true
should skip TLS verification for the test registry. See also the documentation on registry TLS communication configuration.
Edit: Please note the "s" in plugins, there is a typo in your configuration.
NOTE: Be sure to restart containerd aferwards:
$ sudo systemctl restart containerd

Answer 3

In my case, I simply added [[registry]] field into /etc/containers/registries.conf file simply because I was using crio
[[registry]]
insecure = true
location = "IP ADDRESS"
and restart crio
systemctl restart crio.service
Please refer
https://github.com/cri-o/cri-o/blob/main/docs/crio.conf.5.md