Getting "x509: certificate signed by unknown authority" by microk8s - docker

Question

I would like to use microk8s with private registry, but pull image is not working (I'm using self-signed cert):
root#master-1:/var/snap/microk8s/common/var/lib/containerd# microk8s.ctr --debug images pull priv.repo:5000/busybox/hellomicrok8s:latest
DEBU[0000] fetching image="priv.repo:5000/busybox/hellomicrok8s:latest"
DEBU[0000] resolving host="priv.repo:5000"
DEBU[0000] do request host="priv.repo:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.4 request.method=HEAD url="https://priv.repo:5000/v2/busybox/hellomicrok8s/manifests/latest"
ctr: failed to resolve reference "priv.repo:5000/busybox/hellomicrok8s:latest": failed to do request: Head "https://priv.repo:5000/v2/busybox/hellomicrok8s/manifests/latest": x509: certificate signed by unknown authority
here is my containerd-template.tom:
root#master-1:/var/snap/microk8s/common/var/lib/containerd# cat /var/snap/microk8s/current/args/containerd-template.toml
version = 2
oom_score = 0
[grpc]
uid = 0
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
[debug]
address = ""
uid = 0
gid = 0
[metrics]
address = "127.0.0.1:1338"
grpc_histogram = false
[cgroup]
path = ""
[plugins."io.containerd.grpc.v1.cri"]
stream_server_address = "127.0.0.1"
stream_server_port = "0"
enable_selinux = false
sandbox_image = "k8s.gcr.io/pause:3.1"
stats_collect_period = 10
enable_tls_streaming = false
max_container_log_line_size = 16384
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "${SNAPSHOTTER}"
no_pivot = false
default_runtime_name = "${RUNTIME}"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime]
runtime_type = "io.containerd.runc.v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime.options]
BinaryName = "nvidia-container-runtime"
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "${SNAP}/opt/cni/bin"
conf_dir = "${SNAP_DATA}/args/cni-network"
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io", ]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."priv.repo:5000"]
endpoint = ["https://priv.repo:5000"]
I restarted microk8s via systemctl restart snap.microk8s.daemon-containerd.service && microk8s.stop && microk8s.start.
Command docker login docker https://priv.repo:5000 is working and I can pull that image via docker pull priv.repo:5000/busybox/hellomicrok8s:latest. Do you know why it is not working?
Thanks in advance!
EDIT:
This is also set:
root#master-1:/var/snap/microk8s/common/var/lib/containerd# cat /etc/docker/daemon.json
{
"insecure-registries" : ["priv.repo:5000"]
}
EDIT1:
This is working: microk8s.ctr --debug images pull -u ???:??? --skip-verify priv.repo:5000/busybox/hellomicrok8s:latest. How should I set --skip-verify, because when I create a pod via microk8s kubectl apply -f ... still getting x509: certificate signed by unknown authority.

Answer 1

I added my crt file to /etc/ssl/certs (on master node) and it started working.
BTW newly added rows in containerd-template.tom file are not needed for me.

Answer 2

if you are using ubuntu microk8s cert-manager, you can fetch the certificate and install it like this:
Find the correct certificates name (you could have multiple)
microk8s kubectl get secrets -n cert-manager --field-selector type=kubernetes.io/tls
if the correct name is e.g. dev-da
microk8s kubectl -n cert-manager get secrets dev-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > cert-manager-ca.crt
sudo cp cert-manager-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
Hereafter, you can check with curl, if the certificate is installed correctly.
And when genstart microk8s.
microk8s stop && microk8s start

Answer 3

I had the same issue and these commands below may fix this issue for others
openssl s_client -showcerts -connect <IP>:<PORT>< /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ca.crt
cp ca.crt /etc/ssl/certs
update-ca-certificates