Kubeflow could not be enabled: mutatingwebhookconfigurations.admissionregistration.k8s.io "katib-mutating-webhook-config" not found - kubeflow

I have installed microk8s on Windows using the installer. I have enabled some services with microk8s enable dashboard dns registry. But when I try to enable kubeflow I get the following error:
C:\Users\biogeek>microk8s enable kubeflow
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling metallb:10.64.140.43-10.64.140.49...
Waiting for DNS and storage plugins to finish setting up
Deploying Kubeflow...
Kubeflow deployed.
Waiting for operator pods to become ready.
Waited 0s for operator pods to come up, 34 remaining.
Waited 15s for operator pods to come up, 34 remaining.
[...]
Waited 390s for operator pods to come up, 2 remaining.
Operator pods ready.
Waiting for service pods to become ready.
Kubeflow could not be enabled:
Error from server (NotFound): mutatingwebhookconfigurations.admissionregistration.k8s.io "katib-mutating-webhook-config" not found
Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "katib-validating-webhook-config" not found
Command '('microk8s-kubectl.wrapper', 'delete', 'mutatingwebhookconfigurations/katib-mutating-webhook-config', 'validatingwebhookconfigurations/katib-validating-webhook-config')' returned non-zero exit status 1
Failed to enable kubeflow
An error occurred when trying to execute 'sudo microk8s.enable kubeflow' with 'multipass': returned exit code 1.
(Error is similar to the one in this Github issue from 2 weeks ago, but which hasn't had any response yet.)
Output of microk8s status --wait-ready:
microk8s is running
addons:
dashboard: enabled
dns: enabled
ingress: enabled
kubeflow: enabled
metallb: enabled
storage: enabled
cilium: disabled
fluentd: disabled
gpu: disabled
helm: disabled
helm3: disabled
istio: disabled
jaeger: disabled
knative: disabled
linkerd: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: disabled
When I run microk8s inspect to troubleshoot the issue, it generates a logfile at /var/snap/microk8s/1702/... although I am on Windows!?
C:\Users\biogeek>microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Service snap.microk8s.daemon-etcd is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Building the report tarball
Report tarball is at /var/snap/microk8s/1702/inspection-report-20201123_042132.tar.gz
How can I resolve this and successfully install Kubeflow on Windows using Microk8s?

According to this comment on #1698, a fix for this issue is included in microk8s 1.20/stable.
sudo snap install microk8s --classic --channel=1.20/edge

Related

GitLab Pipeline / Runner fails after update: failed adding cache volume ... AppArmor enabled ... docker-default profile could not be loaded

After updating GitLab and GitLab Runner to v15.8.1 on a Ubuntu 20.04.05 LTS suddenly none of our pipelines succeeded anymore.
Every job showed the following error message:
ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "XXXXXXX" for volume "runner-zx-dagtx-project-41-concurrent-0-cache-YYYYYYY": starting permission container: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running apparmor_parser apparmor_parser --version failed with output:
error: exec: "apparmor_parser": executable file not found in $PATH (linux_set.go:105:0s)
Google miss leaded me to re-installations of Docker and the GitLab Runner, to deleting all docker volumes, images and containers and so on. All without success.
How to get GitLab Runner back to work?
What fixed the problem was the installation of AppArmour and the AppArmour Utilities.
sudo apt-get install apparmor apparmor-utils
Another solution would have been to disable AppArmour during the boot process by adding apparmor=0 as boot parameter to the grub file (what I didn't try, so no guarantee for that).

Docker socket is not found while using Intellij IDEA and Docker desktop on MacOS

I downloaded Docker using Docker Desktop for Apple M1 chips. I can run containers, the integration with VsCode works okay but I can't integrate it with Intellij IDEA Ultimate. It keeps giving this error.
But I can run my containers and create images from the terminal, I can also see the containers and images in Docker Desktop too. What could be the reason behind this? I also tried to check whether var/run/docker.sock is existing and it really isn't, there is no such file as that.
I also tried the same steps on my second computer and the exact same thing happened. Steps to reproduce: 1- Download Intellij IDEA Ultimate, open a repo that uses docker 2- Download Docker Desktop for Mac M1 3- Try to add Docker service to Intellij
I didn't do anything else because I think Docker Desktop is enough to configure everything on Mac. I am trying to run an FT on intellij and I get the error
[main] ERROR o.t.d.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
[main] ERROR o.t.d.DockerClientProviderStrategy - UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock)
[main] ERROR o.t.d.DockerClientProviderStrategy - DockerMachineClientProviderStrategy: failed with exception ShellCommandException (Exception when executing docker-machine status ). Root cause InvalidExitValueException (Unexpected exit value: 1, allowed exit values: [0], executed command [docker-machine, status, ], output was 122 bytes:
Docker machine "" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.)
[main] ERROR o.t.d.DockerClientProviderStrategy - As no valid configuration was found, execution cannot continue
I've been trying everything for the last 2 days but I can't seem to find a solution.
EDITED 2022-10-31
As per the latest release notes for Docker Desktop (4.13.1), there is no need to create the symlink anymore, citing notes:
Added back the /var/run/docker.sock symlink on Mac by default, to increase compatibility with tooling like tilt and docker-py. Fixes docker/for-mac#6529.
The official fix now is to UPGRADE your Docker Desktop installation.
For the Docker Desktop (4.13.0) version:
By default Docker will not create the /var/run/docker.sock symlink on the host and use the docker-desktop CLI context instead. (see: https://docs.docker.com/desktop/release-notes/)
That will prevent IntelliJ from finding Docker using the default context.
You can see the current contexts in your machine by running docker context ls, which should produce an output like:
NAME TYPE DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default moby Current DOCKER_HOST based configuration unix:///var/run/docker.sock https://kubernetes.docker.internal:6443 (default) swarm
desktop-linux * moby unix:///Users/<USER>/.docker/run/docker.sock
As a workaround that will allow IntelliJ to connect to Docker you can use the TCP Socket checkbox and put in the Engine API URL the value that appears under DOCKER ENDPOINT in the active context.
The case for this example will be: unix:///Users/<USER>/.docker/run/docker.sock
Then IntelliJ will be able to connect to Docker Desktop.
**Hacky option**
Another way to make IntelliJ (and other components that rely on the default config) to find Docker will be to manually create a symlink to the new DOCKER ENDPOINT by running:
sudo ln -svf /Users/<USER>/.docker/run/docker.sock /var/run/docker.sock
In that way all the components looking for Docker under /var/run/docker.sock will find it.

Why doesn't TeamCity recognize docker server properties with the jetbrains/agent image?

I have an ECS Fargate service running the jetbrains/teamcity-agent image. This is connected to my TeamCity Host which is running on an EC2 instance(windows).
When I check whether the agent is capable of running docker commands, it shows the following errors:
Unmet requirements:
docker.server.osType contains linux
docker.server.version exists
Under Agent Parameters -> Configuration Parameters, I can see the docker version and the dockerCompose.version properly. Is there a setting that I am missing?
If you are trying to access a docker socket in fargate, Fargate does not support running docker commands, there is a proposed ticket for this feature.
the issue with "docker.server.osType" not showing up usually means
that the docker command run from the agent cannot connect with the
docker daemon running. This is usually due to a lack of permissions,
as docker by default only allows connections from root and users of
the group docker
Teamcity-Unmet-requirements-docker-server-osType-contains-linux
I was facing similar issues got them fixed by adding "build agent" user in "docker" group and restarted/rebooted the server.
Where build agent user ==> Means the user with which your TeamCity services are running.
Command to add a user to group
#chmod -a -G docker <userasperyourrequirement>
Command to reboot the server:
#init 6

Docker containers fail to startup > FATA 008 Failed to initialize local MSP

I'm trying to launch a network on hyperledger fabric which is almost identical to the 'first-network' fabric-sample. At the point where I start my fabric the dockers (apart from the cli) exit immediately (see image).
When I inspect the logs I see that the Orderer fails to initialize local MSP because it cannot load a valid signer certificate (see image).
I tried cleaning the docker images and downloading them again same for the binaries.
docker-compose-base.yaml
docker-compose-cli.yaml
Network composes of:
3 Peers, 1 Orderer and the cli
I would be really thankfull If anyone knows how to deal with this.
EDIT: Solved this issue by resetting docker to factory settings. I assume it had something to do with my docker login credentials.

Docker Login: Error when manually logging into private Registry

I can't manually log into my private GitLab Docker Registry from CLI:
# docker login -u "${DOCKER_USER}" -p "${DOCKER_PASS}" "${DOCKER_URL}"
error getting credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`
System info:
Ubuntu 18.04
docker-ce 18.03.1~ce~3-0~ubuntu (from official repo, without install script)
There is no ~/.docker/config.json for any users and I'm executing the docker login as root.
On Google, I just find recommendations to export DISPLAY... Can docker only login to remote registries in a GUI environment?
Exporting DISPLAY=0:0 yields:
error getting credentials - err: exit status 1, out: `Failed to execute child process “dbus-launch” (No such file or directory)`
Am I missing some dependency? Docker runs fine otherwise, but login doesn't work. I know there are backends to store credentials, but I don't want to store credentials. I'm just trying to authenticate against my registry to pull an image, doesn't that work in Docker ootb?
The docker-compose package unnecessarily depend on the broken golang-github-docker-docker-credential-helpers package. Removing the executable fixes this.
rm /usr/bin/docker-credential-secretservice
Note: This is a workaround and will need to be repeated each time the package is updated.
This affects the Ubuntu 18.04 (and possibly other non-LTS releases) and some Debian releases.

Resources