I am trying to understand how Minikube is run on Windows, for the following setup. There are several related questions below, which I hope will help me understand holistically how this works.
Using minikube profile list, I get the following output.
C:\>minikube profile list
|----------|-----------|---------|--------------|------|---------|---------|-------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes |
|----------|-----------|---------|--------------|------|---------|---------|-------|
| minikube | docker | docker | 192.168.49.2 | 8443 | v1.20.7 | Running | 1 |
|----------|-----------|---------|--------------|------|---------|---------|-------|
Is this minikube a container running using my local installation of Docker Desktop? Thus whether it runs on WSL2 or Virtualbox is dependent on how I get my Docker Desktop run?
If I minikube ssh, I get to interact with docker within. From the output below, does it mean that each of the minikube kubernetes component is run as an individual container? Is this an example of docker-in-docker?
C:\>minikube ssh
Last login: Wed Nov 10 14:07:23 2021 from 192.168.49.1
docker#minikube:~$ docker ps --format '{{.Names}}'
k8s_storage-provisioner_storage-provisioner_kube-system_b7c766e9-48fe-45dd-a929-d6fd4b6fcf8b_0
k8s_POD_storage-provisioner_kube-system_b7c766e9-48fe-45dd-a929-d6fd4b6fcf8b_0
k8s_kube-proxy_kube-proxy-4r5hz_kube-system_71dc0877-5a47-4b2c-a106-ee41e5f6a142_0
k8s_coredns_coredns-74ff55c5b-pl7tb_kube-system_6cf31402-c3b4-4d86-8963-8a53e36b7878_0
k8s_POD_kube-proxy-4r5hz_kube-system_71dc0877-5a47-4b2c-a106-ee41e5f6a142_0
k8s_POD_coredns-74ff55c5b-pl7tb_kube-system_6cf31402-c3b4-4d86-8963-8a53e36b7878_0
k8s_kube-scheduler_kube-scheduler-minikube_kube-system_82ed17c7f4a56a29330619386941d47e_0
k8s_kube-apiserver_kube-apiserver-minikube_kube-system_01d7e312da0f9c4176daa8464d4d1a50_0
k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_c7b8fa13668654de8887eea36ddd7b5b_0
k8s_etcd_etcd-minikube_kube-system_c31fe6a5afdd142cf3450ac972274b36_0
k8s_POD_kube-scheduler-minikube_kube-system_82ed17c7f4a56a29330619386941d47e_0
k8s_POD_kube-controller-manager-minikube_kube-system_c7b8fa13668654de8887eea36ddd7b5b_0
k8s_POD_kube-apiserver-minikube_kube-system_01d7e312da0f9c4176daa8464d4d1a50_0
k8s_POD_etcd-minikube_kube-system_c31fe6a5afdd142cf3450ac972274b36_0
docker#minikube:~$
Is a minikube container using the local installation of Docker Desktop?
Minikube is local Kubernetes, focusing on making it easy to learn and develop for Kubernetes.
All you need is Docker (or similarly compatible) container or a Virtual Machine environment, and Kubernetes is a single command away: minikube start.
What you’ll need?
2 CPUs or more
2GB of free memory
20GB of free disk space
Internet connection
Container or virtual machine manager, such as: Docker, Hyperkit, Hyper-V, KVM, Parallels, Podman, VirtualBox, or VMware
I want to point out the last sentence, you can choose from multiple containers or virtual machine managers. Docker is one of the options you could have chosen, and based on your post,your current deployment is using Docker as hypervisor.
Is it running on WSL2 or Virtualbox?
Here is some information about WSL2 and Virtualbox, but the information provided about your environment is not enough to determine if your deployment is in Virtualbox or WSL2.
Virtualbox hardware virtualization option is to allow the virtualization capabilities provided by the processor. This does not help with nested virtualization. You can run in Docker in Virtualbox as long as there is no hypervisor running inside. That is the case when we run Docker on Linux systems in Virtualbox. With Windows server, they run hyperv as well on top of which they run Windows server where Docker runs. That's why nested virtualization is needed here.
With Docker Desktop running on WSL 2, users can leverage Linux workspaces and avoid having to maintain both Linux and Windows build scripts. In addition, WSL 2 provides improvements to file system sharing, boot time, and allows access to some cool new features for Docker Desktop users.
Before you install the Docker Desktop WSL 2 backend, you must complete the following steps:
Install Windows 10, version 1903 or higher or Windows 11.
Enable WSL 2 feature on Windows.
Download and install the Linux kernel update package.
Does each of the minikube kubernetes components run as an individual container?
Minikube is a utility you can use to run Kubernetes on your local machine. It creates a single node cluster contained in a virtual machine (VM). This cluster lets you demo Kubernetes operations without requiring the time and resource-consuming installation of full-blown K8s.
Here are the basic concepts of kubernetes.
Deployment—configured and operational resources. Deployments are the overall processes that enable you to orchestrate your resources.
ReplicaSet—sets of pods that provide the resources for your services.
Pod—a unit that contains one or more containers along with attached storage resources, and configuration definitions. Pods are grouped together in ReplicaSets and all pods in a set run the same container images.
Node cluster—control plane and worker nodes that each contain one or more pods. The workers run your workloads and the control plane orchestrates the workers together. This is what Minikube creates.
Node processes—the various components that you use to connect and manage Kubernetes. Control plane processes include API servers, ectd, Scheduler, kube-controller-manager, and cloud-controller-manager. Worker processes include kubelet, kube-proxy, and your container runtime.
Container—the image you create to hold your applications.
Related
While playing around with Docker and orchestration (kubernetes) I had to install and use minikube to create a simple sandbox environment. At the beginning I thought that minikube installs some kind of VM and run the "minified" kubernetes environment inside the same, however, after the installation listing my local Docker running containers I found minikube running as a container!!
Why minikube itself run as a Docker container? and how can it runs other containers?
Experimental Docker support looks to have been added in minikube 1.7.0, and started becoming the default runtime in minikube 1.9.0. As I'm writing this, current is 1.15.1.
The minikube documentation on the "docker" driver notes, particularly on a native-Linux host, there is not an intermediate virtual machine: if you can run Kubernetes in a container, it can use the entire host system's resources without special configuration or partitioning. The previous minikube-on-VirtualBox installation required preallocating memory and disk to the VM, and it was easy to get those settings wrong. Even on non-Linux hosts, if you're running Docker Desktop, sharing its hidden Linux VM can improve resource utilization, and you don't need to decide to allocate exactly 2 GB RAM to Docker Desktop and exactly 4 GB to the minikube VM.
For a long time it's been possible, but discouraged, to run a separate Docker daemon inside a Docker container; similarly, it's possible, but usually discouraged, to run a multi-process init system in a container. If you do both of these things then you can have the core Kubernetes components (etcd, apiserver, kubelet, ...) inside a single container pretending to be a Kubernetes node. It also helps here that Kubernetes already knows how to pull Docker images, which minimizes some of the confusing issues with running Docker in Docker.
When I run my docker container using Docker Desktop for Windows I am able to connect to it using
docker run -p 5051:5000 my_app
http://0.0.0.0:5051
However when I open another terminal and do this
minikube docker-env | Invoke-Expression
and build and run the same container using the same run command as above
I cannot connect to the running instance.
Should I be running and testing the containers using Docker Desktop, then using minikube to store the images only (for Kubernetes)? Or can you run them and test them as well through minikube?
That's because on your second attempt, the container is not running on the host but on the minikube VM. You'll be able to access it using the minikube VM IP.
To get the minikube ip you can run minikube ip
Why ?
Invoking minikube docker-env sets all the docker env variable on your host to match the minikube environment. This means that when you run a container after that, it is run with the docker daemon on the minikube VM.
I asked you if there are any specific reasons to use Docker Desktop and Minikube together on a single machine as these are two competitive solutions which basically enable you to perform similar tasks and achieve same goals.
This article nicely explains differences between these two tools.
Docker-for-windows uses Type-1 hypervisor, such as Hyper-V, which are
better compared to Type-2 hypervisors, such as VirtualBox, while
Minikube supports both hypervisors. Unfortunately, there are a couple
of limitations in which technology you are using, since you cannot
have Type-1 or Type-2 hypervisors running at the same time on your
machine
If you use Docker Desktop and Minikube at the same time I assume you're using Type-1 hypervisor, such as mentioned Hyper-V, but keep in mind that even if they use the same hypervisor, both tools create their own instances of virtual machine. Basically you are not supposed to use those two tools together expecting that they will work as a kind of hybrid that lets you manage single container environment.
First check what hypervisor you are using exactly. If you're using Hyper-V, simple Get-VM command in Powershell (more details in this article) should tell you what you currently have.
#mario no, I didn't know minikube had a docker daemon until recently
which is why I have both
Yes, Minikube has built in docker environment (in fact it sets everything up, but yes, it also sets up container runtime) so basically you don't need to install docker additionally, and as #Marc ABOUCHACRA already suggested in his answer, Minikube runs the whole environment (single node k8s cluster with docker runtime) on a separate VM. Linux version has an option --vm-driver=none which allows you to use your host container runtime and set-up k8s components on it, but this is not the case with Windows version - here you can only use one of two currently supported hypervisors: Hyper-V or VirtualBox (ref).
I wouldn't say that Docker Destkop runs everything on your host. It also uses Type-1 hypervisor to run the container runtime environment. Please check the Get-VM command on your computer and it should be clear what VMs you have and created by which tool.
I have a Intel Atom Dual Core with 4 GB RAM left over and want to use it to run docker images.
What possible solutions are there for such a local installation? I already found MicroK8s which looks promising, yet wondering which other alternatives there are. Is there maybe a complete distribution focused on only running docker containers?
If I would install MicroK8s, I still have to also manage the Ubuntu installation hosting it. Would be nice to have a distribution that only focuses on running docker containers and updates operating system and docker stuff together, so I know it always works fine together.
If you can run Docker, run Docker's Desktop Kubernetes Cluster.
You also can run minikube (on a top of docker, or hypervisor, or virtualbox)
kind - which is docker in docker k8s cluster.
This is a lab env for playing with Docker containers on Kubernetes without installing nothing: https://labs.play-with-k8s.com/
Minikube: https://github.com/kubernetes/minikube .
Docker Swarm: https://docs.docker.com/engine/swarm/ is an alternative to kubernetes with less features, but easy to setup. (comparison: https://medium.com/faun/kubernetes-vs-docker-swarm-whos-the-bigger-and-better-53bbe76b9d11)
Make your own cluster using VirtualBox: https://medium.com/#KevinHoffman/building-a-kubernetes-cluster-in-virtualbox-with-ubuntu-22cd338846dd
Local-machine Solutions
Community Supported Tools
Minikube is a method for creating a local, single-node Kubernetes cluster for development and testing. Setup is completely automated and doesn’t require a cloud provider account.
Kubeadm-dind is a multi-node (while minikube is single-node) Kubernetes cluster which only requires a docker daemon. It uses docker-in-docker technique to spawn the Kubernetes cluster.
Kubernetes IN Docker is a tool for running local Kubernetes clusters using Docker container “nodes”. It is primarily designed for testing Kubernetes 1.11+. You can use it to create multi-node or multi-control-plane Kubernetes clusters.
Ecosystem Tools
Docker Desktop is an easy-to-install application for your Mac or Windows environment that enables you to start coding and deploying in containers in minutes on a single-node Kubernetes cluster.
Minishift installs the community version of the Kubernetes enterprise platform OpenShift for local development & testing. It offers an all-in-one VM (minishift start) for Windows, macOS, and Linux. The container start is based on oc cluster up (Linux only). You can also install the included add-ons.
MicroK8s provides a single command installation of the latest Kubernetes release on a local machine for development and testing. Setup is quick, fast (~30 sec) and supports many plugins including Istio with a single command.
IBM Cloud Private-CE (Community Edition) can use VirtualBox on your machine to deploy Kubernetes to one or more VMs for development and test scenarios. Scales to full multi-node cluster.
IBM Cloud Private-CE (Community Edition) on Linux Containers is a Terraform/Packer/BASH based Infrastructure as Code (IaC) scripts to create a seven node (1 Boot, 1 Master, 1 Management, 1 Proxy and 3 Workers) LXD cluster on Linux Host.
Ubuntu on LXD supports a nine-instance deployment on localhost.
My very opionated answer: you should use k3s by Rancher Labs https://k3s.io/
Most tutorials I've seen for developing with Kubernetes locally use Minikube. In the latest Edge release of Docker for Windows, you can also enable Kubernetes. I'm trying to understand the differences between the two and which I should use.
Minikube lets you choose the version of Kubernetes you want, can Docker for Windows do that? I don't see a way to configure it.
Minikube has CLI commands to enable the dashboard, heapster, ingress and other addons. I'm not sure why because my undertstanding is that these are simply executing kubectl apply -f http://....
With Minikube I can do a minikube ip to get the cluster IP address for ingress, how can I do this with Docker for Windows?
Is there anything else different that I should care about.
I feel like you largely understand the space, and mostly have answers to your questions already. You might find Docker for Mac vs. Docker Toolbox an informative read, even if it's about the Mac equivalent rather than Windows and about Docker packaged as a VM rather than Kubernetes specifically.
In fact you are stuck with the specific version of Kubernetes the Docker Edge desktop distribution publishes.
is answered in the question.
I believe NodePort-type Services are published on your host's IP address; there isn't an intermediate VM address like there is with Docker Toolbox.
Docker Toolbox and minikube always use a full-blown virtual machine with an off-the-shelf hypervisor. The Docker desktop application might use a lighter-weight virtualization engine if one is available.
Kubernetes can involve some significant background work. If you're using Kubernetes-in-Docker it's hard to "turn off" Kubernetes and still have Docker available; but if you have a separate minikube VM you can just stop it.
Here is a part from Docker documentation:
Kubernetes is only available in Docker for Windows 18.02 CE Edge.
Kubernetes support is not included in Docker for Windows 18.02 CE
Stable. To find out more about Stable and Edge channels and how to
switch between them, see General configuration.
Docker for Windows 18.02 CE Edge includes a standalone Kubernetes
server and client, as well as Docker CLI integration. The Kubernetes
server runs locally within your Docker instance, is not configurable,
and is a single-node cluster.
The Kubernetes server runs within a Docker container on your local
system, and is only for local testing. When Kubernetes support is
enabled, you can deploy your workloads, in parallel, on Kubernetes,
Swarm, and as standalone containers. Enabling or disabling the
Kubernetes server does not affect your other workloads.
See Docker for Windows > Getting started to enable Kubernetes and
begin testing the deployment of your workloads on Kubernetes.
If you have enough CPU and RAM resources, you can easily have both, minikube and docker-for-desktop on the same machine and switch between them by selecting the context, e.g.:
> kubectl config get-contexts
> kubectl config use-context docker-for-desktop
There is no place at the moment to choose the Kubernetes version for docker-for-desktop. It is preselected in the Docker distribution package.
Unfortunately, version of docker inside the minikube is a step behind compared to available on the docker website, and some features could be missing, but if you are interested in Kubernetes environment, it doesn't really matter in most cases.
Some of the features of Kubernetes rely on external resources and could be unavailable for you in case of local installation. For example, you need an ability to create Cloud Load Balancer to use some types of the Ingress or Service objects.
But other resources can be easily started inside the minikube, or docker-for-desktop using kubectl create/apply.
To expose your cluster resources externally, you can use Webhook Relay Ingress controller.
Here is a link to the installation manual.
I guess that minikube needs VirtualBox because of dockers' dependencies, but my current docker version doesn't need it.
So do I still need VirtualBox or another VM to start minikube?
this is only case on window or osx, do need virtualbox, vmware to create minikube vm using boot2docker, in Linux even Linux VM minikube can directly run kubernetes in host docker when choice --vm-driver none, which uses localkube.
Minikube uses the docker machine to manage the kubernetes VM, so it abstracts the plugin architecture that Docker Machine uses to provide a consistent way to manage various VM providers. Minikube embeds VirtualBox and VMware Fusion drivers so there are no additional steps to use them. However, other drivers require an extra binary to be present in the host PATH like KVM, xhyve, HyperV, etc.
Yes you need the virtualbox along with respective driver to run the kubernetes cluster on your machine and it is because of docker dependencies. Minikube started a virtual machine for us(based on our local environment), and a Kubernetes cluster is running in that VM i.e. all your nodes and services are running under the VM box.
You can work on Minikube even without installing VirtualBox.
Minikube also supports a --driver=none option that runs the Kubernetes components on the host and not in a VM. Using this driver requires Docker and a Linux environment but not a hypervisor.
If you’re using the none driver in Debian or a derivative, use the .deb packages for Docker rather than the snap package, which does not work with Minikube. You can download .deb packages from Docker.
You can also use --driver=docker
For more information, you can refer here
Minikube creates a simple local kubernetes cluster having one VM. Minikube needs a hypervisor such as VirtualBox or KVM... to create this VM.
Minikube won't be able to start if VirtualBox (which is the default) is not available.
minikube start
You will get this error:
VBoxManage not found. Make sure VirtualBox is installed and VBoxManage
is in the path
As per minikube Windows Getting started you can now choose between Virtual Box and Hyper-V for Windows, meaning Virtual Box is not mandatory if you have Hyper-V.
Prerequisites
Windows 8 or above
A hypervisor, such as Hyper-V or VirtualBox
Hardware virtualization support must be enabled in BIOS
4GB of RAM
Using minikube start --driver=docker works just fine. The docker engine could act as a virtual machine.