I was able to get into an Azure Kubernetes Service (AKS) node by referring to Connect to Azure Kubernetes Service (AKS) cluster nodes for maintenance or troubleshooting.
I am trying to list the images present in the worker node. Do I need to install anything like nerdctl/crictl in the nodes or is there any other command I can use which is readily available in the nodes?
In short, what's the alternative for Docker commands in AKS worker nodes?
containerd://1.4.9+azure is the CONTAINER-RUNTIME
It seems that you are using it inside the container. Go to the host process with chroot /host and use it.
The image is --image=mcr.microsoft.com/dotnet/runtime-deps:6.0. I used this.
You can try using ctr cli tool which come prepackaged with containerd.
ctr -n <namespace> image list
NOTE: for checking the namespace kindly run
ctr ns list
Check Why and how to use containerd from the command line. I am not sure if it helps or not. But it does have containerd commands to check.
The reference is Debugging Kubernetes nodes with crictl.
Use these commands to check:
sudo crictl --help
sudo crictl ps
sudo crictl images
Related
Is there any way to access local docker images directly (without using 'docker save') with k3s?
Like minikube accesses local docker images after running this command
eval $(minikube docker-env)
A little bit of background.
I have set up a machine using Ubuntu 19.04 as 'master' and raspberry pi as 'worker' using k3s. Now, I want to use a local image to create a deployment on the worker node.
Update
Adding screenshot as said in the comment below.
Screenshot for the image listings
You can start k3s like this sudo k3s server --docker which will use host's Docker rather than containerd. This will make all local images available to k3s and if your ImagePullPolicy is IfNotPresent k3s will use it rather than trying to pull it.
While this doesn't make all Docker images available,, a useful work-around is to export local Docker images and import them to your ctr:
docker save my/local-image:v1.2.3 | sudo k3s ctr images import -
This will make them available on-demand to your k3s cluster.
This is useful for users who cannot get k3s server to work with the --docker flag.
Docker provides a way to run the container using docker run
Or just pull the container image using docker pull
Found a doc showing mapping between docker commands and kubectl.
Can't find docker pull equivalent in this doc.
If there is no any such equivalent to docker pull, then is there any way to just pull an image using kubectl cli.
In short - no, there is not.
And why would there be? Kubernetes is an orchestration tool for Docker, it will automatically pull pods for you if it needs them, so there's really no need to have a command to pull containers manually.
I think there isn't a kubectl ... equivalent and some of the reasons might be:
they are not equivalent š. When you docker pull an image, you are planning to use it afterwards on your docker host. When you kubectl ... a deployment, you want the platform to schedule everything. For example if you have many worker nodes and the replicas are going to be scheduled to only two of them, then the other nodes don't have to pull the image.
kubectl is a tool that talks to the API server to control the cluster. It would be wrong to make it also responsible for container images (see, Leaky Abstractions) since you have available a lower level tool that talks to the Container Runtime Interface for that: crictl.
k8s-master:~$ crictl --help
NAME:
crictl - client for CRI
USAGE:
crictl [global options] command [command options] [arguments...]
VERSION:
v1.12.0
COMMANDS:
attach Attach to a running container
create Create a new container
exec Run a command in a running container
version Display runtime version information
images List images
inspect Display the status of one or more containers
inspecti Return the status of one or more images
inspectp Display the status of one or more pods
logs Fetch the logs of a container
port-forward Forward local port to a pod
ps List containers
pull Pull an image from a registry
...
pic from: www.aquasec.com/wiki/display/containers/Kubernetes+Architecture+101
what takes place with container run-times under the hood is complicated and keeps evolving. Think about this, people started creating Kubernetes clusters and the container engine used was Docker. Then Docker adopted containerd so we had Kubernetes on top of Docker on top of containerd, which caused problems like this:
Users won't see Kubernetes pulled images with the docker images command... And vice versa, Kubernetes won't see images created by docker pull, docker load or docker build commands...
source / more details: Kubernetes Containerd Integration Goes GA
crictl pull <image name>
There is no need to pull by kubernetes in cli.
Why?
Becuase when you run kubectl create -f template.yml it containe an image and it cjecked that the image is exist or not. If it does not exist it pull image automatically.
You will not find equivalent of docker pull in Kubernetes because this command is related to images management. Explanation below.
One of Docker features is abbility to create Images. You can create your own image using Dockerfile (docker build .) or pull from Docker Hub which contains many pre-built images.
If you use pull command it will just download image, it will not deploy any container.
$ docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest fce289e99eb9 5 months ago 1.84kB
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
As you see, $ docker pull will only display download image. As Docker is also responsible for image management you can pull or push images to repository (DockerHub).
To create container in Docker you have to use $ docker run. This command will automatically download image and run container.
$ docker run --name mynginx -p 80:80 -d nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
...
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4abf804611a8 nginx "nginx -g 'daemon ofā¦" 4 minutes ago Up 4 minutes 0.0.0.0:80->80/tcp mynginx
In short:
Create adds a writeable container on top of your image and sets it up for running whatever command you specified in your CMD. The container ID is reported back but itās not started.
Start will start any stopped containers. This includes freshly created containers.
Run is a combination of create and start. It creates the container and starts it.
Kubernetes is container-orchestration system so it is not responsible for creating or edit images. That is why you will not find equivalent of docker pull (download only image).
Commands like kubectl apply -f <deployment> with image inside YAML file or kubectl run nginx --image=nginx are based on images from DockerHub (more like docker create).
Hope it helped.
It could be a bit tricky, but it is possible to achieve similar to docker pull results using kubectl. You just need to know how to exit the containers with zero exit code.
The idea is to pull several images on all nodes in the Kubernetes cluster.
For doing this you could create a DaemonSet which will try to create Pods on every applicable node in the cluster. To pull several images at once, just add several initContainers to the DaemonSet template. ImagePullPolicy set to IfNotPresent, restartPolicy set to Never. Set command individually to each initContainer to make it exit successfully. You need something like sh -c "exit 0", just ensure that container has sh binary inside, or use another possible command that usually gives you zero exit code: <appname_binary> version or <appname_binary> --help.
After applying that DaemonSet to the cluster, Kubernetes creates Pods from DaemonSet templates on each node and runs each initContainer in the Pod in order of presence. Before starting each container kubelet pulls the image required to run that container.
When you see that all DaemonSet Pods completed successfuly - you can be sure that on every node you have all images, required for running those containers.
You can play with nodeAffinity or taints/tolerations if you want to run the DaemonSet only on specific nodes.
I am currently working on the following scenario
I am trying to setup a container in OpenShift that runs a Jenkins that is itsself able to run docker to make use of declarative pipelines where the build is running in it's own docker container. This basically makes it necessary to install and run docker inside this container.
I have been working on it on quite some time now. Checked dozens of posts and threads online but I have not been able to accomplish it. Basically I got so far
I can install docker in my container (from the baseimage openshift/jenkins-2-centos7:latest)
I can't get docker to run as this makes use of systemctl which
Now I read that systemctl is not working inside docker containers or at least highly unrecommended as it interferes with the PID 1 in the system. Without
systemctl start docker
that will leave me with docker beeing unable to connect with the daemon (as expected) and the error message
Can't connect to docker daemon. Is 'docker -d' running on this host?
So I tried to set up the daemon myself using
the follwoing in my Dockerfile
RUN usermod -aG docker $(whoami)
RUN dockerd -H unix:///var/run/docker.sock
which will also not work telling me that cgroups cannot be mounted. After some more research I found that this could be handled with the cgroupfs-mount script from
https://github.com/tianon/cgroupfs-mount/tree/master
But also here I got no luck leaving me with the following error
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.4.21: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
Now after hours I am out of ideas. Does anyone have an idea how to make docker work inside of OpenShift? Would be really greatful
I am trying to setup a container in OpenShift that runs a Jenkins that is itsself able to run docker to make use of declarative pipelines where the build is running in it's own docker container. This basically makes it necessary to install and run docker inside this container.
I don't think your conclusion here is the only possibility, and what I'll describe below is an easier approach to get what (I think) you want! :) If there are any other use cases that you have than these 3 I'll describe, let me know and I'll try to update to cover them:
Pipelines running in their own containers
Running additional containers from Pipelines
Building container images from Pipelines
Pipelines running in their own containers
For this case, there's the excellent Kubernetes plugin.
With this plugin, you add a Kubernetes/OpenShift cloud to the Jenkins global config. This can either be the one in which Jenkins is running (if you use the Jenkins image provided by OpenShift, this gets added by default at least), or an external cluster.
Inside that configuration, you can define PodTemplates (again, there are a couple of examples provided in the Jenkins image provided by OpenShift), or you can specify that in your pipeline directly also I think. When your pipeline requests a node/agent with a label that matches one of these (and there are no long-running agents that match), then a pod will be created from that template, and your pipeline execution will happen inside a container in that. Once it's no longer needed, it will be deprovisioned again.
Here are the pipeline steps exposed by this plugin: https://jenkins.io/doc/pipeline/steps/kubernetes/
Running additional containers from Pipelines
As part of your pipeline, you may want to run some tests, and those may expect to be able to interact with e.g. a database. You can create resources for that in your OpenShift project (e.g. a Deployment & expose it with a Service), and tear them down after. The openshift-client plugin is very useful here and has docs on how to interact with OpenShift.
Building container images from Pipelines
If your goal is to build container images from pipelines, remember that OpenShift also exposes this capability (depending on the security configuration) through Builds. Just like in the previous section, you can use the openshift-client plugin to create and trigger builds.
For more information on the Jenkins image that's maintained by OpenShift (and generally how to do useful things in Jenkins on OpenShift), there's this dedicated page in the OpenShift docs.
You have this article by #jpetazzo, from Docker team, about Docker In Docker (DinD):
article:
The primary purpose of Docker-in-Docker was to help with the development of Docker itself. Many people use it to run CI (e.g. with Jenkins), which seems fine at first, but they run into many āinterestingā problems that can be avoided by bind-mounting the Docker socket into your Jenkins container instead.
DinD Repo:
This work is now obsolete, thanks to the combined efforts of some amazing people like #jfrazelle and #tianon, who also are black belts in the art of putting IKEA furniture together.
If you want to run Docker-in-Docker today, all you need to do is:
docker run --privileged -d docker:dind
So here is an article using another approach to build docker containers with Jenkins inside a docker container:
docker run -p 8080:8080 \
-v /var/run/docker.sock:/var/run/docker.sock \
--name jenkins \
jenkins/jenkins:lts
So you may want to adapt one of this solutions to your OpenShift scenario. I hope it solves your issue.
You'll need a privileged pod running jenkins wich mounts the openshift node docker socket. This is a bad idea as jenkins'll launch container outside kubernetes semantics and control.
Why do not use s2i service shipped with openshift ?
Regards.
I have a compute cluster of 16 nodes running centos 6.7, with each node having a local disk and a shared storage between all nodes which is FhGFS based. the shared path is '/cluster'.
How to install Docker so that the image repository is allocated on /cluster, and any node could run containers from that repo. Is there a way to allocate the image repo in the shared area, while installing only the docker engine on each of the nodes? or even better, installing both the image repo and the engine on the shared area and making this installation usable by all nodes?
You can just modify your docker daemon configs to have the runtime root be /cluster
docker daemon --graph="/cluster"
or
docker daemon -g "/cluster"
Say you are using CentOS or RHEL you could add these options under
/etc/sysconfig/docker
If you are using Debian or Ubuntu you would change:
/etc/defaults/docker
So this way all the pulls that you do for images will be stored under /cluster also all your container runtimes will be under /cluster. So if you mount /cluster on all your machines then all of them will be able to see them.
If you want to share the binary, just put it under say /cluster/bin and then add it to your $PATH.
You might also want to look at Docker Swarm which is Docker's native clustering support. Although not ready for primetime as of Today, it's worth looking at.
What's the procedure for installing and running Docker on Google Compute Engine?
Until the recent GA release of Compute Engine, running Docker was not supported on GCE (due to kernel restrictions) but with the newly announced ability to deploy and use custom kernels, that restriction is no longer intact and Docker now works great on GCE.
Thanks to proppy, the instructions for running Docker on Google Compute Engine are now documented for you here: http://docs.docker.io/en/master/installation/google/. Enjoy!
They now have a VM which has docker pre-installed now.
$ gcloud compute instances create instance-name
--image projects/google-containers/global/images/container-vm-v20140522
--zone us-central1-a
--machine-type f1-micro
https://developers.google.com/compute/docs/containers/container_vms
A little late, but I wanted to add an answer with a more detailed workflow and links, since answers are still rather scattered:
Create a Docker image
a. Locally
b. Using Google Container Builder
Push local Docker image to Google Container Repository
docker tag <current name>:<current tag> gcr.io/<project name>/<new name>
gcloud docker -- push gcr.io/<project name>/<new name>
UPDATE
If you have upgraded to Docker client versions above 18.03, gcloud docker commands are no longer supported. Instead of the above push, use:
docker push gcr.io/<project name>/<new name>
If you have issues after upgrading, see more here.
Create a compute instance.
This process actually obfuscates a number of steps. It creates a virtual machine (VM) instance using Google Compute Engine, which uses a Google-provided, container-optimized OS image. The image includes Docker and additional software responsible for starting our docker container. Our container image is then pulled from the Container Repository, and run using docker run when the VM starts. Note: you still need to use docker attach even though the container is running. It's worth pointing out only one container can be run per VM instance. Use Kubernetes to deploy multiple containers per VM (the steps are similar). Find more details on all the options in the links at the bottom of this post.
gcloud beta compute instances create-with-container <desired instance name> \
--zone <google zone> \
--container-stdin \
--container-tty \
--container-image <google repository path>:<tag> \
--container-command <command (in quotes)> \
--service-account <e-mail>
Tip You can view available gcloud projects with gcloud projects list
SSH into the compute instance.
gcloud beta compute ssh <instance name> \
--zone <zone>
Stop or Delete the instance. If an instance is stopped, you will still be billed for resources such as static IPs and persistent disks. To avoid being billed at all, use delete the instance.
a. Stop
gcloud compute instances stop <instance name>
b. Delete
gcloud compute instances delete <instance name>
Related Links:
More on deploying containers on VMs
More on zones
More create-with-container options
As of now, for just Docker, the Container-optimized OS is certainly the way to go:
gcloud compute images list --project=cos-cloud --no-standard-images
It comes with Docker and Kubernetes preinstalled. The only thing it lacks is the Cloud SDK command-line tools. (It also lacks python3, despite Google's announce of Python 2 sunset on 2020-01-01. Well, it's still 27 days to go...)
As an additional piece of information I wanted to share, I was searching for a standard image that would offer both docker and gcloud/gsutil preinstalled (and found none, oops). I do not think I'm alone in this boat, as gcloud is the thing you could hardly go by without on GCEĀ¹.
My best find so far was the Ubuntu 18.04 image that came with their own (non-Debian) package manager, snap. The image comes with the Cloud SDK preinstalled, and Docker installs literally in a snap, 11 seconds on an F1 instance initial test, about 6s on an n1-standard-1. The only snag I hit was the error message that the docker authorization helper was not available; an attempt to add it with gcloud components install failed because the SDK was installed as a snap, too. However, the helper is actually there, only not in the PATH. The following was what got me the both tools available in a single transient builder VM in the least amount of setup script runtime, starting off the supported Ubuntu 18.04 LTS imageĀ²:
snap install docker
ln -s /snap/google-cloud-sdk/current/bin/docker-credential-gcloud /usr/bin
gcloud -q auth configure-docker
Ā¹ I needed both for a Daisy workflow imaging a disk with both artifacts from GS buckets and a couple huge, 2GB+ library images from the local gcr.io registry that were shared between the build (as cloud builder layers) and the runtime (where I had to create and extract containers to the newly built image). But that's besides the point; one may needs both tools for a multitude of possible reasons.
Ā² Use gcloud compute images list --uri | grep ubuntu-1804 to get the most current one.
Google's GitHub site offers now a gce image including docker. https://github.com/GoogleCloudPlatform/cloud-sdk-docker-image
It's as easy as:
creating a Compute Engine instance
curl https://get.docker.io | bash
Using docker-machine is another way to host your google compute instance with docker.
docker-machine create \
--driver google \
--google-project $PROJECT \
--google-zone asia-east1-c \
--google-machine-type f1-micro $YOUR_INSTANCE
If you want to login this machine on google cloud compute instance, just use docker-machine ssh $YOUR_INSTANCE
Refer to docker machine driver gce
There is now improved support for containers on GCE:
Google Compute Engine is extending its support for Docker containers. This release is an Open Preview of a container-optimized OS image that includes Docker and an open source agent to manage containers. Below, you'll find links to interact with the community interested in Docker on Google, open source repositories, and examples to get started. We look forward to hearing your feedback and seeing what you build.
Note that this is currently (as of 27 May 2014) in Open Preview:
This is an Open Preview release of containers on Virtual Machines. As a result, we may make backward-incompatible changes and it is not covered by any SLA or deprecation policy. Customers should take this into account when using this Open Preview release.
Running Docker on GCE instance is not supported. The instance goes down and not able to login again.
We can use the Docker image given by the GCE, to create a instance.
If your google cloud virtual machine is based on ubuntu use the following command to install docker
sudo apt install docker.io
You may use this link: https://cloud.google.com/cloud-build/docs/quickstart-docker#top_of_page.
The said link explains how to use Cloud Build to build a Docker image and push the image to Container Registry. You will first build the image using a Dockerfile and then build the same image using the Cloud Build's build configuration file.
Its better to get it while creating compute instance
Go to the VM instances page.
Click the Create instance button to create a new instance.
Under the Container section, check Deploy container image.
Specify a container image name under Container image and configure options to run the container if desired. For example, you can specify gcr.io/cloud-marketplace/google/nginx1:1.12 for the container image.
Click Create.
Installing Docker on GCP Compute Engine VMs:
This is the link to GCP documentation on the topic:
https://cloud.google.com/compute/docs/containers#installing
In it it links to the Docker install guide, you should follow the instructions depending on what type of Linux you have running in the vm.