how to check running docker containers and images for the deployed pods inside aks nodes?
I have deployed company pods and services inside the azure aks cluster.
Need to login as a root user inside containers running inside nodes of managed aks cluster. Those containers are of rabbitmq pods deployed with bitnami helm chart.
I was able to login into worker nodes by following this link https://learn.microsoft.com/en-us/azure/aks/node-access, but couldn't find the docker package running/installed inside them.
They do have containerd://1.4.9+azure as the CONTAINER-RUNTIME.
Tried below commands of 'containerd' inside those nodes, but nothing came, empty response, no running containers or downloaded images.
ctr container ls
ctr images ls
So how to check running docker containers and images for the deployed pods inside aks nodes?
We can use the below commands to check running containers inside worker nodes of managed k8s clusters at least in the case of aks clusters.
Here's the reference link https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/
use below commands
sudo crictl --help
sudo crictl ps
sudo crictl images
I use nerdctl instead. Here are the steps work for me..
Create a debug pod to access the aks node
kubectl debug node/<nodeName> -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0
You can interact with the node session by running below command:
chroot /host
Install nerdctl tool:
wget https://github.com/containerd/nerdctl/releases/download/v0.20.0/nerdctl-0.20.0-linux-amd64.tar.gz
tar zxvf nerdctl-0.20.0-linux-amd64.tar.gz -C /usr/local/bin/
alias n=nerdctl
Check the containers in the AKS node
nerdctl --namespace k8s.io ps -a
Reference:
https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/
https://learn.microsoft.com/en-us/azure/aks/node-access
https://github.com/containerd/nerdctl#whale-nerdctl-load
Related
I am trying to setup a docker swarm on WSL2 for testing purposes. I want to know, if it is possible to have a swarm with multiple "dummy" nodes on a single machine.
Here are the two ways that I trid:
Run multiple WSL instances as suggested here.
PS C:\Users\jdu> wsl -l
Windows-Subsystem für Linux-Distributionen:
Ubuntu3
Ubuntu
Ubuntu2
Docker is installed and run in each WSL instance. So I manage to initialize a swarm on Ubuntu and let Ubuntu2 and Ubuntu3 to join as workers.
On Ubuntu
$ docker swarm init
Swarm initialized: current node (hude19jo7t9dqpe0akg55ipmy) is now a manager.
On Ubuntu2
$ docker swarm join --token SWMTKN-1-xxxxxxxxx-xxxxxxxxx 192.168.189.5:2377 --listen-addr 0.0.0.0:12377
This node joined a swarm as a manager.
Then if I check on Ubuntu
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
hude19jo7t9dqpe0akg55ipmy * laptop-ebc155 Ready Active Leader 20.10.21
ozeq43yukgfbltjnfya0tlx08 laptop-ebc155 Ready Active Reachable 20.10.20
Inspired by the ideas here, I have tried with docker-in-docker containers, e.g. I deploy multiple docker instances on a single WSL.
# Init Swarm master
docker swarm init
# Get join token:
SWARM_TOKEN=$(docker swarm join-token -q worker)
echo $SWARM_TOKEN
# Get Swarm master IP (Docker for Mac xhyve VM IP)
SWARM_MASTER_IP=$(docker info | grep -w 'Node Address' | awk '{print $3}')
echo $SWARM_MASTER_IP
DOCKER_VERSION=dind
# setup deploy Docker-in-Docker containers and join them to a swarm
docker run -d --privileged --name worker-1 --hostname=worker-1 -p 12377:2377 docker:${DOCKER_VERSION}
docker exec worker-1 docker swarm join --token ${SWARM_TOKEN} ${SWARM_MASTER_IP}:2377
docker run -d --privileged --name worker-2 --hostname=worker-2 -p 22377:2377 docker:${DOCKER_VERSION}
docker exec worker-2 docker swarm join --token ${SWARM_TOKEN} ${SWARM_MASTER_IP}:2377
docker run -d --privileged --name worker-3 --hostname=worker-3 -p 32377:2377 docker:${DOCKER_VERSION}
docker exec worker-3 docker swarm join --token ${SWARM_TOKEN} ${SWARM_MASTER_IP}:2377
After that
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
s371tmygu9h640xfosn6kyca4 * laptop-ebc155 Ready Active Leader 20.10.21
w1ina9ttvje4hn6r13p3gzbge worker-1 Ready Active 20.10.20
m8mqky6jchjao01nz8t5e392a worker-2 Ready Active 20.10.20
n29afhbb090tlyn9p0byga9au worker-3 Ready Active 20.10.20
To test the above two swarm setup, I use a very simple compose file as suggested by the official docs. As you can expect, these two swarm setup didn't work that well :/
If the MongoDB and MongoExpress are deployed on different nodes, both of the swarm setups show a same error MongoNetworkError: failed to connect to server [mongo:27017] on first connect. My understanding to this error is, that MongoExpress can not reach MongoDB under mongo:27017, which seems like a problem of the docker internal DNS. Can someone help me out? Or just feel free to tell me, dont try this single-multi nodes ideas anymore :D I am very appreciate to any help!
I just tried the same two exercises :)
Approach 1 - swarm nodes in WSL instances
I think it is currently impossible because of WSL2 design see https://github.com/microsoft/WSL/issues/4304. WSL2 instances are in fact sharing network setup - ip, interfaces, network namespaces, and so on. Every change made in one of them is immediately visible in all others and this conflicts with virtual interfaces and namespaces created by docker swarm nodes when they start up.
I tried configuring multiple ip addresses on eth0 interface, so that each node can have it's own (like here), and then used --advertise-addr --listen-addr options in docker swarm init and docker swarm join commands. Still I'm getting this error in dockerd logs:
moving interface ov-001000-yis5e to host ns failed, invalid argument, after config error error setting interface \"ov-001000-yis5e\" IP to 10.0.0.1/24: cannot program address 10.0.0.1/24 in sandbox interface because it conflicts with existing route {Ifindex: 4 Dst: 10.0.0.0/24 Src: 10.0.0.1 Gw: <nil> Flags: [] Table: 254}"
I believe here docker swarm hits a problem, because it already sees master's interfaces when it tries to to set up routing mesh networking for the worker. All because master and node share network config.
Approach 2 - swarm nodes as docker containers (docker-in-docker)
But I've got no 2. working with just a small change in swarm init command:
# advertise swarm on default bridge network
docker swarm init --advertise-addr 172.17.0.1
For me, the standard docker swarm init selected by default the eth0 address, which was only working for communication from dind -> wsl, but not the other way round.
Another but probably unrelated problem was that I could not access services/stacks executed this way from Windows host. This seems to be a wls bug and luckily there is a workaround.
One last hint about this mongo stack is ... patience. The stack consists of 2 services: mongo - the database and mongo-express - the client. Mongo image is a lot bigger ~600MB while mongo-express just ~135MB. The mongo-express image will be downloaded faster and it will be recreated by swarm multiple times before mongo is even started. Note also that docker images are independently downloaded for each worker in this setup, so also rebalancing may take some time.
I found these commands useful to see what is really happening:
# overview of services
docker service ls
# containers in each swarm service
docker service ps $(docker service ls --format {{.Name}})
# images in each dind worker
for i in $(seq "${NUM_WORKERS}"); do
docker exec worker-${i} docker images
done
#containers in each dind worker
for i in $(seq "${NUM_WORKERS}"); do
docker exec worker-${i} docker ps -a
done
Full listing of commands necessary to get working docker swarm using dind:
docker swarm init --advertise-addr docker0
SWARM_TOKEN=$( docker swarm join-token -q worker)
echo $SWARM_TOKEN
SWARM_MASTER_IP=$( docker info 2>&1 | grep -w 'Node Address' | awk '{print $3}')
echo $SWARM_MASTER_IP
DOCKER_VERSION=20.10.12-dind
NUM_WORKERS=3
# Run NUM_WORKERS workers with SWARM_TOKEN
for i in $(seq "${NUM_WORKERS}"); do
docker run -d --privileged --name worker-${i} --hostname=worker-${i} docker:${DOCKER_VERSION}
sleep 5
docker exec worker-${i} docker swarm join --token ${SWARM_TOKEN} ${SWARM_MASTER_IP}:2377
done
# Setup the visualizer
docker service create \
--detach=true \
--name=viz \
--publish=8000:8080/tcp \
--constraint=node.role==manager \
--mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
dockersamples/visualizer
####### play with mongo
mkdir mongodemo && cd mongodemo
wget https://raw.githubusercontent.com/docker-library/docs/f6c9b596064e2eed9c3b6ac75bea606cb6d94099/mongo/stack.yml
docker stack deploy -c stack.yml mongo
# from windows:
# mongo will be available under <eth0>:8081
# visualizer under <eth0>:8000
ip -4 addr | grep eth0
I am running kubernetes cluster on AWS using Kops. We deploy Jenkins on it and we want to run docker, kubectl commands inside jenkins pods and want output to show of kubernetes clusters nodes, pods, and containers. We are able to achieve this via dockerfile or docker compose. We tried to mount docker socket as volume but didn't workout.
The regular jenkins image does not ship with the docker command.
Check out this jenkins image, it contains the docker and docker-compose commands:
https://hub.docker.com/r/trion/jenkins-docker-client
I found the solution, i need to mount docker socket and docker binary inside jenkins container and it will workout like charm.
I have deployed my application via docker on Google Kubernetes cluster. I am facing an application error due to which I want to enter inside the container and check a few things. At local, all I had to do was run sudo docker ps and then exec -it into the container.
How can I do that when containers are deployed on a Kubernetes cluster?
You need to use kubectl
kubectl get pods
kubectl exec -it pod-container-name -- /bin/bash
I have three swarm nodes.
Deployed containerized service with mode "global" through docker swarm.
Later, add one more swarm node to current to be four nodes.
How can i deploy the container service to new added nodes?
The command(docker service scale) only be used with "replicated" mode.
I would recommend running the following command if you have a lot of services running. This command will rebalance all services evenly across all the docker nodes.
docker service ls -q > docker_services && for i in `cat docker_services`; do docker service update "$i" --detach=false --force ; done
In case you have only one service then use this command
docker service update <service_name>
I have a setup which runs my Docker container like this.
run-docker.sh
docker build -t wordpress-gcloud
container=$(docker run -d wordpress-gcloud)
ipOfContainer=$(docker inspect "$container" | jq -r '.[0].NetworkSettings.IPAddress')
But now I have setup a Docker Swarm (1 manager + 2 workers).
How should I convert the above bash script to run the container on the swarm?
Typically, you can access your Swarm cluster via Swarm APIs, which is similar with Docker API. To access Swarm APIs, you can use -H parameter with docker commands. For example, if you have a swarm manager running on your local machine, and the port number is 3376, then you can get your swarm cluster info with:
docker -H 127.0.0.1:3376 info
You can also inspect the swarm cluster containers by:
docker -H 127.0.0.1:3376 inspect <container ID>
More details about communciate with Swarm cluster can be found here: https://docs.docker.com/swarm/install-manual/#/step-6-communicate-with-the-swarm
But in your case, I think that docker build command could be a problem. In my understanding, Swarm will find a random node from your cluster to execute this docker build process, so if the Dockerfile is not existing on the node where docker build has been executed, you will get error. My idea is to consider to build your image in a certain place, and push the image to a image registry, then pull and run the image in any place you want.