I want to capture all system calls from a k8 pod.
Sysdig supports the -k flag for specifying a url to the kubernetes kubectl api.
I exposed the kubectl api using the kubectl proxy command below
kubectl proxy --port=8080 &
I want to filter system calls for a specific k8 pod called 'mypod'
sudo sysdig -k http://127.0.0.1:8080 k8s.pod.name=mypod
No events are captured using this filter. It is also worth noting that I am running this sysdig command from the master node, and that 'mypod' is running on a different worker machine that is a part of the k8 cluster.
what am I missing?
Sysdig OSS should run on the same machine where the process/container you want to monitor is.
If you try to filter syscalls that happen in another node it'll be impossible, since a process never calls another machine's kernel.
Sysdig OSS, like Falco, works at the kernel level to monitor syscalls. If you were trying to monitor K8S Audit events that'd be different since they are sent to the plugin socket.
Related
I was handed a kubernetes cluster to manage. But in the same node, I can see running docker containers (via docker ps) that I could not able to find/relate in the pods/deployments (via kubectl get pods/deployments).
I have tried kubectl describe and docker inspect but could not pick out any differentiating parameters.
How to differentiate which is which?
There will be many. At a minimum you'll see all the pod sandbox pause containers which are normally not visible. Plus possibly anything you run directly such as the control plane if not using static pods.
I have a kubernetes setup in which one is master node and two worker nodes. After the deployment, which is a daemonset, it starts pods on both the worker nodes. These pods contain 2 containers. These containers have a python script running in them. The python scripts runs normally but at a certain point, after some time, it needs to send a shutdown command to the host. I can directly issue command shutdown -h now but this will run on the container not on the host and gives below error:
Failed to connect to bus: No such file or directory
Failed to talk to init daemon.
To resolve this, I can get the ip address of the host and then I can ssh into it and then run the command to safely shutdown the host.
But is there any other way I can issue command to the host in kubernetes/dockers.?
You can access your cluster using kube api.
https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/
Accessing the API from a Pod When accessing the API from a pod,
locating and authenticating to the apiserver are somewhat different.
The recommended way to locate the apiserver within the pod is with the
kubernetes.default.svc DNS name, which resolves to a Service IP which
in turn will be routed to an apiserver.
The recommended way to authenticate to the apiserver is with a service
account credential. By kube-system, a pod is associated with a service
account, and a credential (token) for that service account is placed
into the filesystem tree of each container in that pod, at
/var/run/secrets/kubernetes.io/serviceaccount/token.
Draining the node you can use this
The Eviction API
https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
But i dont really sure about on pod can drain own node. Workaround can be controlling other pod from different node.
I have pod and its purpose is to take the incoming data and write it to the host volume. I'm running this pod in all the minions.
Now when i setup NodePort service to this pods, traffic will go to 1 pod at a time.
But how do i send request to all this pods in different minions? How to i bypass the load-balancing here? I want that data to be available in all the minions host volume.
A service uses a selector to identify the list of pods to proxy to (if they're in the Ready state). You could simply ask for the same list of pods with a GET request:
$ curl -G "$MASTER/api/v1/namespaces/$NAMESPACE/pods?labelSelector=$KEY=$VALUE"
And then manually send your request to each of the pod ip:port endpoints. If you need to be able to send the request from outside the cluster network, you could create a proxy pod (exposed to the external network through the standard means). The proxy pod could watch for pods with your label (similar to above), and forward any requests it receives to the list of ready pods.
A similar effect could be achieved using hostPort and forwarding to nodes, but the use of hostPort is discourage (see best practices).
Here's a method that works as long as you can send the requests from a container inside the k8s network (this may not match the OP's desire exactly, but I'm guessing this may work for someone googling this).
You have to look up the pods somehow. Here I'm finding all pods in the staging namespace with the label app=hot-app:
kubectl get pods -l app=hot-app -n staging -o json | jq -r '.items[].status.podIP'
this example uses the awesome jq tool to parse the resulting json and grab the pod ips, but you can parse the json in other ways, including with kubectl itself.
this returns something like this:
10.245.4.253
10.245.21.143
you can find the internal port like this (example has just one container, so one unique port):
kubectl get pods -l app=hot-app -n staging -o json | jq -r '.items[].spec.containers[].ports[].containerPort' | sort | uniq
8080
then you get inside a container in your k8s cluster with curl, combine the ips and port from the previous commands, and hit the pods like this:
curl 10.245.4.253:8080/hot-path
curl 10.245.21.143:8080/hot-path
You need to define a hostPort for the container and address each pod on each node individually via the host IP.
See caveats in the best-practice guide's Services section.
I have a problem to understand the kubernetes workflow:
So as I understand the flow:
You have a master which contains etcd, api-server, controller manager and scheduler.
You have nodes which contain pods (wich contain containers), kubelet and a proxy.
The proxy is working as a basic proxy to make it possible for a service to communicate with other nodes.
When a pod dies, the controller manager will see this (it 'reads' the replication controller which describes how many pods there normally are).
unclear:
The controller manager will inform the API-server (I'm not right about this).
The API-server will tell the scheduler to search a new place for the pod.
After the scheduler has found a good place, the API will inform kubelet to create a new pod.
I'm not sure about the last scenario? Can you tell me the right proces is a clear way?
Which component is creating the pod and container? Is it kubelet?
So it's the kubelet that actually creates the pods and talks to the docker daemon. If you do a docker ps -a on your nodes (as in not master) in your cluster, you'll see the containers in your pod running. So the workflow is run a kubectl command, that goes to the API server, which passes it to the controller, say that command was to spawn a pod, the controller relays that to the API server which then goes to the scheduler and tells it to spawn the pod. Then the kubelet is told to spawn said pod.
I suggest reading the Borg paper that Kubernetes is based on to better understand things in further detail. http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf
I would like to run Aerospike cluster on Docker containers managed by Kubernetes on CoreOS on Google Compute Engine (GCE). But since GCE does not permit multicast, I have to use Mesh heartbeat as described here, which has to be set up by specifying all node's IP addresses and ports; it seems so inflexible to me.
Is there any recommended cloud-config settings for Aerospike cluster on Kubernetes/CoreOS/GCE with flexibility of the cluster being kept?
An alternative to specifying all mesh seed IP addresses is to use the asinfo tip command.
Please see:
http://www.aerospike.com/docs/reference/info/#tip
the tip command
asinfo -v 'tip:host=172.16.121.138;port=3002'
The above command could be added to a script or orchestration tool with correct ips.
You may also find addtional info on the aerospike Forum:
Aerospike Forum
You can get the pod IPs from a service via a DNS query with the integrated DNS - if you set clusterIP: "none", a
dig +short svcname.namespace.svc.cluster.local
Will return each pod ip in the service.
When we talked with the Aerospike engineers during pre-sales they recommended against running Aerospike inside a Docker container (Kubernetes or not). Their reasoning was that when running inside of Docker Aerospike is prevented from accessing the SSD hardware directly and the SSD drivers running through Docker aren't as efficient as running on bare metal (or VM). Many of the optimizations they have written weren't able to be taken advantage of.