I want to run certain job on every single node in specific node groups.
Can kubernetes do the same thing like Swarm global mode?
To complete #David Maze answer:
A DaemonSet is used to create a Pod on each Node. More information here.
Example:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd-elasticsearch
namespace: kube-system
labels:
k8s-app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd-elasticsearch
template:
metadata:
labels:
name: fluentd-elasticsearch
spec:
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd-elasticsearch
image: k8s.gcr.io/fluentd-elasticsearch:1.20
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
If you want to schedule Pods not on every Node, you can use Taints and Tolerations concept. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. For more information, look through the link.
For example:
You can add a Taint to a Node:
kubectl taint nodes <Node_name> key=value:NoSchedule
After that, Pods will have no opportunity to schedule on that Node, even from a DaemonSet.
You can add toleration to a Pod (or to a DaemonSet in your case) to allow it schedule on the Node with the toleration:
tolerations:
- key: "key"
operator: "Equal"
value: "value"
effect: "NoSchedule"
You're looking for a DaemonSet.
Related
I am using Cassandra image w.r.t.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cassandra
labels:
app: cassandra
spec:
serviceName: cassandra
replicas: 3
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
terminationGracePeriodSeconds: 1800
containers:
- name: cassandra
image: gcr.io/google-samples/cassandra:v13
imagePullPolicy: Always
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
limits:
cpu: "500m"
memory: 1Gi
requests:
cpu: "500m"
memory: 1Gi
securityContext:
capabilities:
add:
- IPC_LOCK
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- nodetool drain
env:
- name: MAX_HEAP_SIZE
value: 512M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.default.svc.cluster.local"
- name: CASSANDRA_CLUSTER_NAME
value: "K8Demo"
- name: CASSANDRA_DC
value: "DC1-K8Demo"
- name: CASSANDRA_RACK
value: "Rack1-K8Demo"
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
initialDelaySeconds: 15
timeoutSeconds: 5
# These volume mounts are persistent. They are like inline claims,
# but not exactly because the names need to match exactly one of
# the stateful pod volumes.
volumeMounts:
- name: cassandra-data
mountPath: /cassandra_data
# These are converted to volume claims by the controller
# and mounted at the paths mentioned above.
# do not use these in production until ssd GCEPersistentDisk or other ssd pd
volumeClaimTemplates:
- metadata:
name: cassandra-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: fast
resources:
requests:
storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
type: pd-ssd
Now I need to add below line to cassandra-env.sh in postStart or in cassandra yaml file:
-JVM_OPTS="$JVM_OPTS
-javaagent:$CASSANDRA_HOME/lib/cassandra-exporter-agent-<version>.jar"
Now I was able to achieve this, but after this step, Cassandra requires a restart but as it's already running as a pod, I don't know how to restart the process. So is there any way that this step is done prior to running the pod and not after it is up?
I was suggested below solution:-
This won’t work. Commands that run postStart don’t impact the running container. You need to change the startup commands passed to Cassandra.
The only way that I know to do this is to create a new container image in the artifactory based on the existing image. and pull from there.
But I don't know how to achieve this.
I have deployed my Jenkins as part of kubernetes yaml file and also enabled Persist volume claim, when my Jenkins pod is restarts, i lost my all the jobs and configuration which means i need to re-install all Jenkins suggest plugin, configure kubernetes cloud, configure git repo, and create new pipeline job.
cloud you please help me how to avoid above scenario.
vi jenkins-deployment.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: jenkins-master
namespace: jenkins
labels:
app: jenkins-master
spec:
replicas: 1
selector:
matchLabels:
app: jenkins-master
template:
metadata:
labels:
app: jenkins-master
spec:
securityContext:
fsGroup: 1000
containers:
- name: jenkins
image: jenkins/jenkins:lts
imagePullPolicy: Always
ports:
- containerPort: 8080
- containerPort: 50000
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 300
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 2
failureThreshold: 5
volumeMounts:
- mountPath: "/var"
name: jenkins-home
subPath: jenkins_home
resources:
limits:
cpu: 800m
memory: 3Gi
requests:
cpu: 100m
memory: 3Gi
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: pvc-jenkins-home
vi jenkins-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-jenkins-home
namespace: jenkins
spec:
storageClassName: efs
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Mi
kubectl get pvc -n jenkins
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-jenkins-home Bound pvc-4ccf3f55-6894-4fee-88d7-58dd7584b837 10Mi RWO efs 59m
Please let me know if any details required from my side
Please remove the subpathfrom volumeMounts as subPath will overwrite everything under the /var directory. so it should be just like this
volumeMounts:
- mountPath: /var
name: jenkins-home
I have a local kubernetes cluster where I added a Fluentd Daemonset using the preconfigured elasticsearch image (fluent/fluentd-kubernetes-daemonset:elasticsearch). Step 2 of this article. I also have an elastic cluster running in the cloud. You can pass some env variables to the fluentd-elasticsearch image for configuration. It looks pretty straightforward, but when running the fluentd Pod I keep getting the error:
"Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"fa0acce34bf64db9bc9e46f98743c185.westeurope.azure.elastic-cloud.com\", :port=>9243, :scheme=>\"https\", :user=>\"username\", :password=>\"obfuscated\"})!" plugin_id="out_es"
when I try to reach the elastic cluster from within the pod with
# wget https://fa0acce34bf64db9bc9e46f98743c185.westeurope.azure.elastic-cloud.com:9243/ I get a 401 unauthorized (cuz I havent submitted user/pass here), but it at least shows that the address is reachable.
Why is it failing to connect?
I already set the FLUENT_ELASTICSEARCH_SSL_VERSION to 'TLSv1_2', i saw that that solved some problems for others.
Daemonset configuration:
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-logging
labels:
app: fluentd
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "fa0acce34bf64db9bc9e46f98743c185.westeurope.azure.elastic-cloud.com"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9243"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "https"
- name: FLUENT_UID
value: "0"
- name: FLUENT_ELASTICSEARCH_SSL_VERIFY
value: "false"
- name: FLUENT_ELASTICSEARCH_SSL_VERSION
value: "TLSv1_2"
- name: FLUENT_ELASTICSEARCH_USER
value: "<user>"
- name: FLUENT_ELASTICSEARCH_PASSWORD
value: "<password>"
resources:
limits:
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
For anyone else who runs into this problem:
I was following a tutorial that used the 'image: fluent/fluentd-kubernetes-daemonset:elasticsearch' image. When you check their DockerHub (https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset) you can see that the :elaticsearch tag is a year old and probably outdated.
I changed the image for the DaemonSet to a more recent and stable tag 'fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch' and boom it works now.
I was able to get a kubernetes job up and running on AKS (uses docker hub image to process a biological sample and then upload the output to blob storage - this is done with a bash command that I provide in the args section of my yaml file). However, I have 20 samples, and would like to spin up 20 nodes so that I can process the samples in parallel (one sample per node). How do I send each sample to a different node? The "parallelism" option in a yaml file processes all of the 20 samples on each of the 20 nodes, which is not what I want.
Thank you for the help.
if you want each instance of the job to be on a different node, you can use daemonSet, thats exactly what it does, provisions 1 pod per worker node.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd-elasticsearch
namespace: kube-system
labels:
k8s-app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd-elasticsearch
template:
metadata:
labels:
name: fluentd-elasticsearch
spec:
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd-elasticsearch
image: k8s.gcr.io/fluentd-elasticsearch:1.20
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
Another way of doing that - using pod antiaffinity:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- zk
topologyKey: "kubernetes.io/hostname"
The requiredDuringSchedulingIgnoredDuringExecution field tells the Kubernetes Scheduler that it should never co-locate two Pods which have app label as zk in the domain defined by the topologyKey. The topologyKey kubernetes.io/hostname indicates that the domain is an individual node. Using different rules, labels, and selectors, you can extend this technique to spread your ensemble across physical, network, and power failure domains
How/where the samples are stored? You could load them (or a pointer to the actual sample) into a queue like Kafka and let the application retrieve each sample once and upload it to the blob after computation. You can then even assure that if a computation fails, another pod will pick it up and restart the computation.
Found this example for Kubernetes EmptyDir volume
apiVersion: v1
kind: Pod
metadata:
name: www
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- mountPath: /srv/www
name: www-data
readOnly: true
- name: git-monitor
image: kubernetes/git-monitor
env:
- name: GIT_REPO
value: http://github.com/some/repo.git
volumeMounts:
- mountPath: /data
name: www-data
volumes:
- name: www-data
emptyDir: {}
I want to volume mount between 2 pods. I am creating these pods using 2 different Replication Controllers. The replication controllers looks like this
Replication Controller 1:
apiVersion: v1
kind: ReplicationController
metadata:
name: node-worker
labels:
name: node-worker
spec:
replicas: 1
selector:
name: node-worker
template:
metadata:
labels:
name: node-worker
spec:
containers:
-
name: node-worker
image: image/node-worker
volumeMounts:
- mountPath: /mnt/test
name: deployment-volume
volumes:
- name: deployment-volume
emptyDir: {}
Replication Controller 2:
apiVersion: v1
kind: ReplicationController
metadata:
name: node-manager
labels:
name: node-manager
spec:
replicas: 1
selector:
name: node-manager
template:
metadata:
labels:
name: node-manager
spec:
containers:
-
name: node-manager
image: image/node-manager
volumeMounts:
- mountPath: /mnt/test
name: deployment-volume
volumes:
- name: deployment-volume
emptyDir: {}
Can Kubernetes emptyDir volume be used for this scenario?
EmptyDir volumes are inherently bound to the lifecycle of a single pod and can't be shared amongst pods in replication controllers or otherwise. If you want to share volumes amongst pods, the best choices right now are NFS or gluster, in a persistent volume. See an example here: https://github.com/kubernetes/examples/blob/master/staging/volumes/nfs/README.md
Why do you want to share the volume mount between pods? This will not work reliably because you aren't guaranteed to have a 1:1 mapping between where pods in replication controller 1 and replication controller 2 are scheduled in your cluster.
If you want to share local storage between containers, you should put both of the containers into the same pod, and have each container mount the emptyDir volume.
You require three things to get this working. More info can be found here and some documentation here, but it's a little confusing at first.
This example mounts a NFS volume.
1. Create a PersistentVolume pointing to your NFS server
file : mynfssharename-pv.yaml
(update server to point to your server)
apiVersion: v1
kind: PersistentVolume
metadata:
name: mynfssharename
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
nfs:
server: yourservernotmine.yourcompany.com
path: "/yournfspath"
kubectl create -f mynfssharename-pv.yaml
2. Create a PersistentVolumeClaim to points to PersistentVolume mynfssharename
file : mynfssharename-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mynfssharename
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
kubectl create -f mynfssharename-pvc.yaml
3. Add the claim to your ReplicationController or Deployment
spec:
containers:
- name: sample-pipeline
image: yourimage
imagePullPolicy: Always
ports:
- containerPort: 8080
name: http
volumeMounts:
# name must match the volume name below
- name: mynfssharename
mountPath: "/mnt"
volumes:
- name: mynfssharename
persistentVolumeClaim:
claimName: mynfssharename