I am pretty new on Kubernetes and cloud computing. I am working with bare metal servers on my home (actually virtual servers on vbox) and trying to run a stateful app with StatefulSet. I have 1 master and 2 worker nodes and I am trying to run a database application on this cluster. So each node has 1 pod and I am very confused about volumes. I use hostpath volume(code below) but volumes working separately(actually they are not synchronizing). So my 2 pods are working different(same apps but they run like 2 different servers) when I reach them.
How can I run that app in 2 synchronized pods?
I've tried to synchronized volume files between 2 slaves. I also tried to synchronize volume files with deployment. I've tried to do this with volume provisioning (persistent volume and persistent volume provisioning).
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cloud
spec:
selector:
matchLabels:
app: cloud
serviceName: "cloud"
replicas: 2
template:
metadata:
labels:
app: cloud
spec:
containers:
- name: cloud
image: owncloud:v2
imagePullPolicy: Never
ports:
- containerPort: 80
name: web
volumeMounts:
- name: cloud-volume
mountPath: /var/www/html/
volumes:
- name: cloud-volume
hostPath:
path: /volumes/cloud/
---
kind: Service
apiVersion: v1
metadata:
name: cloud
spec:
selector:
app: cloud
type: LoadBalancer
ports:
- protocol: TCP
port: 80
Related
I have an EKS cluster with an application load balancer with a target group setup for each application environment. In my cluster I am building my application from a base docker image that is stored in a private ECR repository. I have confirmed that my pods are able to pull from the private ECR repo due to a secret I have setup to allow the private ECR image to be pulled. I am having a problem with the base docker image being able to get into a healthy state in the target group. I updated to containerPort in my deployment to match the port of the target group. I am not sure if that is how it needs to be configured. Below is how I defined everything for this namespace. I also have my dockerfile for the base image. Any advice how I can get a base docker image into a healthy state for me to build my application would be helpful.
dev.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: dev
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: dev-deployment
spec:
selector:
matchLabels:
app.kubernetes.io/name: dev-app
replicas: 2
template:
metadata:
labels:
app.kubernetes.io/name: dev-app
spec:
containers:
- name: dev-app
image: xxxxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/private/base-docker-image:latest
imagePullPolicy: Always
ports:
- containerPort: 30411
imagePullSecrets:
- name: dev
---
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: dev-service
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
type: NodePort
selector:
app.kubernetes.io/name: dev-app
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: dev
name: dev-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: instance
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: dev-service
servicePort: 80
---
dockerfile
FROM private/base-docker-image:latest
COPY . /apps
WORKDIR /apps
RUN npm run build
ENV ML_HOST=$HOST ML_PORT=$PORT ML_USER=$USER ML_PASSWORD=$PASSWORD
CMD ["npm", "run", "dockerstart"]
Registered Targets
Health Check Settings
This is a community wiki answer posted for better visibility.
As confirmed in the comments the solution is to set the targetPort to the port opened by the application which is 30411 as mentioned in the deployment's yaml configuration.
I've created an ACR and AKS, and pushed a container to the ACR.
I then applied the following yaml file to AKS:
apiVersion: apps/v1
kind: Deployment
metadata:
name: readit-cart
spec:
selector:
matchLabels:
app: readit-cart
template:
metadata:
labels:
app: readit-cart
spec:
containers:
- name: readit-cart
image: memicourseregistry.azurecr.io/cart:v2
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 5005
---
apiVersion: v1
kind: Service
metadata:
name: readit-cart
spec:
type: LoadBalancer
ports:
- port: 443
targetPort: 5005
selector:
app: readit-cart
I can run the container locally on port 5005 and it runs just fine.
In Azure portal, in the AKS resources view, I can see the service and the pod, they are both running (green).
And yet, when I try to access the public IP of the service, I get a "This site can't be reached" error.
What am I missing?
OK, it looks like the problem was that the pod run on port 80, and not 5005, even though the container run locally on 5005. Strange...
I'll look further into it.
I have an on-prem kubernetes cluster and I want to deploy to it a docker registry from which the cluster nodes can download images. In my attempts to do this, I've tried several methods of identifying the service: a NodePort, a LoadBalancer provided by MetalLB in Layer2 mode, its Flannel network IP (referring to the IP that, by default, would be on the 10.244.0.0/16 network), and its cluster IP (referring to the IP that, by default, would be on the 10.96.0.0/16 network). In every case, connecting to the registry via docker failed.
I performed a cURL against the IP and realized that while the requests were resolving as expected, the tcp dial step was consistently taking 63.15 +/- 0.05 seconds, followed by the HTTP(s) request itself completing in an amount of time that is within margin of error for the tcp dial. This is consistent across deployments with firewall rules varying from a relatively strict set to nothing except the rules added directly by kubernetes. It is also consistent across network architectures ranging from a single physical server with VMs for all cluster nodes to distinct physical hardware for each node and a physical switch. As mentioned previously, it is also consistent across the means by which the service is exposed. It is also consistent regardless of whether I use an ingress-nginx service to expose it or expose the docker registry directly.
Further, when I deploy another pod to the cluster, I am able to reach the pod at its cluster IP without any delays, but I do encounter an identical delay when trying to reach it at its external LoadBalancer IP or at a NodePort. No delays besides expected network latency are encountered when trying to reach the registry from a machine that is not a node on the cluster, e.g., using the LoadBalancer or NodePort.
As a matter of practice, my main inquiry is what is the "correct" way to do what I am attempting to do? Furthermore, as an academic matter, I would also like to know the source of the very long, very consistent delay that I've been seeing?
My deployment yaml file has been included below for reference. The ingress handler is ingress-nginx.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-pv-claim
namespace: docker-registry
labels:
app: registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: docker-registry
namespace: docker-registry
spec:
replicas: 1
selector:
matchLabels:
app: docker-registry
template:
metadata:
labels:
app: docker-registry
spec:
containers:
- name: docker-registry
image: registry:2.7.1
env:
- name: REGISTRY_HTTP_ADDR
value: ":5000"
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: "/var/lib/registry"
ports:
- name: http
containerPort: 5000
volumeMounts:
- name: image-store
mountPath: "/var/lib/registry"
volumes:
- name: image-store
persistentVolumeClaim:
claimName: registry-pv-claim
---
kind: Service
apiVersion: v1
metadata:
name: docker-registry
namespace: docker-registry
labels:
app: docker-registry
spec:
selector:
app: docker-registry
ports:
- name: http
port: 5000
targetPort: 5000
---
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
kubernetes.io/ingress.class: docker-registry
name: docker-registry
namespace: docker-registry
spec:
rules:
- host: example-registry.com
http:
paths:
- backend:
serviceName: docker-registry
servicePort: 5000
path: /
tls:
- hosts:
- example-registry.com
secretName: tls-secret
For future visitors, seems like your issue is related to Flannel.
The whole problem was described here:
https://github.com/kubernetes/kubernetes/issues/88986
https://github.com/coreos/flannel/issues/1268
including workaround:
https://github.com/kubernetes/kubernetes/issues/86507#issuecomment-595227060
I was experimenting with something with Kubernetes Persistent Volumes, I can't find a clear explanation in Kubernetes documentation and the behaviour is not the one I am expecting so I like to ask here.
I configured following Persistent Volume and Persistent Volume Claim.
kind: PersistentVolume
apiVersion: v1
metadata:
name: store-persistent-volume
namespace: test
spec:
storageClassName: hostpath
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/Volumes/Data/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: store-persistent-volume-claim
namespace: test
spec:
storageClassName: hostpath
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
and the following Deployment and Service configuration.
kind: Deployment
apiVersion: apps/v1beta2
metadata:
name: store-deployment
namespace: test
spec:
replicas: 1
selector:
matchLabels:
k8s-app: store
template:
metadata:
labels:
k8s-app: store
spec:
volumes:
- name: store-volume
persistentVolumeClaim:
claimName: store-persistent-volume-claim
containers:
- name: store
image: localhost:5000/store
ports:
- containerPort: 8383
protocol: TCP
volumeMounts:
- name: store-volume
mountPath: /data
---
#------------ Service ----------------#
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: store
name: store
namespace: test
spec:
type: LoadBalancer
ports:
- port: 8383
targetPort: 8383
selector:
k8s-app: store
As you can see I defined '/Volumes/Data/data' as Persistent Volume and expecting that to mount that to '/data' container.
So I am assuming whatever in '/Volumes/Data/data' in the host should be visible at '/data' directory at container. Is this assumption correct? Because this is definitely not happening at the moment.
My second assumption is, whatever I save at '/data' should be visible at host, which is also not happening.
I can see from Kubernetes console that everything started correctly, (Persistent Volume, Claim, Deployment, Pod, Service...)
Am I understanding the persistent volume concept correctly at all?
PS. I am trying this in a Mac with Docker (18.05.0-ce-mac67(25042) -Channel edge), may be it should not work at Mac?
Thx for answers
Assuming you are using multi-node Kubernetes cluster, you should be able to see the data mounted locally at /Volumes/Data/data on the specific worker node that pod is running
You can check on which worker your pod is scheduled by using the command kubectl get pods -o wide -n test
Please note, as per kubernetes docs, HostPath (Single node testing only – local storage is not supported in any way and WILL NOT WORK in a multi-node cluster) PersistentVolume
It does work in my case.
As you are using the host path, you should check this '/data' in the worker node in which the pod is running.
Like the guy said above. You need to run a 'kubectl get po -n test -o wide' and you will see the node the pod is hosted on. Then if you SSH that worker you can see the volume
I'm trying to setup Jenkins to run in a container on Kubernetes, but I'm having trouble persisting the volume for the Jenkins home directory.
Here's my deployment.yml file. The image is based off jenkins/jenkins
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins-deployment
labels:
app: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: 1234567.dkr.ecr.us-east-1.amazonaws.com/mycompany/jenkins
imagePullPolicy: "Always"
ports:
- containerPort: 8080
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
- name: jenkins-home
emptyDir: {}
However, if i then push a new container to my image repository and update the pods using the below commands, Jenkins comes back online but asks me to start from scratch (enter admin password, none of my Jenkins jobs are there, no plugins etc)
kubectl apply -f kubernetes (where my manifests are stored)
kubectl set image deployment/jenkins-deployment jenkins=1234567.dkr.ecr.us-east-1.amazonaws.com/mycompany/jenkins:$VERSION
Am I misunderstanding how this volume mount is meant to work?
As an aside, I also have backup and restore scripts which backup the Jenkins home directory to s3, and download it again, but that's somewhat outside the scope of this issue.
You should use PersistentVolumes along with StatefulSet instead of Deployment resource if you wish your data to survive re-deployments|restarts of your pod.
You have specified the volume type EmptyDir. This will essentially mount an empty directory on the kube node that runs your pod. Every time you restart your deployment, the pod could move between kube hosts and the empty dir isn't present, so your data isn't persisting across restarts.
I see you're pulling you image from an ECR repository, so I'm assuming you're running k8s in AWS.
You'll need to configure a StorageClass for AWS. If you've provisioned k8s using something like kops, this will already be configured. You can confirm this by doing kubectl get storageclass - the provisioner should be configured as EBS:
NAME PROVISIONER
gp2 (default) kubernetes.io/aws-ebs
Then, you need to specify a persistentvolumeclaim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp2 # must match your storageclass from above
resources:
requests:
storage: 30Gi
You can now the pv claim on your deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins-deployment
labels:
app: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: 1234567.dkr.ecr.us-east-1.amazonaws.com/mycompany/jenkins
imagePullPolicy: "Always"
ports:
- containerPort: 8080
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
persistentVolumeClaim:
claimName: jenkins-data # must match the claim name from above