I set up a local all-in-one Kubernetes env. I followed below steps to install. When I try to create my first RC, the RC created successfully, but the pod didn't get created:
Env: CentOS7
#systemctl disable firewalld
#systemctl stop firewalld
#yum install -y etcd kubernetes
#systemctl start etcd
#systemctl start docker
#systemctl start kube-apiserver
#systemctl start kube-controller-manager
#systemctl start kube-scheduler
#systemctl start kubelet
#systemctl start kube-proxy
All services started successful.
mysql-rc.yaml:
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
command:
[root#terryhu82 yaml]# kubectl create -f mysql-rc.yaml
replicationcontroller "mysql" created
[root#terryhu82 yaml]# kubectl get rc
NAME DESIRED CURRENT READY AGE
mysql 1 0 0 48s
[root#terryhu82 yaml]# kubectl get pods
No resources found.
[root#terryhu82 yaml]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root#terryhu82 yaml]# kubectl get nodes
NAME STATUS AGE
127.0.0.1 Ready 23h
[root#terryhu82 yaml]# kubectl describe node 127.0.0.1
Name: 127.0.0.1
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=127.0.0.1
Taints: <none>
CreationTimestamp: Mon, 06 Nov 2017 00:22:58 +0800
Phase:
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Mon, 06 Nov 2017 23:38:05 +0800 Mon, 06 Nov 2017 00:22:58 +0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Mon, 06 Nov 2017 23:38:05 +0800 Mon, 06 Nov 2017 00:22:58 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Mon, 06 Nov 2017 23:38:05 +0800 Mon, 06 Nov 2017 00:22:58 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
Ready True Mon, 06 Nov 2017 23:38:05 +0800 Mon, 06 Nov 2017 00:23:08 +0800 KubeletReady kubelet is posting ready status
Addresses: 127.0.0.1,127.0.0.1,127.0.0.1
Capacity:
alpha.kubernetes.io/nvidia-gpu: 0
cpu: 4
memory: 16416476Ki
pods: 110
Allocatable:
alpha.kubernetes.io/nvidia-gpu: 0
cpu: 4
memory: 16416476Ki
pods: 110
System Info:
Machine ID: 52ac3151ed7d485d98fa44e0da0e817b
System UUID: 564D434D-F7CF-9923-4B1D-A494E3391AE1
Boot ID: 148e293c-9631-4421-b55b-115ba72bc1d3
Kernel Version: 3.10.0-693.5.2.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://1.12.6
Kubelet Version: v1.5.2
Kube-Proxy Version: v1.5.2
ExternalID: 127.0.0.1
Non-terminated Pods: (0 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
0 (0%) 0 (0%) 0 (0%) 0 (0%)
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
13m 13m 1 {kubelet 127.0.0.1} Normal Starting Starting kubelet.
13m 13m 1 {kubelet 127.0.0.1} Warning ImageGCFailed unable to find data for container /
13m 13m 6 {kubelet 127.0.0.1} Normal NodeHasSufficientDisk Node 127.0.0.1 status is now: NodeHasSufficientDisk
13m 13m 6 {kubelet 127.0.0.1} Normal NodeHasSufficientMemory Node 127.0.0.1 status is now: NodeHasSufficientMemory
13m 13m 6 {kubelet 127.0.0.1} Normal NodeHasNoDiskPressure Node 127.0.0.1 status is now: NodeHasNoDiskPressure
13m 13m 1 {kubelet 127.0.0.1} Warning Rebooted Node 127.0.0.1 has been rebooted, boot id: 148e293c-9631-4421-b55b-115ba72bc1d3
I didn't perform any configuration for the components. Can anyone help to guide me why the pods and container didn't get created? Where I can see the log?
to run this mysql you need 173MB memory. you defined 128M as upper limit, thats why its not starting. you can use the below one.
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "200Mi"
cpu: "500m"
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
htop output
Related
I want to trigger a manual workflow in Argo. I am using Openshift and ArgoCD, have scheduled workflows that are running successfully in Argo but failing when triggering a manual run for one workflow.
The concerned workflow is:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: "obslytics-data-exporter-manual-workflow-"
labels:
owner: "obslytics-remote-reader"
app: "obslytics-data-exporter"
pipeline: "obslytics-data-exporter"
spec:
arguments:
parameters:
- name: start_timestamp
value: "2020-11-18T20:00:00Z"
entrypoint: manual-trigger
templates:
- name: manual-trigger
steps:
- - name: trigger
templateRef:
name: "obslytics-data-exporter-workflow-triggers"
template: trigger-workflow
volumes:
- name: "obslytics-data-exporter-workflow-secrets"
secret:
secretname: "obslytics-data-exporter-workflow-secrets"
When I run the command:
argo submit trigger.local.yaml
The build pod is completed but the rest pods fail:
➜ dh-workflow-obslytics git:(master) ✗ oc get pods
NAME READY STATUS RESTARTS AGE
argo-ui-7fcf5ff95-9k8cc 1/1 Running 0 3d
gateway-controller-76bb888f7b-lq84r 1/1 Running 0 3d
obslytics-data-exporter-1-build 0/1 Completed 0 3d
obslytics-data-exporter-calendar-gateway-fbbb8d7-zhdnf 2/2 Running 1 3d
obslytics-data-exporter-manual-workflow-m7jdg-1074461258 0/2 Error 0 4m
obslytics-data-exporter-manual-workflow-m7jdg-1477271209 0/2 Error 0 4m
obslytics-data-exporter-manual-workflow-m7jdg-1544087495 0/2 Error 0 4m
obslytics-data-exporter-manual-workflow-m7jdg-1979266120 0/2 Completed 0 4m
obslytics-data-exporter-sensor-6594954795-xw8fk 1/1 Running 0 3d
opendatahub-operator-8994ddcf8-v8wxm 1/1 Running 0 3d
sensor-controller-58bdc7c4f4-9h4jw 1/1 Running 0 3d
workflow-controller-759649b79b-s69l7 1/1 Running 0 3d
The pods starting with obslytics-data-exporter-manual-workflow are the concerned pods that are failing. When I attempt to debug by describing pods:
➜ dh-workflow-obslytics git:(master) ✗ oc describe pods/obslytics-data-exporter-manual-workflow-4hzqz-3278280317
Name: obslytics-data-exporter-manual-workflow-4hzqz-3278280317
Namespace: dh-dev-argo
Priority: 0
PriorityClassName: <none>
Node: avsrivas-dev-ocp-3.11/10.0.111.224
Start Time: Tue, 24 Nov 2020 07:27:57 -0500
Labels: workflows.argoproj.io/completed=true
workflows.argoproj.io/workflow=obslytics-data-exporter-manual-workflow-4hzqz
Annotations: openshift.io/scc=restricted
workflows.argoproj.io/node-message=timeout after 0s
workflows.argoproj.io/node-name=obslytics-data-exporter-manual-workflow-4hzqz[0].trigger[1].run[0].metric-split(0:cluster_version)[0].process-metric(0)
workflows.argoproj.io/template={"name":"run-obslytics","arguments":{},"inputs":{"parameters":[{"name":"metric","value":"cluster_version"},{"name":"start_timestamp","value":"2020-11-18T20:00:00Z"},{"na...
Status: Failed
IP: 10.128.0.69
Controlled By: Workflow/obslytics-data-exporter-manual-workflow-4hzqz
Init Containers:
init:
Container ID: docker://25b95c684ef66b13520ba9deeba353082142f3bb39bafe443ee508074c58047e
Image: argoproj/argoexec:v2.4.2
Image ID: docker-pullable://docker.io/argoproj/argoexec#sha256:4e393daa6ed985cf680bcf0ecf04f7b0758940f0789505428331fcfe99cce06b
Port: <none>
Host Port: <none>
Command:
argoexec
init
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 24 Nov 2020 07:27:59 -0500
Finished: Tue, 24 Nov 2020 07:27:59 -0500
Ready: True
Restart Count: 0
Environment:
ARGO_POD_NAME: obslytics-data-exporter-manual-workflow-4hzqz-3278280317 (v1:metadata.name)
ARGO_CONTAINER_RUNTIME_EXECUTOR: k8sapi
Mounts:
/argo/podmetadata from podmetadata (rw)
/argo/staging from argo-staging (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qpggm (ro)
Containers:
wait:
Container ID: docker://a94e7f1bc1cfec4c8b549120193b697c91760bb8f3af414babef1d6f7ccee831
Image: argoproj/argoexec:v2.4.2
Image ID: docker-pullable://docker.io/argoproj/argoexec#sha256:4e393daa6ed985cf680bcf0ecf04f7b0758940f0789505428331fcfe99cce06b
Port: <none>
Host Port: <none>
Command:
argoexec
wait
State: Terminated
Reason: Completed
Message: timeout after 0s
Exit Code: 0
Started: Tue, 24 Nov 2020 07:28:00 -0500
Finished: Tue, 24 Nov 2020 07:28:01 -0500
Ready: False
Restart Count: 0
Environment:
ARGO_POD_NAME: obslytics-data-exporter-manual-workflow-4hzqz-3278280317 (v1:metadata.name)
ARGO_CONTAINER_RUNTIME_EXECUTOR: k8sapi
Mounts:
/argo/podmetadata from podmetadata (rw)
/mainctrfs/argo/staging from argo-staging (rw)
/mainctrfs/etc/obslytics-data-exporter from obslytics-data-exporter-workflow-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qpggm (ro)
main:
Container ID: docker://<some_id>
Image: docker-registry.default.svc:5000/<some_id>
Image ID: docker-pullable://docker-registry.default.svc:5000/<some_id>
Port: <none>
Host Port: <none>
Command:
/bin/sh
-e
Args:
/argo/staging/script
State: Terminated
Reason: Error
Exit Code: 126
Started: Tue, 24 Nov 2020 07:28:01 -0500
Finished: Tue, 24 Nov 2020 07:28:01 -0500
Ready: False
Restart Count: 0
Limits:
memory: 1Gi
Requests:
memory: 1Gi
Environment: <none>
Mounts:
/argo/staging from argo-staging (rw)
/etc/obslytics-data-exporter from obslytics-data-exporter-workflow-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qpggm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
podmetadata:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.annotations -> annotations
obslytics-data-exporter-workflow-secrets:
Type: Secret (a volume populated by a Secret)
SecretName: obslytics-data-exporter-workflow-secrets
Optional: false
argo-staging:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-qpggm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-qpggm
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/compute=true
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 27m default-scheduler Successfully assigned dh-dev-argo/obslytics-data-exporter-manual-workflow-4hzqz-3278280317 to avsrivas-dev-ocp-3.11
Normal Pulled 27m kubelet, avsrivas-dev-ocp-3.11 Container image "argoproj/argoexec:v2.4.2" already present on machine
Normal Created 27m kubelet, avsrivas-dev-ocp-3.11 Created container
Normal Started 27m kubelet, avsrivas-dev-ocp-3.11 Started container
Normal Pulled 27m kubelet, avsrivas-dev-ocp-3.11 Container image "argoproj/argoexec:v2.4.2" already present on machine
Normal Created 27m kubelet, avsrivas-dev-ocp-3.11 Created container
Normal Started 27m kubelet, avsrivas-dev-ocp-3.11 Started container
Normal Pulling 27m kubelet, avsrivas-dev-ocp-3.11 pulling image "docker-registry.default.svc:5000/dh-dev-argo/obslytics-data-exporter:latest"
Normal Pulled 27m kubelet, avsrivas-dev-ocp-3.11 Successfully pulled image "docker-registry.default.svc:5000/dh-dev-argo/obslytics-data-exporter:latest"
Normal Created 27m kubelet, avsrivas-dev-ocp-3.11 Created container
Normal Started 27m kubelet, avsrivas-dev-ocp-3.11 Started container
The only thing I learn from the above description is that the pods fail due to an error. I am unable to see any error in order to debug this issue.
When I attempt to read the Argo watch logs:
Name: obslytics-data-exporter-manual-workflow-8wzcc
Namespace: dh-dev-argo
ServiceAccount: default
Status: Running
Created: Tue Nov 24 08:01:10 -0500 (8 minutes ago)
Started: Tue Nov 24 08:01:10 -0500 (8 minutes ago)
Duration: 8 minutes 10 seconds
Progress:
Parameters:
start_timestamp: 2020-11-18T20:00:00Z
STEP TEMPLATE PODNAME DURATION MESSAGE
● obslytics-data-exporter-manual-workflow-8wzcc manual-trigger
└───● trigger obslytics-data-exporter-workflow-triggers/trigger-workflow
├───✔ get-labels(0) obslytics-data-exporter-workflow-template/get-labels obslytics-data-exporter-manual-workflow-8wzcc-2604296472 6s
└───● run obslytics-data-exporter-workflow-template/init
└───● metric-split(0:cluster_version) metric-worker
└───● process-metric run-obslytics
├─✖ process-metric(0) run-obslytics obslytics-data-exporter-manual-workflow-8wzcc-4222496183 6s failed with exit code 126
└─◷ process-metric(1) run-obslytics obslytics-data-exporter-manual-workflow-8wzcc-531670266 7m PodInitializing
I have a 2 master and 2 worker kubernetes cluster. Each node has private IP in the range 192.168.5.X and public IP.
After creating the weave daemonset, the weave pod picked the correct internal IP on one node but on the other node it picked the public IP. Is there any way we can instruct weave pod to pick the private IP on the node?
Im creating the cluster from scratch by doing everything manually on the VMs created on Virtual Box on local laptop. I refer the below link
https://github.com/mmumshad/kubernetes-the-hard-way
After deploying weave pods on worker node, weave pod on one of the worker nodes uses the NAT ip as below.
10.0.2.15 is the NAT IP and 192.168.5.12 is internal IP
kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
weave-net-p4czj 2/2 Running 2 26h 192.168.5.12 worker1 <none> <none>
weave-net-pbb86 2/2 Running 8 25h 10.0.2.15 worker2 <none> <none>
[#master1 ~]$ kubectl describe node
Name: worker1
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=worker1
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 10 Dec 2019 02:07:09 -0500
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Wed, 11 Dec 2019 04:50:15 -0500 Wed, 11 Dec 2019 04:50:15 -0500 WeaveIsUp Weave pod has set this
MemoryPressure False Wed, 11 Dec 2019 07:13:43 -0500 Tue, 10 Dec 2019 02:09:09 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 11 Dec 2019 07:13:43 -0500 Tue, 10 Dec 2019 02:09:09 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 11 Dec 2019 07:13:43 -0500 Tue, 10 Dec 2019 02:09:09 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 11 Dec 2019 07:13:43 -0500 Tue, 10 Dec 2019 04:16:26 -0500 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.5.12
Hostname: worker1
Capacity:
cpu: 1
ephemeral-storage: 14078Mi
hugepages-2Mi: 0
memory: 499552Ki
pods: 110
Allocatable:
cpu: 1
ephemeral-storage: 13285667614
hugepages-2Mi: 0
memory: 397152Ki
pods: 110
System Info:
Machine ID: 455146bc2c2f478a859bf39ac2641d79
System UUID: D4C6F432-3C7F-4D27-A21B-D78A0D732FB6
Boot ID: 25160713-e53e-4a9f-b1f5-eec018996161
Kernel Version: 4.4.206-1.el7.elrepo.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.3
Kubelet Version: v1.13.0
Kube-Proxy Version: v1.13.0
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default ng1-6677cd8f9-hws8n 0 (0%) 0 (0%) 0 (0%) 0 (0%) 26h
kube-system weave-net-p4czj 20m (2%) 0 (0%) 0 (0%) 0 (0%) 26h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 20m (2%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>
Name: worker2
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=worker2
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 10 Dec 2019 03:14:01 -0500
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Wed, 11 Dec 2019 04:50:32 -0500 Wed, 11 Dec 2019 04:50:32 -0500 WeaveIsUp Weave pod has set this
MemoryPressure False Wed, 11 Dec 2019 07:13:43 -0500 Tue, 10 Dec 2019 03:14:03 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 11 Dec 2019 07:13:43 -0500 Tue, 10 Dec 2019 03:14:03 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 11 Dec 2019 07:13:43 -0500 Tue, 10 Dec 2019 03:14:03 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 11 Dec 2019 07:13:43 -0500 Tue, 10 Dec 2019 03:56:47 -0500 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.0.2.15
Hostname: worker2
Capacity:
cpu: 1
ephemeral-storage: 14078Mi
hugepages-2Mi: 0
memory: 499552Ki
pods: 110
Allocatable:
cpu: 1
ephemeral-storage: 13285667614
hugepages-2Mi: 0
memory: 397152Ki
pods: 110
System Info:
Machine ID: 455146bc2c2f478a859bf39ac2641d79
System UUID: 68F543D7-EDBF-4AF6-8354-A99D96D994EF
Boot ID: 5775abf1-97dc-411f-a5a0-67f51cc8daf3
Kernel Version: 4.4.206-1.el7.elrepo.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.3
Kubelet Version: v1.13.0
Kube-Proxy Version: v1.13.0
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default ng2-569d45c6b5-ppkwg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 26h
kube-system weave-net-pbb86 20m (2%) 0 (0%) 0 (0%) 0 (0%) 26h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 20m (2%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>
I can see that you have different IPs not only in your pods, but also in your nodes.
As you can see in the kubectl describe node output InternalIP for worker1 is 192.168.5.12 and for worker2 is 10.0.2.15.
This is not expected behavior, so it's important to make sure you have attached both of your VirtualBox VMs to the same adapter type.
Both should be in the same network and in the comments you confirmed that this was the case and that explains this behavior.
Here is an example of that configuration:
As you mentioned on comments, the first node was added manually and the second was added during the TLS bootstraping and it got added even with "wrong" IP Address.
To solve this issue the best thing you can do is to bootstrap you cluster from scratch again using the same Adapter Settings on Virtual Box for all nodes.
PLEASE READ UPDATE 2
I have a very simple EventHubClient app. It will just listen to an EventHub messages.
I get it running with the Docker support given in Visual Studio 2017 (Linux Container).
But when I try to deploy it in Kubernetes, I get "Back-off restarting failed container"
C# Code:
public static void Main(string[] args)
{
// Init Mapper
AutoMapper.Mapper.Initialize(cfg =>
{
cfg.AddProfile<AiElementProfile>();
});
Console.WriteLine("Registering EventProcessor...");
var eventProcessorHost = new EventProcessorHost(
EventHubPath,
ConsumerGroupName,
EventHubConnectionString,
AzureStorageConnectionString,
ContainerName
);
// Registers the Event Processor Host and starts receiving messages
eventProcessorHost.RegisterEventProcessorAsync<EventProcessor>();
Console.WriteLine("Receiving. Press ENTER to stop worker.");
Console.ReadLine();
}
Kubernetes Manifest file (.yaml):
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: historysvc-deployment
spec:
selector:
matchLabels:
app: historysvc
replicas: 2
template:
metadata:
labels:
app: historysvc
spec:
containers:
- name: historysvc
image: vncont.azurecr.io/historysvc:v1
ports:
- containerPort: 80
imagePullSecrets:
- name: acr-auth
kubectl get pods:
NAME READY STATUS RESTARTS AGE
historysvc-deployment-558fc5649f-bln8f 0/1 CrashLoopBackOff 17 1h
historysvc-deployment-558fc5649f-jgjvq 0/1 CrashLoopBackOff 17 1h
kubectl describe pod historysvc-deployment-558fc5649f-bln8f
Name: historysvc-deployment-558fc5649f-bln8f
Namespace: default
Node: aks-nodepool1-81522366-0/10.240.0.4
Start Time: Tue, 24 Jul 2018 10:15:37 +0200
Labels: app=historysvc
pod-template-hash=1149712059
Annotations: <none>
Status: Running
IP: 10.244.0.11
Controlled By: ReplicaSet/historysvc-deployment-558fc5649f
Containers:
historysvc:
Container ID: docker://59e66f1e6420146f6eca4f19e2801a4ee0435a34c7ac555a8d04f699a1497f35
Image: vncont.azurecr.io/historysvc:v1
Image ID: docker-pullable://vncont.azurecr.io/historysvc#sha256:636d81435bd421ec92a0b079c3841cbeb3ad410509a6e37b1ec673dc4ab8a444
Port: 80/TCP
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 24 Jul 2018 10:17:10 +0200
Finished: Tue, 24 Jul 2018 10:17:10 +0200
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 24 Jul 2018 10:16:29 +0200
Finished: Tue, 24 Jul 2018 10:16:29 +0200
Ready: False
Restart Count: 4
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mt8mm (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-mt8mm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mt8mm
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1m default-scheduler Successfully assigned historysvc-deployment-558fc5649f-bln8f to aks-nodepool1-81522366-0
Normal SuccessfulMountVolume 1m kubelet, aks-nodepool1-81522366-0 MountVolume.SetUp succeeded for volume "default-token-mt8mm"
Normal Pulled 8s (x5 over 1m) kubelet, aks-nodepool1-81522366-0 Container image "vncont.azurecr.io/historysvc:v1" already present on machine
Normal Created 7s (x5 over 1m) kubelet, aks-nodepool1-81522366-0 Created container
Normal Started 6s (x5 over 1m) kubelet, aks-nodepool1-81522366-0 Started container
Warning BackOff 6s (x8 over 1m) kubelet, aks-nodepool1-81522366-0 Back-off restarting failed container
What am I missing?
UPDATE 1
kubectl describe pod historysvc-deployment-558fc5649f-jgjvq
Name: historysvc-deployment-558fc5649f-jgjvq
Namespace: default
Node: aks-nodepool1-81522366-0/10.240.0.4
Start Time: Tue, 24 Jul 2018 10:15:37 +0200
Labels: app=historysvc
pod-template-hash=1149712059
Annotations: <none>
Status: Running
IP: 10.244.0.12
Controlled By: ReplicaSet/historysvc-deployment-558fc5649f
Containers:
historysvc:
Container ID: docker://ccf83bce216276450ed79d67fb4f8a66daa54cd424461762478ec62f7e592e30
Image: vncont.azurecr.io/historysvc:v1
Image ID: docker-pullable://vncont.azurecr.io/historysvc#sha256:636d81435bd421ec92a0b079c3841cbeb3ad410509a6e37b1ec673dc4ab8a444
Port: 80/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 25 Jul 2018 09:32:34 +0200
Finished: Wed, 25 Jul 2018 09:32:35 +0200
Ready: False
Restart Count: 277
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mt8mm (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-mt8mm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mt8mm
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 2m (x6238 over 23h) kubelet, aks-nodepool1-81522366-0 Back-off restarting failed container
UPDATE 2
When I run it localy with:
docker run <image>
it ends instantly (ignores the read line) (completes), which seems to be the problem.
I have to write
docker run -it <image>
-it at the end for it to do the read line.
How does kubernetes runs the docker image? Where can I set that?
This can be done by attaching an argument to run with your deployment.
In your case the Kubernetes Manifest file (.yaml) should look like this:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: historysvc-deployment
spec:
selector:
matchLabels:
app: historysvc
replicas: 2
template:
metadata:
labels:
app: historysvc
spec:
containers:
- name: historysvc
image: vncont.azurecr.io/historysvc:v1
ports:
- containerPort: 80
args: ["-it"]
imagePullSecrets:
- name: acr-auth
You can find this explained in k8s docs inject-data-application/define-command-argument-container
When you create a Pod, you can define a command and arguments for the containers that run in the Pod. To define a command, include the command field in the configuration file. To define arguments for the command, include the args field in the configuration file. The command and arguments that you define cannot be changed after the Pod is created.
The command and arguments that you define in the configuration file override the default command and arguments provided by the container image. If you define args, but do not define a command, the default command is used with your new arguments.
i've a kubernetes cluster with a master node and 3 minions, i've already a glusterfs cluster, every node of kubernetes cluster have glusterfs-client installed and working.
i'm trying to run a pod ( a simple mysql ) mounting /var/lib/mysql on glusterfs but i see:
Image: mysql:5.6 is ready, container is creating
i run:
kubectl get event
i see:
Thu, 18 Feb 2016 10:08:01 +0100 Thu, 18 Feb 2016 10:08:01 +0100 1 mysql-9ym10 Pod scheduled {scheduler } Successfully assigned mysql-9ym10 to nodeXX
Thu, 18 Feb 2016 10:08:01 +0100 Thu, 18 Feb 2016 10:08:01 +0100 1 mysql ReplicationController successfulCreate {replication-controller } Created pod: mysql-9ym10
Thu, 18 Feb 2016 10:08:02 +0100 Thu, 18 Feb 2016 10:08:12 +0100 2 mysql-9ym10 Pod failedMount {kubelet nodeXX} Unable to mount volumes for pod "mysql-9ym10_default": exit status 1
Thu, 18 Feb 2016 10:08:02 +0100 Thu, 18 Feb 2016 10:08:12 +0100 2 mysql-9ym10 Pod failedSync {kubelet nodeXX} Error syncing pod, skipping: exit status 1
if i run
kubectl describe pod mysql-9ym10
i see:
Name: mysql-9ym10
Namespace: default
Image(s): mysql:5.6
Node: nodeXX/nodeXX
Labels: app=mysql
Status: Pending
Reason:
Message:
IP:
Replication Controllers: mysql (1/1 replicas created)
Containers:
mysql:
Image: mysql:5.6
State: Waiting
Reason: Image: mysql:5.6 is ready, container is creating
Ready: False
Restart Count: 0
Conditions:
Type Status
Ready False
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Thu, 18 Feb 2016 10:08:01 +0100 Thu, 18 Feb 2016 10:08:01 +0100 1 {scheduler } scheduled Successfully assigned mysql-9ym10 to nodeXX
Thu, 18 Feb 2016 10:08:02 +0100 Thu, 18 Feb 2016 10:10:22 +0100 15 {kubelet nodeXX} failedMount Unable to mount volumes for pod "mysql-9ym10_default": exit status 1
Thu, 18 Feb 2016 10:08:02 +0100 Thu, 18 Feb 2016 10:10:22 +0100 15 {kubelet nodeXX} failedSync Error syncing pod, skipping: exit status 1
this is the yaml file for container:
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
name: mysql
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:5.6
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: password
volumeMounts:
- mountPath: /var/lib/mysql
name: glusterfsvol
volumes:
- glusterfs:
endpoints: glusterfs-cluster
path: glustervolume
readOnly: false
name: glusterfsvol
i've got and endpoint that is configured with glusterfs ip addresses.
i know the posted link, i've followed it but the result is on my first post!
On first: To use a GlusterFS you don't need to install glusterfs-client on kubernetes node. Kubernetes have the volume mounting option for glusterfs by default.
To use a glusterfs with kubernetes you need to things.
a working glusterfs server. a running volume in the glusterfs server. I assume you have those. If anyone don't then create a glusterfs server and start your volumes with the following commands
$ gluster volume create <volume-name> replica 2 transport tcp \
peer1:/directory \
peer2:/directory \
force
$ gluster volume start <vonlume-name>
$ sudo gluster volume info
if this is ok, you need an kubernetes endpoint to use with the pod. as far an example a end point is like this.
kind: Endpoints
apiVersion: v1
metadata:
name: glusterfs
subsets:
- addresses:
- ip: peer1
ports:
- port: 1
- addresses:
- ip: peer2
ports:
- port: 1
And at third mount the gfs volume to a pod with the end point.
containers:
- name: mysql
image: mysql:5.6
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: password
volumeMounts:
- mountPath: /var/lib/mysql
name: glusterfsvol
volumes:
- glusterfs:
endpoints: glusterfs-cluster
path: <volume-name>
name: glusterfsvol
**The path must match the volume name with the glusterfs.
this all should work fine.
You need to configure Endpoints https://github.com/kubernetes/kubernetes/blob/release-1.1/examples/glusterfs/README.md , otherwise kubernetes doesn't know how to access your gluster cluster.
i've a simple kubernetes cluster with a master and 3 minions. In this scenario, if i run a simple pod of a nginx or a mysql it works properly but, if i change type of KIND into yaml file and i try to run a replicated service, pods will start but i can't access to the service.
this is my yaml file for nginx with 3 replicas:
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx
spec:
replicas: 3
selector:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
this is service yaml config file:
apiVersion: v1
kind: Service
metadata:
labels:
name: nginx
name: nginx
spec:
ports:
- port: 80
selector:
name: nginx
i run it with:
# kubectl create -f nginx-rc.yaml
# kubectl create -f nginx-rc-service.yaml
if i run:
# kubectl get pod,svc,rc -o wide
i see:
NAME READY STATUS RESTARTS AGE NODE
nginx-kgq1s 1/1 Running 0 1m node01
nginx-pomx3 1/1 Running 0 1m node02
nginx-xi54i 1/1 Running 0 1m node03
NAME LABELS SELECTOR IP(S) PORT(S)
kubernetes component=apiserver,provider=kubernetes <none> 10.254.0.1 443/TCP
nginx name=nginx name=nginx 10.254.47.150 80/TCP
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS
nginx nginx nginx app=nginx 3
i can see description for pod:
Name: nginx-kgq1s
Namespace: default
Image(s): nginx
Node: node01/node01
Labels: app=nginx
Status: Running
Reason:
Message:
IP: 172.17.52.3
Replication Controllers: nginx (3/3 replicas created)
Containers:
nginx:
Image: nginx
State: Running
Started: Thu, 11 Feb 2016 16:28:08 +0100
Ready: True
Restart Count: 0
Conditions:
Type Status
Ready True
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Thu, 11 Feb 2016 16:27:47 +0100 Thu, 11 Feb 2016 16:27:47 +0100 1 {scheduler } scheduled Successfully assigned nginx-kgq1s to node01
Thu, 11 Feb 2016 16:27:57 +0100 Thu, 11 Feb 2016 16:27:57 +0100 1 {kubelet node01} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Thu, 11 Feb 2016 16:28:02 +0100 Thu, 11 Feb 2016 16:28:02 +0100 1 {kubelet node01} implicitly required container POD created Created with docker id bed30a90c6eb
Thu, 11 Feb 2016 16:28:02 +0100 Thu, 11 Feb 2016 16:28:02 +0100 1 {kubelet node01} implicitly required container POD started Started with docker id bed30a90c6eb
Thu, 11 Feb 2016 16:28:07 +0100 Thu, 11 Feb 2016 16:28:07 +0100 1 {kubelet node01} spec.containers{nginx} created Created with docker id 0a5c69cd0481
Thu, 11 Feb 2016 16:28:08 +0100 Thu, 11 Feb 2016 16:28:08 +0100 1 {kubelet node01} spec.containers{nginx} started Started with docker id 0a5c69cd0481
this is what i see if i get description for rc:
Name: nginx
Namespace: default
Image(s): nginx
Selector: app=nginx
Labels: app=nginx
Replicas: 3 current / 3 desired
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Thu, 11 Feb 2016 16:27:47 +0100 Thu, 11 Feb 2016 16:27:47 +0100 1 {replication-controller } successfulCreate Created pod: nginx-kgq1s
Thu, 11 Feb 2016 16:27:47 +0100 Thu, 11 Feb 2016 16:27:47 +0100 1 {replication-controller } successfulCreate Created pod: nginx-pomx3
Thu, 11 Feb 2016 16:27:47 +0100 Thu, 11 Feb 2016 16:27:47 +0100 1 {replication-controller } successfulCreate Created pod: nginx-xi54i
and this is what i see if i get description of service:
Name: nginx
Namespace: default
Labels: name=nginx
Selector: name=nginx
Type: ClusterIP
IP: 10.254.47.150
Port: <unnamed> 80/TCP
Endpoints: <none>
Session Affinity: None
No events.
as i can see, the problem may be that i don't have an ENDPOINT but i don't have any idea how i could solve.
It looks to me like the selector for your service is wrong. It's looking for a label of name: nginx, but your pods actually have app: nginx.
Try changing your service file to:
apiVersion: v1
kind: Service
metadata:
labels:
name: nginx
name: nginx
spec:
ports:
- port: 80
selector:
app: nginx
... or change your replication controller template to use name: nginx instead of app: nginx as the label. Basically, the labels have to match so that the service knows how to present a unified facade over your pods.
To build on #jonskeet's answer, the reason the labels have to match is because the Pods can run on any node in your k8s cluster and Services need a way to locate them.
Therefore, the Service you're slapping in front of the Pod, needs to be able to filter through the cluster and, particularly, the set of Pods in the namespace it's in, and it leverages these matching k/v's in both selectors as its methodology to do so.