I would like to create a kubernetes pod that contains 2 containers, both with different images, so I can start both containers together.
Currently I have tried the following configuration:
{
"id": "podId",
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "podId",
"containers": [{
"name": "type1",
"image": "local/image"
},
{
"name": "type2",
"image": "local/secondary"
}]
}
},
"labels": {
"name": "imageTest"
}
}
However when I execute kubecfg -c app.json create /pods I get the following error:
F0909 08:40:13.028433 01141 kubecfg.go:283] Got request error: request [&http.Request{Method:"POST", URL:(*url.URL)(0xc20800ee00), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{}, B
ody:ioutil.nopCloser{Reader:(*bytes.Buffer)(0xc20800ed20)}, ContentLength:396, TransferEncoding:[]string(nil), Close:false, Host:"127.0.0.1:8080", Form:url.Values(nil), PostForm:url.Values(nil), Multi
partForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil)}] failed (500) 500 Internal Server Error: {"kind":"Status","creationTimestamp":
null,"apiVersion":"v1beta1","status":"failure","message":"failed to find fit for api.Pod{JSONBase:api.JSONBase{Kind:\"\", ID:\"SSH podId\", CreationTimestamp:util.Time{Time:time.Time{sec:63545848813, nsec
:0x14114e1, loc:(*time.Location)(0xb9a720)}}, SelfLink:\"\", ResourceVersion:0x0, APIVersion:\"\"}, Labels:map[string]string{\"name\":\"imageTest\"}, DesiredState:api.PodState{Manifest:api.ContainerMa
nifest{Version:\"v1beta1\", ID:\"podId\", Volumes:[]api.Volume(nil), Containers:[]api.Container{api.Container{Name:\"type1\", Image:\"local/image\", Command:[]string(nil), WorkingDir:\"\", Ports:[]ap
i.Port(nil), Env:[]api.EnvVar(nil), Memory:0, CPU:0, VolumeMounts:[]api.VolumeMount(nil), LivenessProbe:(*api.LivenessProbe)(nil)}, api.Container{Name:\"type2\", Image:\"local/secondary\", Command:[]string(n
il), WorkingDir:\"\", Ports:[]api.Port(nil), Env:[]api.EnvVar(nil), Memory:0, CPU:0, VolumeMounts:[]api.VolumeMount(nil), LivenessProbe:(*api.LivenessProbe)(nil)}}}, Status:\"\", Host:\"\", HostIP:\"\
", PodIP:\"\", Info:api.PodInfo(nil), RestartPolicy:api.RestartPolicy{Type:\"RestartAlways\"}}, CurrentState:api.PodState{Manifest:api.ContainerManifest{Version:\"\", ID:\"\", Volumes:[]api.Volume(nil
), Containers:[]api.Container(nil)}, Status:\"\", Host:\"\", HostIP:\"\", PodIP:\"\", Info:api.PodInfo(nil), RestartPolicy:api.RestartPolicy{Type:\"\"}}}","code":500}
How can I modify the configuration accordingly?
Running kubernetes on a vagrant vm (yungsang/coreos).
The error in question here is "failed to find fit". This generally happens when you have a port conflict (try and use the same hostPort too many times or perhaps you don't have any worker nodes/minions.
I'd suggest you either use the Vagrant file that is in the Kubernetes git repo (see http://kubernetes.io) as we have been trying to make sure that stays working as Kubernetes is under very active development. If you want to make it work with the CoreOS single machine set up, I suggest you hop on IRC (#google-containers on freenode) and try and get in touch with Kelsey Hightower.
Your pod spec file looks like invalid.
According to http://kubernetes.io/v1.0/docs/user-guide/walkthrough/README.html#multiple-containers, a valid multiple containers pod spec should like this
apiVersion: v1
kind: Pod
metadata:
name: www
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- mountPath: /srv/www
name: www-data
readOnly: true
- name: git-monitor
image: kubernetes/git-monitor
env:
- name: GIT_REPO
value: http://github.com/some/repo.git
volumeMounts:
- mountPath: /data
name: www-data
volumes:
- name: www-data
emptyDir: {}
Latest doc at http://kubernetes.io/docs/user-guide/walkthrough/#multiple-containers
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- name: wp
image: wordpress
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
- name: ng
image: nginx
imagePullPolicy: IfNotPresent
Related
I have deployed a service on Knative. I iterated on the service code/Docker image and I try to redeploy it at the same address. I proceeded as follow:
Pushed the new Docker image on our private Docker repo
Updated the service YAML file to point to the new Docker image (see YAML below)
Delete the service with the command: kubectl -n myspacename delete -f myservicename.yaml
Recreate the service with the command: kubectl -n myspacename apply -f myservicename.yaml
During the deployment, the service shows READY = Unknown and REASON = RevisionMissing, and after a while, READY = False and REASON = ProgressDeadlineExceeded. When looking at the logs of the pod with the following command kubectl -n myspacename logs revision.serving.knative.dev/myservicename-00001, I get the message:
no kind "Revision" is registered for version "serving.knative.dev/v1" in scheme "pkg/scheme/scheme.go:28"
Here is the YAML file of the service:
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: myservicename
namespace: myspacename
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/metric: concurrency
autoscaling.knative.dev/target: '1'
autoscaling.knative.dev/minScale: '0'
autoscaling.knative.dev/maxScale: '5'
autoscaling.knative.dev/scaleDownDelay: 60s
autoscaling.knative.dev/window: 600s
spec:
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
volumes:
- name: nfs-volume
persistentVolumeClaim:
claimName: myspacename-models-pvc
imagePullSecrets:
- name: myrobotaccount-pull-secret
containers:
- name: myservicename
image: quay.company.com/project/myservicename:0.4.0
ports:
- containerPort: 5000
name: user-port
protocol: TCP
resources:
limits:
cpu: "4"
memory: 36Gi
nvidia.com/gpu: 1
requests:
cpu: "2"
memory: 32Gi
volumeMounts:
- name: nfs-volume
mountPath: /tmp/static/
securityContext:
privileged: true
env:
- name: CLOUD_STORAGE_PASSWORD
valueFrom:
secretKeyRef:
name: myservicename-cloud-storage-password
key: key
envFrom:
- configMapRef:
name: myservicename-config
The protocol I followed above is correct, the problem was because of a bug in the code of the Docker image that Knative is serving. I was able to troubleshoot the issue by looking at the logs of the pods as follow:
First run the following command to get the pod name: kubectl -n myspacename get pods. Example of pod name = myservicename-00001-deployment-56595b764f-dl7x6
Then get the logs of the pod with the following command: kubectl -n myspacename logs myservicename-00001-deployment-56595b764f-dl7x6
I am using pushgateway to exposes metrics coming from short-lived batch jobs.
At the moment the pushgateway instance is launched on a baremetal machine, where I have a docker volume mounted to allow survival of metrics in case of a container restart (in conjunction with the --persistence.file parameter).
Here an extract of the docker-compose.yml file used to run the container:
pushgateway:
image: prom/pushgateway:v1.2.0
restart: unless-stopped
volumes:
- pushgw-data:/data
ports:
- "${PUSHGW_PORT:-9091}:9091"
command: --persistence.file="/data/metric.store"
I am moving to a (private) kubernetes cluster without persistent volumes, but equipped with an s3-compatible object storage.
From this issue on github it seems possible to target s3 for the checkpointing, but without further input I am not sure how to achieve this, and that's the best I could find by searching the Web for information.
Can anyone point me in the right direction?
So finally https://serverfault.com/questions/976764/kubernetes-run-aws-s3-sync-rsync-against-persistent-volume-on-demand pointed me in the right direction.
This is an extract of the deployment.yaml descriptor which works as expected:
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: {{K8S_NAMESPACE}}
name: {{K8S_DEPLOYMENT_NAME}}
spec:
selector:
matchLabels:
name: {{K8S_DEPLOYMENT_NAME}}
strategy:
type: Recreate
template:
metadata:
labels:
name: {{K8S_DEPLOYMENT_NAME}}
version: v1
spec:
containers:
- name: {{AWSCLI_NAME}}
image: {{IMAGE_AWSCLI}}
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: {{SECRET_NAME}}
key: accesskey
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{SECRET_NAME}}
key: secretkey
command: [ "/bin/bash",
"-c",
"aws --endpoint-url {{ENDPOINT_URL}} s3 sync s3://{{BUCKET}} /data; while true; do aws --endpoint-url {{ENDPOINT_URL}} s3 sync /data s3://{{BUCKET}}; sleep 60; done" ]
volumeMounts:
- name: pushgw-data
mountPath: /data
- name: {{PUSHGATEWAY_NAME}}
image: {{IMAGE_PUSHGATEWAY}}
command: [ '/bin/sh', '-c' ]
args: [ 'sleep 10; /bin/pushgateway --persistence.file=/data/metric.store' ]
ports:
- containerPort: 9091
volumeMounts:
- name: pushgw-data
mountPath: /data
volumes:
- name: pushgw-data
emptyDir: {}
- name: config-volume
configMap:
name: {{K8S_DEPLOYMENT_NAME}}
imagePullSecrets:
- name: harbor-bot
restartPolicy: Always
Note the override of entrypoint for the docker image of the pushgateway. In my case I have put 10 seconds delay to start, you might need to tune the delay to suits your needs. This delay is needed because the pushgateway container will boot faster than the sidecar (also due to the network exchange with s3, I suppose).
If the pushgateway starts when not metric store file is already present, it won't be used/considered. But it gets worse, when you first send data to the pushgateway, it will override the file. At that point, the "sync" from the sidecar container will also override the original "copy", so please pay attention and be sure you have a backup of the metrics file before experimenting with this delay value.
I am running my kubernetes cluster on AWS EKS which runs kubernetes 1.10.
I am following this guide to deploy elasticsearch in my Cluster
elasticsearch Kubernetes
The first time I deployed it everything worked fine. Now, When I redeploy it gives me the following error.
ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2018-08-24T18:07:28,448][INFO ][o.e.n.Node ] [es-master-6987757898-5pzz9] stopping ...
[2018-08-24T18:07:28,534][INFO ][o.e.n.Node ] [es-master-6987757898-5pzz9] stopped
[2018-08-24T18:07:28,534][INFO ][o.e.n.Node ] [es-master-6987757898-5pzz9] closing ...
[2018-08-24T18:07:28,555][INFO ][o.e.n.Node ] [es-master-6987757898-5pzz9] closed
Here is my deployment file.
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: es-master
labels:
component: elasticsearch
role: master
spec:
replicas: 3
template:
metadata:
labels:
component: elasticsearch
role: master
spec:
initContainers:
- name: init-sysctl
image: busybox:1.27.2
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
containers:
- name: es-master
image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.2
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: CLUSTER_NAME
value: myesdb
- name: NUMBER_OF_MASTERS
value: "2"
- name: NODE_MASTER
value: "true"
- name: NODE_INGEST
value: "false"
- name: NODE_DATA
value: "false"
- name: HTTP_ENABLE
value: "false"
- name: ES_JAVA_OPTS
value: -Xms512m -Xmx512m
- name: NETWORK_HOST
value: "0.0.0.0"
- name: PROCESSORS
valueFrom:
resourceFieldRef:
resource: limits.cpu
resources:
requests:
cpu: 0.25
limits:
cpu: 1
ports:
- containerPort: 9300
name: transport
livenessProbe:
tcpSocket:
port: transport
initialDelaySeconds: 20
periodSeconds: 10
volumeMounts:
- name: storage
mountPath: /data
volumes:
- emptyDir:
medium: ""
name: "storage"
I have seen a lot of posts talking about increasing the value but I am not sure how to do it. Any help would be appreciated.
Just want to append to this issue:
If you create EKS cluster by eksctl then you can append to NodeGroup creation yaml:
preBootstrapCommand:
- "sed -i -e 's/1024:4096/65536:65536/g' /etc/sysconfig/docker"
- "systemctl restart docker"
This will solve the problem for newly created cluster by fixing docker daemon config.
Update default-ulimit parameter in the file '/etc/docker/daemon.json'
"default-ulimits": {
"nofile": {
"Name": "nofile",
"Soft": 65536,
"Hard": 65536
}
}
and restart docker daemon.
This is the only thing that worked for me using EKS setting up an EFK stack. Add this to your nodegroup creation YAML file under nodeGroups:. Then create your nodegroup and apply your ES pods on it.
preBootstrapCommands:
- "sysctl -w vm.max_map_count=262144"
- "systemctl restart docker"
I have created a shared library and it has a groovy file named myBuildPlugin.groovy:
def label = "worker-${UUID.randomUUID().toString()}"
podTemplate(label: label, yaml: """
apiVersion: v1
kind: Pod
metadata:
name: my-build
spec:
containers:
- name: jnlp
image: dtr.myhost.com/test/jenkins-build-agent:latest
ports:
- containerPort: 8080
- containerPort: 50000
resources:
limits:
cpu : 1
memory : 1Gi
requests:
cpu: 200m
memory: 256Mi
env:
- name: JENKINS_URL
value: http://jenkins:8080
- name: mongo
image: mongo
ports:
- containerPort: 8080
- containerPort: 50000
- containerPort: 27017
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 1
memory: 512Mi
imagePullSecrets:
- name: dtrsecret""")
{
node(label) {
pipelineParams.step1.call([label : label])
}
}
When in my project I use myBuildPlugin as below, the log shows it waits for an executor forever. When I look at Jenkins I can see the agent is being created but for some reason it can't talk to it via port 50000 (or perhaps the pod can't talk to the agent!)
Later I tried to remove yaml and instead used the following code:
podTemplate(label: 'mypod', cloud: 'kubernetes', containers: [
containerTemplate(
name: 'jnlp',
image: 'dtr.myhost.com/test/jenkins-build-agent:latest',
ttyEnabled: true,
privileged: false,
alwaysPullImage: false,
workingDir: '/home/jenkins',
resourceRequestCpu: '1',
resourceLimitCpu: '100m',
resourceRequestMemory: '100Mi',
resourceLimitMemory: '200Mi',
envVars: [
envVar(key: 'JENKINS_URL', value: 'http://jenkins:8080'),
]
),
containerTemplate(name: 'maven', image: 'maven:3.5.0', command: 'cat', ttyEnabled: true),
containerTemplate(name: 'docker', image: 'docker', command: 'cat', ttyEnabled: true)
],
volumes: [
emptyDirVolume(mountPath: '/etc/mount1', memory: false),
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')
],
imagePullSecrets: [ 'dtrsecret' ],
)
{
node(label) {
pipelineParams.step1.call([label : label])
}
}
Still no luck. Interestingly if I define all these containers in Jenkins configuration, things work smoothly. This is my configuration:
and this is the pod template configuration:
It appears that if I change the label to something other that jenkins-jenkins-slave the issue happens. This is the case even if it's defined via Jenkins' configuration page. If that's the case, how am I suppose to create multiple Pod template for different type of projects?
Just today, I also tried to use pod inheritance as below without any success:
def label = 'kubepod-test'
podTemplate(label : label, inheritFrom : 'default',
containers : [
containerTemplate(name : 'mongodb', image : 'mongo', command : '', ttyEnabled : true)
]
)
{
node(label) {
}
}
Please help me on this issue. Thanks
There's something iffy about your pod configuration, you can't have your Jenkins and Mongo containers using the same port 50000. Generally, you want to specify a unique port since pods share the same port space.
In this case looks like you need port 50000 to set up a tunnel to the Jenkins agent. Keep in mind that the Jenkins plugin might be doing other things such as setting up a Kubernetes Service or using the internal Kubernetes DNS.
In the second example, I don't even see port 50000 exposed.
I've just finished Google's tutorial on how to implement continuous integration for a Go app on Kubernetes using Jenkins, and it works great. I'm now trying to do the same thing with a Node app that is served on port 3001, but I keep getting this error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "services \"gceme-frontend\" not found",
"reason": "NotFound",
"details": {
"name": "gceme-frontend",
"kind": "services"
},
"code": 404
}
The only thing I've changed on the routing side is having the load balancer point to 3001 instead of 80, since that's where the Node app is listening. I have a very strong feeling that the error is somewhere in the .yaml files.
My node server (relevant part):
const PORT = process.env.PORT || 3001;
frontend-dev.yaml: (this is applied to the dev environment)
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: gceme-frontend-dev
spec:
replicas:
template:
metadata:
name: frontend
labels:
app: gceme
role: frontend
env: dev
spec:
containers:
- name: frontend
image: gcr.io/cloud-solutions-images/gceme:1.0.0
resources:
limits:
memory: "500Mi"
cpu: "100m"
imagePullPolicy: Always
ports:
- containerPort: 3001
protocol: TCP
services/frontend.yaml:
kind: Service
apiVersion: v1
metadata:
name: gceme-frontend
spec:
type: LoadBalancer
ports:
- name: http
#THIS PORT ACTUALLY GOES IN THE URL: i.e. gcme-frontend: ****
#when it says "no endpoints available for service, that doesn't mean this one is wrong, it means that target port is not working not exist"
port: 80
#matches port and -port in frontend-*.yaml
targetPort: 3001
protocol: TCP
selector:
app: gceme
role: frontend
Jenkinsfile (for dev branches, which is what I'm trying to get working)
sh("kubectl get ns ${env.BRANCH_NAME} || kubectl create ns ${env.BRANCH_NAME}")
// Don't use public load balancing for development branches
sh("sed -i.bak 's#LoadBalancer#ClusterIP#' ./k8s/services/frontend.yaml")
sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/dev/*.yaml")
sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/services/")
sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/dev/")
echo 'To access your environment run `kubectl proxy`'
echo "Then access your service via http://localhost:8001/api/v1/proxy/namespaces/${env.BRANCH_NAME}/services/${feSvcName}:80/"
Are you creating Service or Ingress resources to expose your application to the outside world?
See tutorials:
https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app
https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer
which have working examples you can copy and modify.