I've built a custom-spark docker image with the following dependencies:
Python 3.6.9
Pip 1.18
Java OpenJDK 64-Bit Server VM, 1.8.0_212
Hadoop 3.2
Scala 2.13.0
Spark 3.0.3
where I pushed to ducker hub: https://hub.docker.com/r/redaer7/custom-spark
Dockerfile,spark-master and spark-worker files are stored under: https://github.com/redaER7/Custom-Spark
I verify /spark-master and /spark-worker works well when creating a container linked to the previous image:
docker run -it -d --name spark_1 redaer7/custom-spark:1.0 bash
docker exec -it $CONTAINER_ID /bin/bash
My issue is when I try to build a K8s cluster from previous image with following yaml file for the spark master pod:
kubectl create namespace sparkspace
kubectl -n sparkspace create -f ./spark-master-deployment.yaml
#yaml file
kind: Deployment
apiVersion: apps/v1
metadata:
name: spark-master
spec:
replicas: 1 # should always be one
selector:
matchLabels:
component: spark-master
template:
metadata:
labels:
component: spark-master
spec:
containers:
- name: spark-master
image: redaer7/custom-spark:1.0
imagePullPolicy: IfNotPresent
command: ["/spark-master"]
ports:
- containerPort: 7077
- containerPort: 8080
resources:
# limits:
# cpu: 1
# memory: 1G
requests:
cpu: 1 #100m
memory: 1G
I get CrashLoopBackOff when viewing pod with kubectl -n sparkspace get pods
When inspecting with kubectl -n sparkspace describe pod $Pod_Name
Any clue about that First warning ? thank you
I simply solved it by re-pulling the image :
imagePullPolicy: Always
Because I edited the Docker Image locally and I haven't changed the following in the config file:
imagePullPolicy: IfNotPresent
Then, I pushed it into Dockerhub for later deployment
Related
I'm still new to Kubernetes. I'm trying to run a ubuntu container and a linux kali container within the same pod on kubernetes. I also need those two containers to be able to be accessed from a browser. My approach right now is using ubuntu and kali docker image with VNC installed.
Here are the docker image that I'm trying to use:
https://hub.docker.com/r/consol/ubuntu-xfce-vnc (Ubuntu image)
https://hub.docker.com/r/jgamblin/kalibrowser-lxde (Kali image)
Here is the YAML file for creating the pod:
apiVersion: v1
kind: Pod
metadata:
name: training
labels:
app: training
spec:
containers:
- name: kali
image: jgamblin/kalibrowser-lxde
ports:
- containerPort: 6080
- name: centos
image: consol/centos-xfce-vnc
ports:
- containerPort: 5901
Here's the problem. When I run the pod with those 2 containers, only the Kali container is having issue running, cause it to keep on restarting.
May I know how I can achieve this?
You can add a simple sleep command to be executed inside then container to keep it running, for example:
apiVersion: v1
kind: Pod
metadata:
name: training
labels:
app: training
spec:
containers:
- name: kali
image: jgamblin/kalibrowser-lxde
ports:
- containerPort: 6080
command: ["bash", "-c"]
args: ["sleep 500"]
- name: centos
image: consol/centos-xfce-vnc
ports:
- containerPort: 5901`
This way the pod will be in running state:
kubectl get pod
NAME READY STATUS RESTARTS AGE
training 2/2 Running 0 81s
jgamblin/kalibrowser-lxde image require tty (display) allocation.
You can see an example command on docker hub page.
Then you should allow it in your Pod manifest:
apiVersion: v1
kind: Pod
metadata:
name: training
labels:
app: training
spec:
containers:
- name: kali
image: jgamblin/kalibrowser-lxde
ports:
- containerPort: 6080
tty: true
- name: centos
image: consol/centos-xfce-vnc
ports:
- containerPort: 5901
Put tty: true in kali container declaration.
I deploy Redis container via Kubernetes and get the following warning:
WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled
Is it possible to disable THP via Kubernetes? Perhaps via init-containers?
Yes, with init-containers it's quite straightforward:
apiVersion: v1
kind: Pod
metadata:
name: thp-test
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 1
volumes:
- name: host-sys
hostPath:
path: /sys
initContainers:
- name: disable-thp
image: busybox
volumeMounts:
- name: host-sys
mountPath: /host-sys
command: ["sh", "-c", "echo never >/host-sys/kernel/mm/transparent_hugepage/enabled"]
containers:
- name: busybox
image: busybox
command: ["cat", "/sys/kernel/mm/transparent_hugepage/enabled"]
Demo (notice that this is a system wide setting):
$ ssh THATNODE cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
$ kubectl create -f thp-test.yaml
pod "thp-test" created
$ kubectl logs thp-test
always madvise [never]
$ kubectl delete pod thp-test
pod "thp-test" deleted
$ ssh THATNODE cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
Ay,
I don't know if what I did is a good idea but we needed to deactivate THP on all our K8S VMs for all our apps. So I used a DaemonSet instead of adding an init-container to all our stacks :
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: thp-disable
namespace: kube-system
spec:
selector:
matchLabels:
name: thp-disable
template:
metadata:
labels:
name: thp-disable
spec:
restartPolicy: Always
terminationGracePeriodSeconds: 1
volumes:
- name: host-sys
hostPath:
path: /sys
initContainers:
- name: disable-thp
image: busybox
volumeMounts:
- name: host-sys
mountPath: /host-sys
command: ["sh", "-c", "echo never >/host-sys/kernel/mm/transparent_hugepage/enabled"]
containers:
- name: busybox
image: busybox
command: ["watch", "-n", "600", "cat", "/sys/kernel/mm/transparent_hugepage/enabled"]
I think it's a little dirty but it works.
Hope doing good all.
Env: centos 7.3.1611, kubernetes : 1.5, docker 1.12
Problem 1 : Extended jboss docker not working but docker image created successfully
POD gets an error see below, step 7.
Problem 2 : Once problem #1 fixed then i wish to upload to docker hub: https://hub.docker.com/
how can i upload steps please if possible.
1) pull
docker pull jboss/wildfly
2) vi Dockerfile
FROM jboss/wildfly
RUN /opt/jboss/wildfly/bin/add-user.sh admin admin123$ --silent
CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]
3) Extend docker image
docker build --tag=nbasetty/wildfly-server .
4) [root#centos7 custom-jboss]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nbasetty/wildfly-server latest c1fbb87faffd 43 minutes ago 583.8 MB
docker.io/httpd latest e0645af13ada 2 weeks ago 177.5 MB
5)vi jboss-wildfly-rc-service-custom.yaml
apiVersion: v1
kind: Service
metadata:
name: wildfly-service
spec:
externalIPs:
- 10.0.2.15
selector:
app: wildfly-rc-pod
ports:
- name: web
port: 8080
#- name: admin-console
# port: 9990
type: LoadBalancer
---
apiVersion: v1
kind: ReplicationController
metadata:
name: wildfly-rc
spec:
replicas: 2
template:
metadata:
labels:
app: wildfly-rc-pod
spec:
containers:
- name: wildfly
image: nbasetty/wildfly-server
ports:
- containerPort: 8080
#- containerPort: 9990
6) kubectl create -f jboss-wildfly-rc-service-custom.yaml
7) [root#centos7 jboss]# kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-pvc-pod 1/1 Running 6 2d
wildfly-rc-d0k3h 0/1 ImagePullBackOff 0 23m
wildfly-rc-hgsfj 0/1 ImagePullBackOff 0 23m
[root#centos7 jboss]# kubectl logs wildfly-rc-d0k3h
Error from server (BadRequest): container "wildfly" in pod
"wildfly-rc-d0k3h" is waiting to start:
trying and failing to pull image
Glad you have found a way to make it working. here is step I followed.
I labeled node-01 as 'dbserver: mysql'
create the docker image in node-01
created this pod, it worked.
apiVersion: v1 kind: ReplicationController metadata: name: wildfly-rc spec: replicas: 2 template:
metadata:
labels:
app: wildfly-rc-pod
spec:
containers:
- name: wildfly
image: nbasetty/wildfly-server
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
nodeSelector:
dbserver: mysql
Re-creating the issue:
docker pull jboss/wildfly
mkdir jw
cd jw
echo 'FROM jboss/wildfly
RUN /opt/jboss/wildfly/bin/add-user.sh admin admin123$ --silent
CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]' | tee Dockerfile
docker build --tag=docker.io/surajd/wildfly-server .
See the images available:
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/surajd/wildfly-server latest 10e96902ea12 11 seconds ago 583.8 MB
Create a config that works:
echo '
apiVersion: v1
kind: Service
metadata:
name: wildfly
spec:
selector:
app: wildfly
ports:
- name: web
port: 8080
type: LoadBalancer
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: wildfly
spec:
replicas: 2
template:
metadata:
labels:
app: wildfly
spec:
containers:
- name: wildfly
image: docker.io/surajd/wildfly-server
imagePullPolicy: Never
ports:
- containerPort: 8080
' | tee config.yaml
kubectl create -f config.yaml
Notice the field imagePullPolicy: Never, this helps you use the image available on the node(the image we built using docker build). This works on single node cluster but may or may not work on multiple node cluster. So not recommended to put that value, but since we are doing experiment on single node cluster we can set it to Never. Always set it to imagePullPolicy: Always. So that whenever the pod is scheduled the image will be pulled from registry. Read about imagePullPolicy and some config related tips.
Now to pull the image from registry the image should be on registry, so to answer your question of pushing it to docker hub run command:
docker push docker.io/surajd/wildfly-server
So in the above example replace surajd with your docker registry username.
Here are steps I used to do setup of single node cluster on CentOS:
My machine version:
$ cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
Here is what I have done:
Setup single node k8s cluster on CentOS as follows (src1 & src2):
yum update -y
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y docker kubelet kubeadm kubectl kubernetes-cni
systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet
sysctl net.bridge.bridge-nf-call-iptables=1
sysctl net.bridge.bridge-nf-call-ip6tables=1
kubeadm init
cp /etc/kubernetes/admin.conf $HOME/
chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf
kubectl taint nodes --all node-role.kubernetes.io/master-
Now k8s version:
# kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
I am very new to Kubernetes and I setup Kubernetes Registry addons just copy and pasting the yaml from Kubernetes Registry Addon just a small change in ReplicationController with emptyDir
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry-upstream
version: v0
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-registry-upstream
version: v0
template:
metadata:
labels:
k8s-app: kube-registry-upstream
version: v0
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: registry
image: registry:2
resources:
limits:
cpu: 100m
memory: 100Mi
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
ports:
- containerPort: 5000
name: registry
protocol: TCP
volumes:
- name: image-store
emptyDir: {}
Then I forward the 5000 port as follows
$POD=$(kubectl get pods --namespace kube-system -l k8s-app=kube-registry-upstream \
-o template --template '{{range .items}}{{.metadata.name}} {{.status.phase}}{{"\n"}}{{end}}' \
| grep Running | head -1 | cut -f1 -d' ')
$kubectl port-forward --namespace kube-system $POD 5000:5000 &
I can push my images fine as follows
$docker tag alpine localhost:5000/nurrony/alpine
$docker push localhost:5000/nurrony/alpine
Then I write a Pod to test it like below
Version: v1
kind: Pod
metadata:
name: registry-demo
labels:
purpose: registry-demo
spec:
containers:
- name: registry-demo-container
image: localhost:5000/nurrony/alpine
command: ["printenv"]
args: ["HOSTNAME", "KUBERNETES_PORT"]
env:
- name: MESSAGE
value: "hello world"
command: ["/bin/echo"]
args: ["$(MESSAGE)"]
It is throwing an error
Failed to pull image "localhost:5000/nurrony/alpine": image pull failed for localhost:5000/nurrony/alpine:latest, this may be because there are no credentials on this request. details: (net/http: request canceled)
Any idea why is this happening? Thanks in advance.
Most likely your proxy is not working.
The Docker Registry K8S addon comes with DaemonSet which defines registry proxy for every node which runes your kubelets. What I would suggest you is to inspect those proxies since they will map Docker Registry (K8S) Service to localhost:5000 on every node.
Please note, that even if you have green check mark on your registry proxies that does not mean they work correctly. Open the logs of them and make sure that everything is working.
If your proxy is configured and you are still getting this error then most likely environment variable REGISTRY_HOST inside kube-registry-proxy is wrong. Are you using DNS here like in example? Is your DNS configured correctely? Is it working if you put this variable to ClusterIP of your service?
Also, please be aware that your RC labels need to match SVC selectors, otherwise service cannot discover your pods.
Hope it helps.
Hi I am running kubernetes cluster where I run mailhog container.
But I need to run it with own docker run parameter. If I would run it in docker directly. I would use command:
docker run mailhog/mailhog -auth-file=./auth.file
But I need to run it via Kubernetes pod. My pod looks like:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mailhog
spec:
replicas: 1
revisionHistoryLimit: 1
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: mailhog
spec:
containers:
- name: mailhog
image: us.gcr.io/com/mailhog:1.0.0
ports:
- containerPort: 8025
How to achieve to run Docker container with parameter -auth-file=./auth.file via kubernetes. Thanks.
I tried adding under containers
command: ["-auth-file", "/data/mailhog/auth.file"]
but then I get
Failed to start container with docker id 7565654 with error: Error response from daemon: Container command '-auth-file' not found or does not exist.
thanks to #lang2
here is my deployment.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mailhog
spec:
replicas: 1
revisionHistoryLimit: 1
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: mailhog
spec:
volumes:
- name: secrets-volume
secret:
secretName: mailhog-login
containers:
- name: mailhog
image: us.gcr.io/com/mailhog:1.0.0
resources:
limits:
cpu: 70m
memory: 30Mi
requests:
cpu: 50m
memory: 20Mi
volumeMounts:
- name: secrets-volume
mountPath: /data/mailhog
readOnly: true
ports:
- containerPort: 8025
- containerPort: 1025
args:
- "-auth-file=/data/mailhog/auth.file"
In kubernetes, command is equivalent of ENTRYPOINT. In your case, args should be used.
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#container-v1-core
You are on the right track. It's just that you also need to include the name of the binary in the command array as the first element. You can find that out by looking​ in the respective Dockerfile (CMD and/or ENTRYPOINT).
In this case:
command: ["Mailhog", "-auth-file", "/data/mailhog/auth.file"]
I needed similar task (my aim was passing the application profile to app) and what I did is the following:
Setting an environment variable in Deployment section of the kubernetes yml file.
env:
- name: PROFILE
value: "dev"
Using this environment variable in dockerfile as command line argument.
CMD java -jar -Dspring.profiles.active=${PROFILE} /opt/app/xyz-service-*.jar