I have a Docker Enterprise k8 bare metal cluster running on Centos8, and following the official docs to install NGINX using manifest files from GIT: https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
The pod seems to be running:
kubectl -n nginx-ingress describe pod nginx-ingress-fzr2j
Name: nginx-ingress-fzr2j
Namespace: nginx-ingress
Priority: 0
Node: server.example.com/172.16.1.180
Start Time: Sun, 16 Aug 2020 16:48:49 -0400
Labels: app=nginx-ingress
controller-revision-hash=85879fb7bc
pod-template-generation=2
Annotations: kubernetes.io/psp: privileged
Status: Running
IP: 192.168.225.27
IPs:
IP: 192.168.225.27
But my issue is the IP address it has selected is a 192.168.225.27. This is a second network on this server. How do I tell nginx to use the 172.16.1.180 address that is has in the Node: part?
The Daemset config is :
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
#annotations:
#prometheus.io/scrape: "true"
#prometheus.io/port: "9113"
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:edge
imagePullPolicy: Always
name: nginx-ingress
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
- name: readiness-port
containerPort: 8081
#- name: prometheus
#containerPort: 9113
readinessProbe:
httpGet:
path: /nginx-ready
port: readiness-port
periodSeconds: 1
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
I can't see any configuration option for which IP address to bind to.
The thing you are likely looking for is hostNetwork: true, which:
Use the host's network namespace. If this option is set, the ports that will be used must be specified. Default to false
spec:
template:
spec:
hostNetwork: true
containers:
- image: nginx/nginx-ingress:edge
name: nginx-ingress
You would only then need to specify a bind address if it bothered you having the Ingress controller bound to all addresses on the host. If that's still a requirement, you can have the Node's IP injected via the valueFrom: mechanism:
...
containers:
- env:
- name: MY_NODE_IP
valueFrom:
fieldRef:
status.hostIP
Related
I have clean ubuntu 18.04 server where installed minikube, kubectl and docker.
And I have several items for it.
One deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongo-express-deployment
labels:
app: mongo-express
spec:
replicas: 1
selector:
matchLabels:
app: mongo-express
template:
metadata:
labels:
app: mongo-express
spec:
containers:
- name: mongo-express
image: mongo-express
ports:
- containerPort: 8081
env:
- name: ME_CONFIG_MONGODB_ADMINUSERNAME
valueFrom:
secretKeyRef:
name: mongo-db-secret
key: mongo-db-root-username
- name: ME_CONFIG_MONGODB_ADMINPASSWORD
valueFrom:
secretKeyRef:
name: mongo-db-secret
key: mongo-db-root-password
- name: ME_CONFIG_MONGODB_SERVER
valueFrom:
configMapKeyRef:
name: mongo-db-configmap
key: mongo-db-url
one internal service. because tried to connect through ingress
apiVersion: v1
kind: Service
metadata:
name: mongo-express-service
spec:
selector:
app: mongo-express
ports:
- protocol: TCP
port: 8081
targetPort: 8081
one ingress for it
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dashboard-ingress
spec:
rules:
- host: my-host.com
http:
paths:
- path: "/"
pathType: "Prefix"
backend:
service:
name: mongo-express-service
port:
number: 8081
And one external service because I tried to connect through them
apiVersion: v1
kind: Service
metadata:
name: mongo-express-external-service
spec:
selector:
app: mongo-express
type: LoadBalancer
ports:
- protocol: TCP
port: 8081
targetPort: 8081
nodePort: 30000
But each of these options does not work for me. I tried add update to host file and add
192.168.47.2 my-host.com
but it also didn't help me.
When I run curl my-host.com in server terminal I receive correct response, but I can't get it from my browser.
My domain set up to my server and when I use nginx only all work fine.
May be need to add something else or update my config?
I hope you can help me.
I work on an open source system that is comprised of a Postgres database and a tomcat server. I have docker images for each component. We currently use docker-compose to test the application.
I am attempting to model this application with kubernetes.
Here is my first attempt.
apiVersion: v1
kind: Pod
metadata:
name: dspace-pod
spec:
volumes:
- name: "pgdata-vol"
emptyDir: {}
- name: "assetstore"
emptyDir: {}
- name: my-local-config-map
configMap:
name: local-config-map
containers:
- image: dspace/dspace:dspace-6_x
name: dspace
ports:
- containerPort: 8080
name: http
protocol: TCP
volumeMounts:
- mountPath: "/dspace/assetstore"
name: "assetstore"
- mountPath: "/dspace/config/local.cfg"
name: "my-local-config-map"
subPath: local.cfg
#
- image: dspace/dspace-postgres-pgcrypto
name: dspacedb
ports:
- containerPort: 5432
name: http
protocol: TCP
volumeMounts:
- mountPath: "/pgdata"
name: "pgdata-vol"
env:
- name: PGDATA
value: /pgdata
I have a configMap that is setting the hostname to the name of the pod.
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: 2016-02-18T19:14:38Z
name: local-config-map
namespace: default
data:
local.cfg: |-
dspace.dir = /dspace
db.url = jdbc:postgresql://dspace-pod:5432/dspace
dspace.hostname = dspace-pod
dspace.baseUrl = http://dspace-pod:8080
solr.server=http://dspace-pod:8080/solr
This application has a number of tasks that are run from the command line.
I have created a 3rd Docker image that contains the jars that are needed on the command line.
I am interested in modeling these command line tasks as Jobs in Kubernetes. Assuming that is a appropriate way to handle these tasks, how do I specify that a job should run within a Pod that is already running?
Here is my first attempt at defining a job.
apiVersion: batch/v1
kind: Job
#https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
metadata:
name: dspace-create-admin
spec:
template:
spec:
volumes:
- name: "assetstore"
emptyDir: {}
- name: my-local-config-map
configMap:
name: local-config-map
containers:
- name: dspace-cli
image: dspace/dspace-cli:dspace-6_x
command: [
"/dspace/bin/dspace",
"create-administrator",
"-e", "test#test.edu",
"-f", "test",
"-l", "admin",
"-p", "admin",
"-c", "en"
]
volumeMounts:
- mountPath: "/dspace/assetstore"
name: "assetstore"
- mountPath: "/dspace/config/local.cfg"
name: "my-local-config-map"
subPath: local.cfg
restartPolicy: Never
The following configuration has allowed me to start my services (tomcat and postgres) as I hoped.
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: 2016-02-18T19:14:38Z
name: local-config-map
namespace: default
data:
# example of a simple property defined using --from-literal
#example.property.1: hello
#example.property.2: world
# example of a complex property defined using --from-file
local.cfg: |-
dspace.dir = /dspace
db.url = jdbc:postgresql://dspacedb-service:5432/dspace
dspace.hostname = dspace-service
dspace.baseUrl = http://dspace-service:8080
solr.server=http://dspace-service:8080/solr
---
apiVersion: v1
kind: Service
metadata:
name: dspacedb-service
labels:
app: dspacedb-app
spec:
type: NodePort
selector:
app: dspacedb-app
ports:
- protocol: TCP
port: 5432
# targetPort: 5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dspacedb-deploy
labels:
app: dspacedb-app
spec:
selector:
matchLabels:
app: dspacedb-app
template:
metadata:
labels:
app: dspacedb-app
spec:
volumes:
- name: "pgdata-vol"
emptyDir: {}
containers:
- image: dspace/dspace-postgres-pgcrypto
name: dspacedb
ports:
- containerPort: 5432
name: http
protocol: TCP
volumeMounts:
- mountPath: "/pgdata"
name: "pgdata-vol"
env:
- name: PGDATA
value: /pgdata
---
apiVersion: v1
kind: Service
metadata:
name: dspace-service
labels:
app: dspace-app
spec:
type: NodePort
selector:
app: dspace-app
ports:
- protocol: TCP
port: 8080
targetPort: 8080
name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dspace-deploy
labels:
app: dspace-app
spec:
selector:
matchLabels:
app: dspace-app
template:
metadata:
labels:
app: dspace-app
spec:
volumes:
- name: "assetstore"
emptyDir: {}
- name: my-local-config-map
configMap:
name: local-config-map
containers:
- image: dspace/dspace:dspace-6_x-jdk8-test
name: dspace
ports:
- containerPort: 8080
name: http
protocol: TCP
volumeMounts:
- mountPath: "/dspace/assetstore"
name: "assetstore"
- mountPath: "/dspace/config/local.cfg"
name: "my-local-config-map"
subPath: local.cfg
After applying the configuration above, I have the following results.
$ kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
dspace-service NodePort 10.104.224.245 <none> 8080:32459/TCP 3s app=dspace-app
dspacedb-service NodePort 10.96.212.9 <none> 5432:30947/TCP 3s app=dspacedb-app
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 22h <none>
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
dspace-deploy-c59b77bb8-mr47k 1/1 Running 0 10m
dspacedb-deploy-58dd85f5b9-6v2lf 1/1 Running 0 10
I was pleased to see that the service name can be used for port forwarding.
$ kubectl port-forward service/dspace-service 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
I am also able to run the following job using the defined service names in the configMap.
apiVersion: batch/v1
kind: Job
metadata:
name: dspace-create-admin
spec:
template:
spec:
volumes:
- name: "assetstore"
emptyDir: {}
- name: my-local-config-map
configMap:
name: local-config-map
containers:
- name: dspace-cli
image: dspace/dspace-cli:dspace-6_x
command: [
"/dspace/bin/dspace",
"create-administrator",
"-e", "test#test.edu",
"-f", "test",
"-l", "admin",
"-p", "admin",
"-c", "en"
]
volumeMounts:
- mountPath: "/dspace/assetstore"
name: "assetstore"
- mountPath: "/dspace/config/local.cfg"
name: "my-local-config-map"
subPath: local.cfg
restartPolicy: Never
Results
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
dspace-create-admin-kl6wd 0/1 Completed 0 5m
dspace-deploy-c59b77bb8-mr47k 1/1 Running 0 10m
dspacedb-deploy-58dd85f5b9-6v2lf 1/1 Running 0 10m
I still have some work to do persisting the volumes.
1) I have SSL certs generated on AWS
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:...fa5298fc
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
labels:
k8s-addon: ingress-nginx.addons.k8s.io
name: ingress-nginx-lb-svc
# namespace: ingress-nginx
spec:
externalTrafficPolicy: Cluster
ports:
- name: https
port: 443
protocol: TCP
targetPort: http
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app: nginx-ingress-control-pod
type: LoadBalancer
2) then I have nginx controller pod
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-ingress-control-pod
labels:
app: nginx-ingress-control-pod
spec:
replicas: 1
selector:
app: nginx-ingress-control-pod
template:
metadata:
labels:
app: nginx-ingress-control-pod
spec:
containers:
- image: nginxdemos/nginx-ingress:1.0.0
imagePullPolicy: Always
name: nginx-ingress-control-pod
ports:
- name: http
containerPort: 80
hostPort: 80
#- name: https
# containerPort: 443
# hostPort: 443
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# Uncomment the lines below to enable extensive logging and/or customization of
# NGINX configuration with configmaps
args:
#- -v=3
#- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
#- -default-server-tls-secret=$(POD_NAMESPACE)/web-secret
3) lastly I am using helm to deploy grafana and prometheus (this setup works when accessing via NodePort)
I just can not figure out setup with ELB and ingress.
Btw ingress is a part of grafana deployment
which is correctly created
3)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
creationTimestamp: 2018-04-06T09:28:10Z
generation: 1
labels:
app: graf-helmf-default-ns-grafana
chart: grafana-0.8.5
component: grafana
heritage: Tiller
release: graf-helmf-default-ns
name: graf-helmf-default-ns-grafana
namespace: default
resourceVersion: "995865"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/graf-helmf-default-ns-grafana
uid: d2991870-397c-11e8-9d...5a37f5a
spec:
rules:
- host: grafana.my.valid.domain.com
http:
paths:
- backend:
serviceName: graf-helmf-default-ns-grafana
servicePort: 80
status:
loadBalancer: {}
I have Docker, Kubernetes(1.7) and Nginx all running on my RHEL7 server with my own services being inside a docker container and being picked up by Kubernetes. I know Kubernetes is working right with docker because I can call a get request of the Kubernete pod using its own IP:PORT addresses and it works. I set up Nginx with a default backend and have all of this working. I know this by calling get pods and get svc commands and everything is running as it should. When I create ingress, I know Nginx is picking it up because when I use the command kubectl describe pods {NGNIX-CONTROLLER} I see it updates its ingress and even logs what I named it. Now I get the IP address of Kubernetes master using kubectl clusterinfo and I use this ip address to attempt to call my services, something along the lines of http://KUBEIPADDRESS/PATH/TO/MY/SERVICE, with no port number but it doesn't work. I have no idea what is going on. Can someone help me why Ingress and/or Nnginx isn't routing properly to my services? I'll give my ingress and nginx file down below.
(Note, for the nginx yaml file, the deployment of the nginx controller is all the way in the bottom.)
Ingress yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway-ingress
annotations:
kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/rewrite-target: /
spec:
backend:
serviceName: default-http-backend
servicePort: 80
rules:
- host: testhost
http:
paths:
- path: /customer
backend:
serviceName: customer
servicePort: 9001
nginx controller yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: ingress
rules:
- apiGroups:
- ""
- "extensions"
resources:
- configmaps
- secrets
- services
- endpoints
- ingresses
- nodes
- pods
verbs:
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- apiGroups:
- ""
resources:
- events
- services
verbs:
- create
- list
- update
- get
- apiGroups:
- "extensions"
resources:
- ingresses/status
- ingresses
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: ingress-ns
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- apiGroups:
- ""
resources:
- services
verbs:
- get
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- create
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: ingress-ns-binding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-ns
subjects:
- kind: ServiceAccount
name: ingress
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: ingress-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress
subjects:
- kind: ServiceAccount
name: ingress
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: default-http-backend
labels:
k8s-app: default-http-backend
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissable as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: gcr.io/google_containers/defaultbackend:1.0
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: kube-system
labels:
k8s-app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
k8s-app: default-http-backend
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
labels:
k8s-app: nginx-ingress-controller
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: nginx-ingress-controller
spec:
# hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
# however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
# that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
# like with kubeadm
hostNetwork: true
terminationGracePeriodSeconds: 60
serviceAccountName: ingress
containers:
- image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.3
name: nginx-ingress-controller
readinessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 1
ports:
- containerPort: 80
hostPort: 80
- containerPort: 443
hostPort: 443
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
Also when I do kubectl describe ing I get
Name: gateway-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
testhost
/customer customer:9001 ({IP}:9001,{IP}:9001)
Annotations:
rewrite-target: /
Events: <none>
Here are my deployment and service of the customer in case anyone needs that
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: customer
labels:
run: customer
spec:
replicas: 2
template:
metadata:
labels:
run: customer
spec:
containers:
- name: customer
image: customer
imagePullPolicy: Always
ports:
- containerPort: 9001
protocol: TCP
---
kind: Service
apiVersion: v1
metadata:
name: customer
spec:
selector:
run: customer
type: NodePort
ports:
- name: port1
protocol: TCP
port: 9001
targetPort: 9001
There are some issues with your setup as far as I can see:
KUBEIPADDRESS in the URL you call: an IP address won't work because you configured your Ingress to listen on testhost. So you need to call http://testhost/customer, and configure your network to resolve testhost to the correct IP address
but what is the correct IP address? You are trying to use k8s master on port 80. That won't work without further configuration. For that you need to use a NodePort service for the Ingress Controller, which exposes it on port 80 (and probably 433). In order to use that low ports, you need to allow it with an option of kube-apiserver, see --service-node-port-range on https://kubernetes.io/docs/admin/kube-apiserver/. Once that works, you can use any IP address of any node of your k8s cluster for testhost. Note: be sure that no other application uses these ports on any node!
Ok, following the examples and documentation on the Kubernetes website along with extensive research on Google, I still cannot get DNS resolution between the containers within my Pod.
I have a Service and a PetSet with 2 containers defined. When I deploy the PetSet and Service, they start and run successfully, but if I attempt to ping the host of one of my containers from the other by hostname or by the full domain name I get destination unreachable. I can ping by IP address though.
Here is my Kubernetes configuration file:
apiVersion: v1
kind: Service
metadata:
name: ml-service
labels:
app: marklogic
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
#restartPolicy: OnFailure
clusterIP: None
selector:
app: marklogic
ports:
- protocol: TCP
port: 7997
#nodePort: 31997
name: ml7997
- protocol: TCP
port: 8000
#nodePort: 32000
name: ml8000
# ... More ports defined
#type: NodePort
---
apiVersion: apps/v1alpha1
kind: PetSet
metadata:
name: marklogic
spec:
serviceName: "ml-service"
replicas: 2
template:
metadata:
labels:
app: marklogic
annotations:
pod.alpha.kubernetes.io/initialized: "true"
spec:
terminationGracePeriodSeconds: 30
containers:
- name: 'marklogic'
image: "{local docker registry ip}:5000/dcgs-sof/ml8-docker-final:v1"
imagePullPolicy: Always
command: ["/opt/entry-point.sh", "-l", "/opt/mlconfig.sh"]
ports:
- containerPort: 7997
name: ml7997
- containerPort: 8000
name: ml8000
- containerPort: 8001
name: ml8001
- containerPort: 8002
name: ml8002
- containerPort: 8040
name: ml8040
- containerPort: 8041
name: ml8041
- containerPort: 8042
name: ml8042
- containerPort: 8050
name: ml8050
- containerPort: 8051
name: ml8051
- containerPort: 8060
name: ml8060
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
lifecycle:
preStop:
exec:
command: ["/etc/init.d/MarkLogic stop"]
volumeMounts:
- name: ml-data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: ml-data
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
I commented out the type: NodePort definition as I thought that might be the culprit, but still no success.
Additionally, if I run docker#minikube:/$ docker exec b4d21c4bc065 /bin/bash -c 'nslookup marklogic-1.marklogic.default.svc.cluster.local' it cannot resolve the name.
What am I missing???
You are resolving the wrong domain name.
See http://kubernetes.io/docs/user-guide/petset/#network-identity
You should try to resolve:
marklogic-0.ml-service.default.svc.cluster.local
If everything is within the default namespace, the DNS name is:
<pod_name>.<svc_name>.default.svc.cluster.local