How to expose two apps/services over unique ports with k3d? - docker

Question

*Cross-posted from k3d GitHub Discussion: https://github.com/rancher/k3d/discussions/690
I am attempting to expose two services over two ports. As an alternative, I'd also love to know how to expose them over the same port and use different routes. I've attempted a few articles and a lot of configurations. Let me know where I'm going wrong with the networking of k3d + k3s / kubernetes + traefik (+ klipper?)...
I posted an example:
https://github.com/ericis/k3d-networking
The goal:
Reach "app-1" on host over port 8080
Reach "app-2" on host over port 8091
Steps
*See: files in repo
Configure k3d cluster and expose app ports to load balancer
ports:
# map localhost to loadbalancer
- port: 8080:80
nodeFilters:
- loadbalancer
# map localhost to loadbalancer
- port: 8091:80
nodeFilters:
- loadbalancer
Deploy apps with "deployment.yaml" in Kubernetes and expose container ports
ports:
- containerPort: 80
Expose services within kubernetes. Here, I've tried two methods.
Using CLI
$ kubectl create service clusterip app-1 --tcp=8080:80
$ kubectl create service clusterip app-2 --tcp=8091:80
Using "service.yaml"
spec:
ports:
- protocol: TCP
# expose internally
port: 8080
# map to app
targetPort: 80
selector:
run: app-1
Expose the services outside of kubernetes using "ingress.yaml"
backend:
service:
name: app-1
port:
# expose from kubernetes
number: 8080

Answer 1

You either have to use an ingress, or have to open ports on each individual node (k3d runs on docker, so you have to expose the docker ports)
Without opening a port during the creation of the k3d cluster, a nodeport service will not expose your app
k3d cluster create mycluster -p 8080:30080#agent[0]
For example, this would open an "outside" port (on your localhost) 8080 and map it to 30080 on the node - then you can use a NodePort service to actually connect the traffic from that port to your app:
apiVersion: v1
kind: Service
metadata:
name: some-service
spec:
ports:
- protocol: TCP
port: 80
targetPort: some-port
nodePort: 30080
selector:
app: pgadmin
type: NodePort
You can also open ports on the server node like:
k3d cluster create mycluster -p 8080:30080#server[0]
Your apps can get scheduled to run on whatever node, and if you force a pod to appear on a specific node (lets say you open a certain port on agent[0] and set up your .yaml files to work with that certain port), for some reason the local-path rancher storage-class just breaks and will not create a persistent volume for your claim. You kinda have to get lucky & have your pod get scheduled where you need it to. (if you find a way to schedule pods on specific nodes without the storage provisioner breaking, let me know)
You also can map a whole range of ports, like:
k3d cluster create mycluster --servers 1 --agents 1 -p "30000-30100:30000-30100#server[0]"
but be careful with the amount of ports you open, if you open too much, k3d will crash.
Using a load balancer - it's similar, you just have to open one port & map to to the load balancer.
k3d cluster create my-cluster --port 8080:80#loadbalancer
You then have to use an ingress, (or the traffic won't reach)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hello
port:
number: 80
I also think that ingress will only route http & https traffic, https should be done on the port 443, supposedly you can map both port 80 and port 443, but I haven't been able to get that to work (I think that certificates need to be set up as well).