I'm wanting to create a service that can do some kind of dynamic proxying back to Kubernetes Pods. Basically I'll have hundreds of K8s Pods that are running the same application that map to a random port on the host (like 10456). However, each Pod is unique and I want traffic directed at a specific pod based on hostname. So when a request comes in for abc123.app.com, I'll have a proxy layer that does a lookup in a database to find what host and port that domain is running on (like 10.0.0.5:10456), then forward the request there. Is there a service that supports this? I've worked with Nginx a lot before, but I'm not clear if it could support this lookup functionality.
Has anyone built something like this before? what's the best way to build a proxy layer that can do lookups like that? How would I update the database when a pod moves from one host to another?
Thanks in advance!
EDIT:
I should have put this in there the first time, but the types of traffic going to these pods are RPC traffic and Peer to Peer traffic
You're describing something very similar to what kubernetes ingress definitions do for http traffic.
An ingress definition configures an ingress controller to point requests for a hostname at a service. The service selects endpoints (pods) via label selectors. When pods move, kubernetes updates the service automatically.
The work on your end just becomes pushing out config changes from your database via one of the API clients to kubernetes rather than directing a proxy. If your environment was extremely dynamic requiring reconfiguration all the time or you need to make dynamic decisions about where traffic should go, you might want to continue looking at a custom proxy or istio, openresty.
It sounds like you have unique deployments going to kubernetes already, so in addition to that include a service and ingress definition.
A simple example including a label on the a pod, a service that use the label. Then an ingress definition using the service.
apiVersion: v1
kind: Pod
metadata:
name: my-app
labels:
app: host-abc123
spec:
containers:
- name: host-abc123
image: me/my-app:1.2.1
ports:
- containerPort: 10456
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: host-abc123
spec:
rules:
- host: abc123.bar.com
http:
paths:
- backend:
serviceName: host-abc123
servicePort: 80
apiVersion: v1
kind: Service
metadata:
name: host-abc123
spec:
ports:
- protocol: TCP
port: 80
targetPort: 10456
The single ingress definition could include all hosts but I'm not sure how kubernetes and the ingress controllers would go replacing that regularly.
There are nginx based ingress controllers too. You end up with a nginx server config per ingress/host definition.
Related
I have a Redis pod, and I expect connection requests to this pod from different clusters and applications not running in the cloud.
Since Redis does not work with the http protocol, accessing as the route I have done below does not work with this connection string "route-redis.local:6379".
route.yml
apiVersion: v1
kind: Route
metadata:
name: redis
spec:
host: route-redis.local
to:
kind: Service
name: redis
service.yml
apiVersion: v1
kind: Service
metadata:
name: redis
spec:
ports:
- port: 6379
targetPort: 6379
selector:
name: redis
You may have encountered this situation. In short, is there any way to access to the redis pod via route? If not, how do you solve this problem?
You already discovered that Redis does not work via the HTTP protocol, which is correct as far as I know. Routes work by inspecting the HTTP Host header for each request, which will not work for Redis. This means that you will not be able to use Routes for non-HTTP workload.
Typically, such non-HTTP services are exposed via a Service and NodePorts. This means that each Worker Node that is part of your cluster will open this port and will forward the traffic to your application.
You can find more information in the Kubernetes documentation:
NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting :.
You can define a NodePort like so (this example is for MySQL, which is also non-HTTP workload):
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
name: mysql
spec:
type: NodePort
ports:
- port: 3306
nodePort: 30036
name: http
selector:
name: mysql
Of course, your administrator may limit the access to these ports, so it may or may not be possible to use these types of services on your OpenShift cluster.
You can expose the tcp via ingress atleast nginx one
https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
We access the container through cluster IP and even we deploy web application containers can be accessed.The issue with how can we access container from outside the host.
Tried with giving external IP to containers.
You can create a service and bind it to a node port, from outside your cluster if you try to access that service using node_ip:port.
apiVersion: v1
kind: Service
metadata:
name: api-server
spec:
ports:
- port: 80
name: http
targetPort: api-http
nodePort: 30004
- port: 443
name: https
targetPort: api-http
type: LoadBalancer
selector:
run: api-server
if you do kubectl get service you can get the external ip.
The best approach would be to expose your pods with ClusterIP type services, and then use an Ingress resource along with Ingress Controller to expose HTTP and/or HTTPS routes so you can access your app outside of the cluster.
For testing purposes it's ok to use NodePort or LoadBalancer type services. Whether you are running on your own infrastructure or using a managed solution, you can use NodePort, while using LoadBalancer requires cloud provider's load balancer.
Source: Official docs
I'm trying to expose a Kubernetes pod on a single node bare metal cluster without a domain.
In my understanding I've the these options:
Expose using NodePort
Expose using an Ingress controller
Expose using ClusterIP and manually set an external IP
As I mentioned already, I only have a single node cluster. This means that the master is master and node at the same time directlly running on a fedora host system.
The simplest solution is to use a NodePort. But the limitation here is (if I'm right), that the service port will be automatically selected from a given port range.
The next better solution is to use an ingress controller. But for this I need a public domain which I haven't. So the ingress controller also doesn't fit to me.
What for other options do I have? I just want to expose my service directly on port 9090.
Why not Option 3 ？ you can setup externalIPs to your node ip.
apiVersion: v1
kind: Service
...
spec:
externalIPs:
- your node ip
Also with NodePort, the service port can be specified.
You can set a custom port range for NodePort by adding this option to your apiserver settings (/etc/kubernetes/manifests/kube-apiserver.yaml):
--service-node-port-range portRange
Default: 30000-32767
A port range to reserve for services with NodePort visibility. Example:
'30000-32767'.
Inclusive at both ends of the range.
This is the part from Kubernetes documentation related to Services:
If you want a specific port number, you can specify a value in the
nodePort field, and the system will allocate you that port or else the
API transaction will fail (i.e. you need to take care about possible
port collisions yourself). The value you specify must be in the
configured range for node ports.
Example for this answer was taken from the article Hosting Your Own Kubernetes NodePort Load Balancer:
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
name: nginx
spec:
type: NodePort
ports:
- port: 80
nodePort: 30080
name: http
- port: 443
nodePort: 30443
name: https
selector:
name: nginx
I have received a public ip address for my kubernetes service which i can configure as a loadbalancer ip in my NGINX ingress. This public ip address can be accessed from public internet.
Is there a way or some configuration through which i can make these services accessible only from my client network in kubernetes?
With Kubernetes Nginx Ingress it is as simple as setting an annotation on your ingress object like :
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/whitelist-source-range: '8.8.8.8/32'
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/annotations.md#whitelist-source-range
You can as suggested make use of the VPN and create an internal LoadBalancer or you can check the Network Policies that I consider that Kubernetes standard way to implement your solution.
By default, if no policies exist in a namespace, then all ingress and egress traffic is allowed to and from pods in that namespace. The following examples let you change the default behavior in that namespace.
You will need to create a NetworkPolicy Resource, in the spec you will have to describe the behaviour making use of the available fields, I recommend you to check the official documentation to retrieve more info regarding the structure.
PolicyTypes:
...
ingress: Each NetworkPolicy may include a list of whitelist ingress rules. Each rule allows traffic which matches both the from and ports sections. The example policy contains a single rule, which matches traffic on a single port, from one of three sources, the first specified via an ipBlock, the second via a namespaceSelector and the third via a podSelector.
...
Keep in mind that in order to implement them you need to use a
networking solution which supports NetworkPolicy, if you just
create the resource without a controller to implement it will have no
effect.
Example of policy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: test-network-policy
namespace: default
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- ipBlock:
cidr: 172.17.0.0/16
except:
- 172.17.1.0/24
- namespaceSelector:
matchLabels:
project: myproject
- podSelector:
matchLabels:
role: frontend
ports:
- protocol: TCP
port: 6379
egress:
- to:
- ipBlock:
cidr: 10.0.0.0/24
ports:
- protocol: TCP
port: 5978
Using Network Policy is nice. But, a simpler approach would be use set ExternalIP of the nginx ingress controller to the IP address in the client network. This exposes the services only on the client network.
Below is the sample configuration for helm:
helm install --name my-ingress stable/nginx-ingress \
--set controller.service.externalIPs=<IP address in client network>
I currently have a service that looks like this:
apiVersion: v1
kind: Service
metadata:
name: httpd
spec:
ports:
- port: 80
targetPort: 80
name: http
protocol: TCP
- port: 443
targetPort: 443
name: https
protocol: TCP
selector:
app: httpd
externalIPs:
- 10.128.0.2 # VM's internal IP
I can receive traffic fine from the external IP bound to the VM, but all of the requests are received by the HTTP with the source IP 10.104.0.1, which is most definitely an internal IP – even when I connect to the VM's external IP from outside the cluster.
How can I get the real source IP for the request without having to set up a load balancer or ingress?
This is not simple to achieve -- because of the way kube-proxy works, your traffic can get forwarded between nodes before it reaches the pod that's backing your Service.
There are some beta annotations that you can use to get around this, specifically service.beta.kubernetes.io/external-traffic: OnlyLocal.
More info in the docs, here: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer
But this does not meet your additional requirement of not requiring a LoadBalancer. Can you expand upon why you don't want to involve a LoadBalancer?
If you only have exactly one pod, you can use hostNetwork: true to achieve this:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: caddy
spec:
replicas: 1
template:
metadata:
labels:
app: caddy
spec:
hostNetwork: true # <---------
containers:
- name: caddy
image: your_image
env:
- name: STATIC_BACKEND # example env in my custom image
value: $(STATIC_SERVICE_HOST):80
Note that by doing this your pod will inherit the host's DNS resolver and not Kubernetes'. That means you can no longer resolve cluster services by DNS name. For example, in the example above you cannot access the static service at http://static. You still can access services by their cluster IP, which are injected by environment variables.