I have a Jenkins build process that deploys our code to several Kubernetes environments. Currently, it works fantastic with known/core branches (develop, release, master) and such, BUT we would like to make all of our feature branches and bug fix branches as well. In order to do this, we currently have it deploying an ingress with branch subdomains, like "{branchName}.domain.tld". The problem here is that there is a proliferation of ingresses and such for each branch.
I would LIKE to do something like "branch.domain.tld/{branchName}" and have the ingress dynamically route based on the "branchName" path. Unfortunately I only seem to get a single ingress per subdomain/host assignment (branch.domain.tld/bugfix_nasty doesn't route because branch.domain.tld/feat_cool already is defined).
What I would like to see is a single ingress that looks something like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dynamic_ingress
annotations:
{...stuff here}
spec:
rules:
- host: branch.domain.tld
http:
paths:
- path: /{branchName}/*
backend:
serviceName: {branchName}
servicePort: 80
Is there anyway to get to such a zenlike ingress?
All the stuff I see about this type of thing is advice on using regex for the path to capture and route to a known, defined service. I want this to route to variable service names.
As far as I know Ingress objects don't allow what you're trying to do. But there could be a workaround if you can modify the Ingress object every time you deploy a new branch.
Whenever a new branch is deployed update your "dynamic_ingress" to include an extra path for the new branch. This can be achieved using a small utility that can read and write YAMLs which can then be applied to update the existing Ingress OR you can use kubectl patch to update the existing Ingress at deploy time.
This way the Ingress definition has static paths but neither do you have to set the paths manually nor do you need multiple Ingress objects.
So you would have something like this.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dynamic_ingress
annotations:
{...stuff here}
spec:
rules:
- host: branch.domain.tld
http:
paths:
- path: /bug_fix_nasty/
pathType: Prefix
backend:
serviceName: bug_fix_nasty
servicePort: 80
- path: /feat_cool/
pathType: Prefix
backend:
serviceName: feat_cool
servicePort: 80
Check out all the examples in the Ingress doc and see which suits you best.
Related
I am trying to automate the hosts in Ingress Controller and I'm facing the problem of generating many hosts into one file. What I mean is, I have this ingress.yaml:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-host
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/default-backend: a
spec:
rules:
- host: a.example.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: a
port:
number: 80
- host: b.example.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: b
port:
number: 80
...
- host: x.example.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: x
port:
number: 80
...
in this example I have multiple instances: a, b, all the way to x and I anticipate a lot more. Right now I am programmatically regenerating the whole ingress.yaml to add/remove certain hosts. This is prone to errors and hard to maintain, as I must constantly be aware about ingress.yaml to be broken for one reason or another.
What would really help me is to put every host into a separate file and (maybe) just tell ingress.yaml to scan the whole directory where those files are to be stored. This way, I can just add/remove a single file and reload Ingress
Is there an option for that? I found somewhere that IngressSpec could be somehow defined, but I do not see any usefull link with a valid example. Maybe someone found a solution to that already and can point me to the right direction?
Hostname wildcards
Hosts can be precise matches (for example “foo.bar.com”) or a wildcard (for example “*.foo.com”).
Or use template system, such as helm or Kustomize
As #jordanm suggested in the comment, I went with multiple Ingress objects on one IngressController, being sure I get rid of nginx.ingress.kubernetes.io/default-backend annotation:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-x-host
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: x.example.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: x
port:
number: 80
I generate a unique file for each of hosts, replacing x with my unique name.
I also have to make sure that metadata.name is unique. If metadata.name is the same for every object, then it just gets replaced as I apply the new configuration. This works perfectly.
I'm following these instructions https://learn.microsoft.com/en-us/azure/aks/ingress-basic using Docker for Desktop/Kubernetes on Windows 10 again. The tutorial works great and now I want to enhance it by adding my C#/ASP.NETCore3 MVC WebApp called KubernetesHelloWorldASPNET (available on docker hub here). This is basically the template created by Visual Studio 2019 for a C# ASP.NET/MVC Core WebApp.
Here is the hello-world-ingress.yaml:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress
namespace: ingress-basic
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- backend:
serviceName: aks-helloworld-one
servicePort: 80
path: /hello-world-one(/|$)(.*)
- backend:
serviceName: aks-helloworld-two
servicePort: 80
path: /hello-world-two(/|$)(.*)
- backend:
serviceName: kuberneteshelloaspnet
servicePort: 80
path: /kuberneteshelloaspnet(/|$)(.*)
- backend:
serviceName: todolistclient
servicePort: 80
path: /todolistclient(/|$)(.*)
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress-static
namespace: ingress-basic
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /static/$2
spec:
rules:
- http:
paths:
- backend:
serviceName: aks-helloworld-one
servicePort: 80
path: /static(/|$)(.*)
This is working with the above ingress tutorial in the sense that I can see the web page for KubernetesHelloWorldASPNET from the browser (http://localhost/kuberneteshelloaspnet) via ingress. However, it is not formatted properly. I believe the problem is that ingress is not allowing the browser to get the CSS and other static files generated by Visual Studio.
How can I make my little HelloWorld program display properly with the CSS?
The web page looks fine when run directly with Visual Studio (no docker, no kubernetes, no ingress).
Thu Sep 09 2020 Update
As per Matt's comments, I used the developer tools in the browser and confirmed that I'm getting 404 errors for for bootstrap.min.css, boostrap.bundle.min.js, site.js, jquery.min.js.
I found a similar post here. However, it looks like the solution precludes having ingress accommodate multiple web sites each having their own CSS and JS directories. They are creating rules for paths like "/css" and "/js" and that makes me think that all the applications must share the same css and js directories?
I'm following the tutorial and creating a path that has wild cards in it. So I have
path: /kuberneteshelloaspnet(/|$)(.*)
This should accommodate (in my example using docker for desktop kubernetes) http://localhost/kuberneteshelloaspnet/lib/jquery/dist/jquery.min.js -- correct? Well it does not seem to be working!
Now that developer tools network display in Chrome seems to indicate that the js and css files have only a file name and no path. But when click on "Inspect" I see the the HTML and the HTML is referencing child directories like "lib" and "css" and this is consistent with the Visual Studio generated project directory.
So apparently my ingress rules are allowing access to the home MVC controller at http://localhost:80/kuberneteshelloaspnet but is not allowing access to http://localhost:80/kuberneteshelloaspnet/lib/jquery/dist/jquery.min.js (for example).
How do I modify my ingress rules to accommodate these child directories of static files?
Sat Sep 12 2020 Update: Progess!
This version of hello-world-ingress.yaml works for a single web-app:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress
namespace: ingress-basic
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- backend:
serviceName: kuberneteshelloaspnet
servicePort: 80
path: /
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress-static
namespace: ingress-basic
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- backend:
serviceName: kuberneteshelloaspnet
servicePort: 80
path: /lib(/|$)(.*)
It was just a guess and I don't know why it works. The web page seems to be working because it is formatted properly and I don't see any more 404 errors in the chrome developer tools panel.
Please help me understand the contents of this hello-world-ingress.yaml file. I was expecting to see kind: service and later kind: deployment and instead I just see kind: Ingress twice. Is this file defining a service or a deployment or both
Is it possible to enhance my solution so I host multiple web sites where each has a separate path and each has has separate sub-paths like /js and /css and /lib so each web site can have its own copy of (for example) jquery.min.js because they need different versions of jquery?
Among other things, I deleted the rewrite rules. When would I want to use rewrite rules? Could rewrite rules help me accommodate multiple web sites?
Fri Sep 18 2020: Progress?
I may have found the solution of using the function UsePathBase in this blog. The symptoms described in this blog match mine. However, as I describe here, I cannot make this work! Calling the function UseBasePath seems to have no affect.
Thanks
Siegfried
I had been using the Microsoft Ingress Example that demonstrates the rewrite rule.
The problem with using the rewrite rule is that it does not accommodate multiple web-apps that have different copies of static files like jquery.min.js.
The solution for C#/VB.NET ASP.NET Core programmers is to use UseBasePath.
I wish the above tutorial/example would explain these different options and their perils and merits. Obviously UseBasePath won't work with java or python webapps (although there might be counterparts).
I did not realize that since the rewrite rule executes first to remove the root segment of the path, then there is nothing for the UseBasePath to remove and it seems to have no affect.
The solution is to remove the rewrite rule after having added a call to UseBasePath.
In chrome check what path is missing. Hit the URL in chrome developer mode(press F12) then you will be to find. Add that path & this will be resolved.
I faced the same issue.
I'm wanting to create a service that can do some kind of dynamic proxying back to Kubernetes Pods. Basically I'll have hundreds of K8s Pods that are running the same application that map to a random port on the host (like 10456). However, each Pod is unique and I want traffic directed at a specific pod based on hostname. So when a request comes in for abc123.app.com, I'll have a proxy layer that does a lookup in a database to find what host and port that domain is running on (like 10.0.0.5:10456), then forward the request there. Is there a service that supports this? I've worked with Nginx a lot before, but I'm not clear if it could support this lookup functionality.
Has anyone built something like this before? what's the best way to build a proxy layer that can do lookups like that? How would I update the database when a pod moves from one host to another?
Thanks in advance!
EDIT:
I should have put this in there the first time, but the types of traffic going to these pods are RPC traffic and Peer to Peer traffic
You're describing something very similar to what kubernetes ingress definitions do for http traffic.
An ingress definition configures an ingress controller to point requests for a hostname at a service. The service selects endpoints (pods) via label selectors. When pods move, kubernetes updates the service automatically.
The work on your end just becomes pushing out config changes from your database via one of the API clients to kubernetes rather than directing a proxy. If your environment was extremely dynamic requiring reconfiguration all the time or you need to make dynamic decisions about where traffic should go, you might want to continue looking at a custom proxy or istio, openresty.
It sounds like you have unique deployments going to kubernetes already, so in addition to that include a service and ingress definition.
A simple example including a label on the a pod, a service that use the label. Then an ingress definition using the service.
apiVersion: v1
kind: Pod
metadata:
name: my-app
labels:
app: host-abc123
spec:
containers:
- name: host-abc123
image: me/my-app:1.2.1
ports:
- containerPort: 10456
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: host-abc123
spec:
rules:
- host: abc123.bar.com
http:
paths:
- backend:
serviceName: host-abc123
servicePort: 80
apiVersion: v1
kind: Service
metadata:
name: host-abc123
spec:
ports:
- protocol: TCP
port: 80
targetPort: 10456
The single ingress definition could include all hosts but I'm not sure how kubernetes and the ingress controllers would go replacing that regularly.
There are nginx based ingress controllers too. You end up with a nginx server config per ingress/host definition.
I have 3 services in my ingress, the first 2 use default namespace. The third service is prometheus-server service which has namespace ingress-nginx.
Now, I want to map my prometheus DNS to the service, but getting error because ingress can't find the prometheus service in default namespace.
How to deal with non-default namespace in ingress definition?
You would want to create a new Ingress in namespace ingress-nginx that would route your DNS to that service. For example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example
namespace: ingress-nginx
spec:
rules:
- host: your.domain.com
http:
paths:
- path: /
backend:
serviceName: prometheus-server
servicePort: 80
You will need to refer to your service in the other namespace with its full path, that is prometheus-server.ingress-nginx.svc.cluster.local.
You shouldn’t need a second Ingress to do this.
I have a small java webapp comprising of three microservices - api-service,book-service and db-service all of which are deployed on a kubernetes cluster locally using minikube.
I am planning to keep separate UIs for api-service and book-service , with the common static files served from a separate pod, probably an nginx:alpine image.
I was able to create a front end that serves the static files from nginx:alpine referring to this tutorial.
I would like to use ingress-nginx controller for routing requests to the two services.
The below diagram crudely shows where I am now.
I am confused as to where I should place the pod that serves the static content, and how to connect it to the ingress resource.I guess that keeping a front end pod before ingress defeats the purpose of ingress-nginx controller. What is the best practice to serve static files. Appreciate any help. Thanks.
Looks like you are confusing the fact that users, browsing online, will trigger standard requests to both "download" your static content, and use your 2 APIs (book and api). It's not the NGINX service serving the static content that is accessing your APIs, but the users browsers/applications, and they do that exactly the same for both static content and APIs (former has more/specific headers and data, like auth...).
On your diagram you'll want to put your new static-service at the exact same level as your book-service and api-service, ie behind the ingress. But your static-service won't have a link with the db-service like the other 2. Then just complete your ingress rules, with the static-service at the end as in this example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: your-global-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: foo.bar.com
http:
paths:
- path: /book-service
backend:
serviceName: book-service
servicePort: 80
- path: /api-service
backend:
serviceName: api-service
servicePort: 80
- path: /
backend:
serviceName: static-service
servicePort: 80
You'll have to adjust your services names and ports, and pick the paths you want your users to access your APIs, in the example above you'd have:
foo.bar.com/book-service for your book-service
foo.bar.com/api-service for the api-service
foo.bar.com/ ie everything else going to the static-service
You should have, 3 distinct pods I guess :
- static
- book-service
- api-service
The static pod will most likely not scale at the same speed than the two other.
Creating the services for each of your deployment. Then use the ingres to route the traffic on the proper endpoint.
Is it something like that you are trying to achieve?