HTTPS is not working with TLS enabled in GKE Ingress - jenkins

I have deployed jenkins in GKE using helm, now i am trying to configure DNS for jenkins. I am using cloudflare for DNS and also created TLS secret using my cloudflare certificates. The ingress that i have created works fine for http but HTTPS is not working. Following is my ingress that i used.
apiVersion: networking.k8s.io/v1beta1 # for versions before 1.14 use extensions/v1beta1i
kind: Ingress
metadata:
name: jenkins-ingress
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/use-forwarded-headers: "true"
nginx.ingress.kubernetes.io/use-proxy-protocol: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- jenkins url
secretName: secret-name
rules:
- host: jenkins url
http:
paths:
- path: /jenkins/*
backend:
serviceName: jenkins
servicePort: 80

The ingress that you have provided does not specify any service or service port for 443 to serve https requests and only has port 80 which is for http.
To enable HTTPS or gRPC over SSL when connecting to the endpoints of services, you need to add the nginx.org/ssl-services annotation to your Ingress resource definition. [1]
[1]https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/ssl-services

Related

502 Bad Gateway error thrown in AKS nginx ingress after redirect back from external service

We have a working Azure Kubernetes Service cluster with dotnet 6.0 web app. The pods are running on port 80 but the public url is running behind https cert which is being handled by an nginx ingress controller with cert secret. All this is working well.
We are adding some new functionality (integration from an external service). When signing into our app, with this new functionality, there's a brief redirect to the external service page. Once the user requests have completed, the external service redirects back to our site using a preconfigured redirect url to which is posted some data (custom header and query string). At this point, our site errors with 502 bad gateway.
When i review the logs on the nginx ingress controller pod, i can see some additional errors:
[error] 664#664: *17279861 upstream prematurely closed connection while reading response header from upstream, client: 10.240.0.5, server: www-dev.application.com, request: "GET /details/c2beac1c-b220-45fa-8fd5-08da12dced76/Success?id=ID-MJCX43A4FJ032551T752200W&token=EC-0G826951TM357702S&SenderID=4FHGRLJDXPXUU HTTP/2.0", upstream: "http://10.244.1.66:80/details/c2beac1c-b220-45fa-8fd5-08da12dced76/Success?id=ID-MJCX43A4FJ032551T752200W&token=EC-0G826951TM357702S&SenderID=4FHGRLJDXPXUU", host: "www-dev.application.com", referrer: "https://www.external.service.com/"
10.244.1.66 is the internal ip of one of the application pods.
at first i thought this was an error related to annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
because the referrer is an https:// site making the request. However adding that annotation makes the site unusuable (probably because the dotnet app pods are listening on port 80).
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: application-web
namespace: application
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
tls:
- hosts:
- www-dev.application.com
secretName: application-ingress-tls
rules:
- host: www-dev.application.com
http:
paths:
- path: /(.*)
pathType: Prefix
backend:
service:
name: applicationwebsvc
port:
number: 80
Here's the application ingress yaml.
Anyway, does anyone have any idea what the problem could be here? thanks
Ingress class moved from annotation to ingressClassName field, also you do not need to specify https before. Can you please try this:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: application-web
namespace: application
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
ingressClassName: nginx
tls:
- hosts:
- www-dev.application.com
secretName: application-ingress-tls
rules:
- host: www-dev.application.com
http:
paths:
- path: /(.*)
pathType: Prefix
backend:
service:
name: applicationwebsvc
port:
number: 80
Please also check to ingress documentation.
This ended up being a resource limits issue. There was one particular request that was causing memory usage to spike. This cause the container to be OOMKilled. This is what was leading to the 502 Bad gateway error message (because when it was killed, the container was no longer there to service the request).

Configuring WSO2 API Manager to Work With Traefik for HTTPS

I am trying to configure Traefik and WSO2 API Manager. Basically, I want to configure Traefik to handle https.
labels:
- "traefik.enable=true"
- "traefik.http.middlewares.service-am-https.redirectscheme.scheme=https"
- "traefik.http.routers.service-am-http.entrypoints=web"
- "traefik.http.routers.service-am-http.rule=Host(`xx.xx.xx`) && Path(`/apim/admin`)"
- "traefik.http.routers.service-am-http.middlewares=service-am-https#docker"
- "traefik.http.routers.service-am.tls=true"
- "traefik.http.routers.service-am.rule=Host(`xx.xx.xx`) && Path(`/apim/admin`)"
- "traefik.http.routers.service-am.entrypoints=web-secure"
- "traefik.http.services.service-am.loadbalancer.server.port=9443"
I also included this in the deployment.toml file for API Manager.
[catalina.valves.valve.properties]
className = "org.apache.catalina.valves.RemoteIpValve"
internalProxies = "*"
remoteIpHeader ="x-forwarded-for"
proxiesHeader="x-forwarded-by"
trustedProxies="*"
When I try to access the service, https://xx.xx.xx/apim/admin, I get this error:
Bad Request
This combination of host and port requires TLS.
Traefik is successfully handling the https part but when it comes to WSO2 API Manager, this issue comes up. Any ideas on how to resolve this?
I just had this problem and solved including
annotations:
ingress.kubernetes.io/protocol: https
in my Ingress.
The full configuration:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: wso2-ingress
namespace: <namespace>
annotations:
kubernetes.io/ingress.class: traefik
traefik.frontend.rule.type: PathPrefixStrip
ingress.kubernetes.io/protocol: https
spec:
rules:
- host: <hostname>
http:
paths:
- path: /
backend:
serviceName: <service-name>
servicePort: 9443

How to set HTTPS as default on GKE Ingress-gce

I currently have a working Frontend and Backend nodeports with an Ingress service setup with GKE's Google-managed certificates.
However, my issue is that by default when a user goes to samplesite.com, it uses http as default. This means that the user needs to specifically type in the browser https://samplesite.com in order to get the https version of my website.
How do I properly disable http on GKE ingress, or how do I redirect all my traffic to https? I understand that this can be forcefully done in my backend code as well but I want to separate concerns and handle this in my Kubernetes setup.
Here is my ingress.yaml file:
kind: Service
apiVersion: v1
metadata:
name: frontend-node-service
namespace: default
spec:
type: NodePort
selector:
app: frontend
ports:
- port: 5000
targetPort: 80
protocol: TCP
name: http
---
kind: Service
apiVersion: v1
metadata:
name: backend-node-service
namespace: default
spec:
type: NodePort
selector:
app: backend
ports:
- port: 8081
targetPort: 9229
protocol: TCP
name: http
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: samplesite-ingress-frontend
namespace: default
annotations:
kubernetes.io/ingress.global-static-ip-name: "samplesite-static-ip"
kubernetes.io/ingress.allow-http: "false"
networking.gke.io/managed-certificates: samplesite-ssl
spec:
backend:
serviceName: frontend-node-service
servicePort: 5000
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: samplesite-ingress-backend
namespace: default
annotations:
kubernetes.io/ingress.global-static-ip-name: "samplesite-backend-ip"
kubernetes.io/ingress.allow-http: "false"
networking.gke.io/managed-certificates: samplesite-api-ssl
spec:
backend:
serviceName: backend-node-service
servicePort: 8081
Currently GKE Ingress does not support out of the box HTTP->HTTPS redirect.
There is an ongoing Feature Request for it here:
Issuetracker.google.com: Issues: Redirect all HTTP traffic to HTTPS when using the HTTP(S) Load Balancer
There are some workarounds for it:
Use different Ingress controller like nginx-ingress.
Create a HTTP->HTTPS redirection in GCP Cloud Console.
How do I properly disable http on GKE ingress, or how do I redirect all my traffic to https?
To disable HTTP on GKE you can use following annotation:
kubernetes.io/ingress.allow-http: "false"
This annotation will:
Allow traffic only on port: 443 (HTTPS).
Deny traffic on port: 80 (HTTP) resulting in error code: 404.
Focusing on previously mentioned workarounds:
Use different Ingress controller like nginx-ingress
One of the ways to have the HTTP->HTTPS redirection is to use nginx-ingress. You can deploy it with official documentation:
Kubernetes.github.io: Ingress-nginx: Deploy: GCE-GKE
This Ingress controller will create a service of type LoadBalancer which will be the entry point for your traffic. Ingress objects will respond on LoadBalancer IP. You can download the manifest from installation part and modify it to support the static IP you have requested in GCP. More reference can be found here:
Stackoverflow.com: How to specify static IP address for Kubernetes load balancer?
You will need to provide your own certificates or use tools like cert-manager to have HTTPS traffic as the annotation: networking.gke.io/managed-certificates will not work with nginx-ingress.
I used this YAML definition and without any other annotations I was always redirected to the HTTPS:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
kubernetes.io/ingress.class: "nginx" # IMPORTANT
spec:
tls: # HTTPS PART
- secretName: ssl-certificate # SELF PROVIDED CERT NAME
rules:
- host:
http:
paths:
- path: /
backend:
serviceName: hello-service
servicePort: hello-port
Create a HTTP->HTTPS redirection in GCP Cloud Console.
There is also an option to manually create a redirection rule for your Ingress resource. You will need to follow official documentation:
Cloud.google.com: Load Balancing: Docs: HTTPS: Setting up HTTP -> HTTPS Redirect
Using the part of above documentation, you will need to create a HTTP LoadBalancer responding on the same IP as your Ingress resource (reserved static IP) redirecting traffic to HTTPS.
Disclaimer!
Your Ingress resource will need to have following annotation:
kubernetes.io/ingress.allow-http: "false"
Lack there of will result in forbidding you to create a redirection mentioned above.

Nginx Ingress Controller Returns 404 Kubernetes

I am trying to create an ingress controller that points to a service that I have exposed via NodePort.
Here is the yaml file for the ingress controller (taken from https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/):
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
- host: hello-world.info
http:
paths:
- path: /
backend:
serviceName: appName
servicePort: 80
I can connect directly to the node port and the frontend is displayed.
Please note that I am doing this because the frontend app is unable to connect to other deployments that I have created and I read that an ingress controller would be able to solve the issue. Will I still have to add an Nginx reverse proxy? If so how would I do that? I have tried adding this to the nginx config file but with no success.
location /middleware/ {
proxy_pass http://middleware/;
}
You must use a proper hostname to reach the route defined in the Ingress object. Either update your /etc/hosts file or use curl -H "hello-world.info" localhost type command. Alternatively, you can delete the host mapping and redirect all traffic to one default service.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: appName
servicePort: 80

Nginx Ingress Looking for Service in all Custom Namespaces

I have three namespaces dev, test and staging. test and staging have no pods in them. In dev I have nginx, ingress and a frontend service. For all requests to the nginx it's forwarded to the frontend service.
But the issue is nginx in dev trying to find frontend service in test and staging namespaces also. It's doing round robin between the 3 namespaces. So sometimes the page is loading and sometimes it's 503 error.
Here is the ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: frontend-ingress
namespace: dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: frontend
servicePort: 80
And here is the log of nginx:
I0327 07:54:50.867120 1 command.go:76] change in configuration detected. Reloading...
W0327 07:54:50.867339 1 controller.go:841] service test/frontend does not have any active endpoints
W0327 07:54:50.867370 1 controller.go:841] service staging/frontend does not have any active endpoints
W0327 07:54:50.868198 1 controller.go:777] upstream test-frontend-80 does not have any active endpoints. Using default backend
W0327 07:54:50.868219 1 controller.go:777] upstream staging-frontend-80 does not have any active endpoints. Using default backend
Specify --force-namespace-isolation=true argument when deploying nginx pod. And update image to quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.12.0
I'd like to elaborate on #Narayan-Prusty's answer.
I had to add --force-namespace-isolation=true and set image to quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.12.0 but also had to add --watch-namespace=$(POD_NAMESPACE).

Resources