I set up a Cloud Run instance with gRPC and HTTP2. It works well. But, I'd like to open a new port externally and route that traffic to gRPC over HTTPS.
This is the current YAML for the container:
- image: asia.gcr.io/assets-320007/server5c2b87a8444cb42e566e130a907015df7dd841b4
ports:
- name: h2c
containerPort: 5000
resources:
limits:
cpu: 1000m
memory: 512Mi
I cannot add new ports because if I do, I get:
spec.template.spec.containers[0].ports should contain 0 or 1 port (field: spec.template.spec.containers[0].ports)
Also, the YAML doesn't specify a forwarding port. It seems to be just assuming that you would only ever set up one port which automatically routes to the one open port on the Docker container. Is that true?
Note: it would be really nice if the YAML came with reference documentation or a schema. That way we could tell what all the possible permutations could be.
Yes, you can only expose one port for a Cloud Run service.
I also find this a curious limitation.
I'm deploying services that use gRPC and expose Prometheus metrics and have been able to multiplex both HTTP/2 and HTTP/1 services in a single port but it requires additional work and is inconsistent with the conceptually underlying Kubernetes.
An excellent feature of GCP is comprehensive and current documentation. Here's Cloud Run Service.
NOTE Found using [APIs Explorer] (https://developers.google.com/apis-explorer) and then finding the Cloud Run Admin API
There are some differences between these knative types and the similar Kubernetes types. One approach I've used is to deploy a known-good service using e.g. gcloud and then corroborate the YAML produced by the service.
For example... top of my head... container ports can't have arbitrary names but must be .... http1 (see link).
Related
i have a handful of dockerized microservices, each is listening for http requests on a certain port, and i have these deployments formalized as kubernetes yaml files
however, i can't figure out a working strategy to expose my deployments on the interwebs (in terms of kubernetes services)
each deployment has multiple replicas, and so i assume each deployment should have a matching load balancer service to expose it to the outside
now i can't figure out a strategy to sanely expose these microservices to the internet... here's what i'm thinking:
the whole cluster is exposed on a domain name, and services are subdomains
say the cluster is available at k8s.mydomain.com
each loadbalancer service (which exposes a corresponding microservice) should be accessible by a subdomain
auth-server.k8s.mydomain.com
profile-server.k8s.mydomain.com
questions-board.k8s.mydomain.com
so requests to each subdomain would be load balanced to the replicas of the matching deployment
so how do i actually achieve this setup? is this desirable?
can i expose each load balancer as a subdomain? is this done automatically?
or do i need an ingress controller?
am i barking up the wrong tree?
i'm looking for general advice on how to expose a single app which is a mosaic of microservices
each service is exposed on the same ip/domain, but each gets its own port
perhaps the whole cluster is accessible at k8s.mydomain.com again
can i map each port to a different load balancer?
k8s.mydomain.com:8000 maps to auth-server-loadbalancer
k8s.mydomain.com:8001 maps to profile-server-loadbalancer
is this possible? it seems less robust and less desirable than strategy 1 above
each service is exposed on its own ip/domain?
perhaps each service specifies a static ip, and my domain has A records pointing each subdomain at each of these ip's in a manual way?
how do i know which static ip's to use? in production? in local dev?
maybe i'm conceptualizing this wrong? can a whole kubernetes cluster map to one ip/domain?
what's the simplest way to expose a bunch of microservies in kubernetes? on the other hand, what's the most robust/ideal way to expose microservices in production? do i need a different strategy for local development in minikube? (i was just going to edit /etc/hosts a lot)
thanks for any advice, cheers
I think the first option is by far the best.
Your Ingress might look like this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: name-virtual-host-ingress
spec:
rules:
- host: auth-server.k8s.mydomain.com
http:
paths:
- backend:
serviceName: service1
servicePort: 80
- host: profile-server.k8s.mydomain.com
http:
paths:
- backend:
serviceName: service2
servicePort: 80
- host: questions-board.k8s.mydomain.com
http:
paths:
- backend:
serviceName: service3
servicePort: 80
You can read more about it on Kubernetes docs regarding Ingress and Name based virtual hosting.
You can also use many Ingress Controllers depending where you will end up setting your cluster. You mentioned that you will be testing this on Minikube so I think nginx ingress will be a good choice here.
If you are thinking about managing your traffic you could consider istio .
Here is a nice guide Setting up HTTP(S) Load Balancing with Ingress and another once Configuring Domain Names with Static IP Addresses.
The first method is typically the format that everyone follows ie each microservice gets its own subdomain. You can achieve the same using Kubernetes ingress (for example Nginx Ingress https://kubernetes.github.io/ingress-nginx/)
They need not be in the same domain also ie you can have both *.example.com and *.example2.com
The second method doesn't scale up as you would have a limited number of available ports and running on non-standard ports comes with its own issues.
Use an ingress:
https://kubernetes.io/docs/concepts/services-networking/ingress/#types-of-ingress
With an ingress, you can assign subdomains to different services, or you can serve all the services under different context roots with some url rewriting.
I don't suggest exposing services using different ports. Nonstandard ports have other problems.
I have built a .NET Core Azure Function using a ServiceBusTrigger. The function works fine when deployed in a regular App Service plan once the appropriate Application settings, such as the Service Bus connection string.
However, I would prefer to host the function as a Docker container on Azure Kubernetes Service (AKS). I have AKS setup and have a number of .NET Core Docker containers running fine there, including some Azure Functions on TimerTriggers.
When I deploy the function using the ServiceBusTrigger, it fails to properly tun and I get "Function host is not running." when I visit the functions IP address. I believe this is because the app settings are not being found.
The problem is I do not know how to include them when hosting in the Docker/Kubernetes environment. I've tried including the appropriate ENV entries in the Docker file, but then I cannot find the corresponding values in the deployment YAML viewed via the Kubernetes dashboard after I've successfully run func deploy from PowerShell.
Most of the Microsoft documentation addresses the TimerTrigger and HttpTrigger cases, but I can find little on the ServiceBusTrigger when using Docker/Kubernetes.
So, how do I include with the appropriate app settings with my deployment?
From this Blog ：Playing with Azure Functions kubernetes integration, you could find a description about add environment variables.
In the deployment.yml, add the env(like AzureWebJobsStorage as an environment variable).
containers:
- image: tsuyoshiushio/queuefunction-azurefunc
imagePullPolicy: Always
name: queuefunction-deployment
env:
- name: AzureWebJobsStorage
value: YOUR_STORAGE_ACCOUNT_CONNECTION_STRING_HERE
ports:
- containerPort: 80
protocol: TCP
Then apply it, it will works.
I have a container which runs a chatbot using python, exposed port 5000 where the bot is running. Now when i deploy this container on kubernetes, I have few questions
Do i need to run nginx container in the pod where my app container is
running ? If yes why do i need to ? since kubernetes does load
balancing
If i run nginx container on port 80, do I need to run my
app container also on 80 or (can i use a different port like 5000)
what role does gunicorn play here ?
I am a little confused because most of the examples i see online everyone pretty much have nginx container in their pods along with the app containers
As you mentioned Kubernetes can handle its own load balancing, so the answer to your first question is no, you don't need to run nginx especially in the pod where your application is.
Typically, services and pods have IPs that are routable by the cluster network and all traffic which ends at an edge router will be dropped. So in Kubernetes there is a collection of rules that allows inbound connections to reach a cluster services. Which we call Ingress:
An API object that manages external access to the services in a
cluster, typically HTTP.
Confusing part is that Ingress on it's own does not do much. You will have to create an Ingress controller which is a daemon deployed as a Pod. Its job is to read the Ingress Resource information and to process that accordingly. Actually any system capable of reverse proxying can be ingress controller. You can read more about Ingress and Ingress controller in practical approach in this article. Also I do not know about your environment so please remember that you should use type LoadBalancer if you are on the Cloud and type NodePort if in bare-metal environment.
Going to your second question you can run your application on any port you want, just remember to adjust that port in all other configuration files.
About ports and how to expose services you should check the sources in documentation on how the Kubernetes model works in comparison to containers model. You can find a an instructive article here.
Unfortunately I do not have experience with gunicorn so I won't be able to tell you what role does it play in here. Hope this helps.
I'm having an issue where because an application was originally configured to execute on docker-compose.
I managed to port and rewrite the .yaml deployment files to Kubernetes, however, the issue lies within the communication of the pods.
The frontend communicates with the backend to access the services, and I assume as it should be in the same network, the frontend calls the services from the localhost.
I don't have access to the code, as it is an proprietary application that was developed by a company and it does not support Kubernetes, so modifying the code is out of question.
I believe the main reason is because the frontend and backend are runnning on different pods, with different IPs.
When the frontend tries to call the APIs, it does not find the service, and returns an error.
Therefore, I'm trying to deploy both the frontend image and backend image into the same pod, so they share the same Cluster IP.
Unfortunately, I do not know how to make a yaml file to create both containers within a single pod.
Is it possible to have both frontend and backend containers running on the same pod, or would there be another way to make the containers communicate (maybe a proxy)?
Yes, you just add entries to the containers section in your yaml file, example:
apiVersion: v1
kind: Pod
metadata:
name: two-containers
spec:
restartPolicy: Never
containers:
- name: nginx-container
image: nginx
- name: debian-container
image: debian
Therefore, I'm trying to deploy both the frontend image and backend image into the same pod, so they share the same Cluster IP.
Although you have the accepted answer already in place that is tackling example of running more containers in the same pod I'd like to point out few details:
Containers should be in the same pod only if they scale together (not if you want to communicate over clusterIP amongst them). Your scenario of frontend/backend division doesn't really look like good candidate to cram them together.
If you opt for containers to be in the same pod they can communicate over localhost (they see each other as if two processes are running on the same host (except the part their file systems are different) and can use localhost for direct communication and because of that can't allocate both the same port. Using cluster IP is as if on same host two processes are communicating over external ip).
More kubernetes philosophy approach here would be to:
Create deployment for backend
Create service for backend (exposing necessary ports)
Create deployment for frontend
Communicate from frontend to backend using backend service name (kube-dns resolves this to cluster ip of backend service) and designated backend ports.
Optionally (for this example) create service for frontend for external access or whatever goes outside. Note that here you can allocate same port as for backend service since they are not living on the same pod (host)...
Some of the benefits of this approach include: you can isolate backend better (backend-frontend communication is within cluster only, not exposed to outside world), you can schedule them independently on nodes, you can scale them independently (say you need more backend power but fronted is handling traffic ok or viceversa), you can replace any of them independently etc...
I've been working on using Rancher for manager our dashboard applications, part of this has involved exposing multiple kibana containers from the same port, and one kibana 3 container exposing on port 80.
I want to therefore send requests on specified ports: 5602, 5603, 5604 to specific containers, so I setup the following docker-compose.yml config:
kibana:
image: rancher/load-balancer-service
ports:
- 5602:5602
- 5603:5603
- 5604:5604
links:
- kibana3:kibana3
- kibana4-logging:kibana4-logging
- kibana4-metrics:kibana4-metrics
labels:
io.rancher.loadbalancer.target.kibana3: 5602=80
io.rancher.loadbalancer.target.kibana4-logging: 5603=5601
io.rancher.loadbalancer.target.kibana4-metrics: 5604=5601
Everything works as expected, but I get sporadic 503's. When I go into the container and look at the haproxy.cfg I see:
frontend d898fb95-ec51-4c73-bdaa-cc0435d8572a_5603_frontend
bind *:5603
mode http
default_backend d898fb95-ec51-4c73-bdaa-cc0435d8572a_5603_2_backend
backend d898fb95-ec51-4c73-bdaa-cc0435d8572a_5603_2_backend
mode http
timeout check 2000
option httpchk GET /status HTTP/1.1
server cbc23ed9-a13a-4546-9001-a82220221513 10.42.60.179:5603 check port 5601 inter 2000 rise 2 fall 3
server 851bdb7d-1f6b-4f61-b454-1e910d5d1490 10.42.113.167:5603
server 215403bb-8cbb-4ff0-b868-6586a8941267 10.42.85.7:5601
The IPs listed are all three Kibana containers, the first container has a health check has it, but none of the others do (kibana3/kibana4.1 dont have a status endpoint). My understanding of the docker-compose config is it should have only the one server per backend, but all three appear to be listed, I assume this is in part down to the sporadic 503s, and removing this manually and restarting the haproxy service does seem to solve the problem.
I am configuring the load balancer incorrectly or is this worth raising as a Github issue with Rancher?
I posted on the Rancher forums as that was suggested from Rancher Labs on twitter: https://forums.rancher.com/t/load-balancer-sporadic-503s-with-multiple-port-bindings/2358
Someone from rancher posted a link to a github issue which was similar to what I was experiencing: https://github.com/rancher/rancher/issues/2475
In summary, the load balancers will rotate through all matching backends, there is a work around involving "dummy" domains, which I've confirmed with my configuration does work, even if it is slightly inelegant.
labels:
# Create a rule that forces all traffic to redis at port 3000 to have a hostname of bogus.com
# This eliminates any traffic from port 3000 to be directed to redis
io.rancher.loadbalancer.target.conf/redis: bogus.com:3000
# Create a rule that forces all traffic to api at port 6379 to have a hostname of bogus.com
# This eliminates any traffic from port 6379 to be directed to api
io.rancher.loadbalancer.target.conf/api: bogus.com:6379
(^^ Copied from rancher github issue, not my workaround)
I'm going to see how easy it would be to route via port and raise a PR/Github issue as I think it's a valid usecase for an LB in this scenario.
Make sure that you are using the port initially exposed on the docker container. For some reason, if you bind it to a different port, HAProxy fails to work. If you are using a container from DockerHub that is using a port already taken on your system, you may have to rebuild that docker container to use a different port by routing it through a proxy like nginx.