Currently I am working on a project where we have a single trusted master server, and multiple untrusted (physically in an unsecured location) hosts (which are all replicas of each other in different physical locations).
We are using Ansible to automate the setup and configuration management however I am very unimpressed in how big of a gap we have in our development and testing environments, and production environment, as well as the general complexity we have in configuration of the network as well as containers themselves.
I'm curious if Kubernetes would be a good option for orchestrating this? Basically, multiple unique copies of the same pod(s) on all untrusted hosts must be kept running, and communication should be restricted between the hosts, and only allowed between specific containers in the same host and specific containers between the hosts and the main server.
There's a little bit of a lack of info here. I'm going to make the following assumptions:
K8s nodes are untrusted
K8s masters are trusted
K8s nodes cannot communicate with each other
Containers on the same host can communicate with each other
Kubernetes operates on the model that:
all containers can communicate with all other containers without NAT
all nodes can communicate with all containers (and vice-versa) without NAT
the IP that a container sees itself as is the same IP that others see it as
Bearing this in mind, you're going to have some difficulty here doing what you want.
If you can change your physical network requirements, and ensure that all nodes can communicate with each other, you might be able to use Calico's Network Policy to segregate access at the pod level, but that depends entirely on your flexibility.
Related
I understand that if I use the host network driver for a container, that container’s network stack is not isolated from the Docker host.
I also believe understand conceptually that a good reasons to still do it might be when "Security is not an Issue or concern" and network throughput performance is important but I am struggling to think of a real world example of when I can or should do this. A naive example I can think of is a public facing load-balancer or static file web server.
I realize it may be possible to mitigate the security concerns outside of using host services like AWS or Google Cloud if hosted there but what if that wasn't an option!
When would or should an you use it in a production environment?
How can you mitigate the security concerns regardless of hosting environment?
How should you interact with other services in other docker networks?
I am struggling to think of a real world example of when I can or should do this. ... When would or should an you use it in a production environment?
Your application does not run on TCP or UDP, but another protocol
Your application requires a large range of incoming ports to be published (by default a docker-proxy process is spawned per published port, this can be excessive for a large range)
Your application works with multi-cast or broadcast network traffic
Your application needs to modify the networking layer of the host itself, e.g. a VPN
How can you mitigate the security concerns regardless of hosting environment?
You need to trust this application. You've removed a layer of docker namespacing and at that point, the container is a packaging format and likely fits in with the rest of your tooling, but doesn't require the same security approach you may have for other containers.
How should you interact with other services in other docker networks?
You would interact via published ports of the other containers, same as you would an application running outside of a container that needs to connect to an application inside of a container.
but I am struggling to think of a real world example of when I can or should do this.
Here is real world example: We use host network to speed up build stage of our gitlab ci/cd pipeline.
Container in question is up and running only during build phase, doesn't have any port exposed, needs faster network to download all the necessary pieces to build and push docker image and we experienced (in some intermittent occasions) issues with throughput and inconsistent behavior during build stage that we resolved with host network. Although with host network we "expose" ip of such a container, we still don't expose any ports and after build phase is finished container is discarded.
I know this doesn't answers all of your questions, but is requested real world example.
I am trying to set up an OrientDB server cluster, where multiple pods have their own orientDB server, and they all contain the same data, no matter which pod's database you modify.
When I set this up locally, I can just start up multiple containers that run OrientDB's dserver.sh, and OrientDB's multicast ability allows the different containers/servers to find themselves. However when I'm running multiple instances on different pods they can't find themselves.
I'm currently thinking I'm supposed to have the pods on the same kubernetes service, and then they should be able to do the same? But I'm not making any headway.
How do I get multiple pods on OpenShift all to connect to each other, as if I was doing this with different containers on my local machine?
Thanks.
You can get multicast support in your cluster using Weave network solution.
Summary
With multicast, data can be distributed to multiple receivers that are in the same multicast group. It is used by
applications that generate and distribute data feeds to other
applications such as financial services.
Weave Net fully emulates a layer 2 network, and operates just like a
server that is attached to a standard Ethernet network. Multicast with
Weave Net works even if the underlying network doesn’t support it,
which is the case with Amazon Web Services (AWS), Azure, GCP and other
public clouds.
What it Does
Provides full multicast support for contained and uncontained applications
Works in data centers and public clouds, regardless of the underlying network
Uses industry-standard VXLAN and in-kernel switching for top performance
Automatic service discovery through DNS lookups
In our company ~7 projects, each based on Docker. Each project contain base services, like MySQL, Nginx, PHP. Some of projects communicate with other projects. Because of many services on same port, we make new docker host (docker-machine) for each project. From here few problems are coming:
VirtualBox assign random IP to each Docker host, depends on sequence of executing.
Hard to switch from project to project, need to set different shell envs all the time. Easy to make mistake.
Well, I'm searching for more enterprise solution to manage many docker machines. Or a some technique that can help me with current situation.
I had similar problems last summer.
First, I started to deploy my projects to swarm-cluster as services, instead of clustering several docker VMs. This enabled me to play around services with only the service IDs. It is important that how to separate projects into services, this part may be cumbersome depending on your project.
https://docs.docker.com/engine/swarm/swarm-tutorial/deploy-service/
Then, I build my configuration and monitoring software once on swarm-manager and use it. You can use your automation tools on docker-manager to control services.
A virtual machine consumes resources and it is better to avoid it if is not necessarily. Instead you could deploy the projects in the docker swarm on bare metals.
But because every project has an entry point that needs to be accesible from the outside world (i.e. https://site1.com and https://site2.com) you can't expose the same port (443 or 80) for all the frontend services in the same swarm. For this you can use a reverse proxy like HAProxy or Nginx that forwards the requests to the right service based on the hostname. The reverse proxy could be also a service in the swarm. In this situation you should not expose the projects' ports anymore.
A reverse proxy has many other advantages, like SSL termination (this makes the SSL certificate management a lot easier).
If you add the projects to the same custom network then the services from different projects could communicate securely and directly, using their docker service name and the internal port (i.e. 80).
I have a cluster of database nodes hosted in VMs or Bare Metal and I'd like to create additional database nodes (hosted in Kubernetes Pods) and have them join the existing cluster (ones hosted in VMs or bare metal).
In order to have them join the cluster, each database must be able to resolve the other via distinct IP and port. Within the Kubernetes network environment, there is no issue with this and no issue between the existing VM-hosted DBs. The sticking point is that I can't seem to see a way for the VM-hosted DBs to individually route to each POD-hosted DB. Is there a Kubernetes configuration that will allow me to have each pod/DB individually routable on specific ports while sharing the same NIC for the host running the cluster? Do I need to front each POD with it's own Service?
Here is the sort of configuration I'm trying to achieve with conceptual IP address spaces.
The approach I take personaly for a similar case is to actualy make it possible for nodes in the non-kubernetes environment to be able to talk to the pods them selves. Depending on your network configuration this might be quite easy to achieve.
In my case I simply have 2 additional elements running on VMs that need to access my k8s internals :
- flannel : this actually ties my VMs to the same flannel network as k8s pods operate in
- kube-proxy : translates service ips to pod ips using iptables (in cases where I need to access by service IP)
You could avoid setting this up on VMs or their hosts if you can solve this on a gateway level (ie. have flannel/proxy on your network gate and augment it with some SNAT rules).
Having NodePort/LB service per in-k8s db might work if your DB sticks to the IPs you give (not only use for discovery bootstraping where later on the IPs are replaced with actual IPs of DBs - iirc mongo usually does something like that)
I installed an 8-node kubernetes cluster (1 master + 7 minion) but I faced a networking problem among minions.
I installed my cluster according to this step-by-step Fedora manual, so I use Fedora 20 with its testing repository to get kubernetes binaries.
After installing, I wanted to try the guestbook example, but it seems to me there is a problem with the inter-container networking.
Although containers/PODs are in running state and I can reach my 3 frontend containers (via browser) and the redis containers as well (via natcat), but the frontend, which not on the same host with the redis, cannot reach redis master. The frontend's PHP give back network exception.
Can anybody help me why the containers cannot reach each other among the hosts?
I hope I could describe my setup enough accurately and thanks in advance.
The Fedora guide you followed will only get you running on a single machine. It avoids the issues around setting up networking across nodes.
For kubernetes to work, the following network set up must be satisfied:
Every container should be able to talk to every other container, even across nodes. This means also that the bridge IP range for those containers must not overlap.
Code running on any node that isn't in a container should be able to reach every container (and vise-versa), even across nodes.
It is not necessary (but useful) if computers on the network that aren't part of the cluster can reach the containers directly.
There are a lot of ways to achieve this -- for instance the set up for vagrant sets up GRE tunnels between each node. On GCE we use features of the platform to do the routing. If you are on physical machines on a switch you can probably just do a big layer 2 network w/ bridges. A bulletproof way to get started (but perhaps not the most performant, depending on your set up) is to use something like flannel.
We are working on making this stuff easier to start up (without using a mess of shell scripts) and are thinking of building something like flannel in so that there is a reasonable default.