I'm trying to build and run a simple Docker container (using docker-compose to do this) on a GCP Instance (Ubuntu 20.04), and it seems that the container cannot access the internet, unless I run it using
docker run --net=host [...]
or use in my docker-compose.yml something like:
service:
build:
...
network: host
network_mode: host
...
I'm wondering why it is so, that a simple docker container on a standard GCP instance with Ubuntu 20.04 should require some specific configuration to access Internet, and why I see almost no mention of this while searching for this issue on the web.
Am I doing something wrong, is there a better way to do this?
See Container networking for Docker and the principle is applied consistently across other container runtimes too.
Using --net=host or network_mode: host binds container(s) to the host's network.
Rather than broadly publishing all of a container's or service's ports to the host network (and thus making them host public), you can be more precise using --publish=[HOST-PORT]:[CONTAINER-PORT] or ports to expose container ports as host ports (and potentially remap these too).
One (of several advantages) to the not-published-by-default behavior is that you must take a second step to publish a container's ports to a host where there is increased possibility that the service may be accessed (via its ports) by undesired actors.
Related
This is a followup to an earlier question that I had asked, "https://stackoverflow.com/questions/72046646/does-docker-persist-the-resolv-conf-from-the-physical-etc-resolv-conf-in-the-co".
I've been testing with containers on 2 different machines, and using "--network host" and from that earlier thread in that case it is using a default "Host" mode network named "host"(?).
Since with "host" mode networking, the container and the app inside the container are basically on the same IP as the physical host where the container is running, under what (example) scenarios would you actually want to create a named "host" mode network and then have container use that named "host" mode network?
What would the advantages/differences be between using the custom/named "host" mode network vs. just using "--network host"?
It seems like both situations (using "--network host" vs. "create network xyz" where xyz is a named host network, and then doing the container "docker run --network xyz" would functionally be the same?
Sorry for the newbie question :( and thanks again in advance.
Jim
I don't think you can create a host-mode named network, and if you did, there'd be no reason to use it. If you need host networking – and you almost certainly don't – use docker run --net host or Compose network_mode: host.
But really, you don't need host networking.
With standard Docker networking, you can use docker run -p to publish individual ports out to the host. You get a choice to not publish a given port, and can remap the port. This also means that if, for example, you're running three services each with their own PostgreSQL server, there's no conflict over the single port 5432.
The cases where you actually need it are pretty limited. If an application listens on a very large number of ports or it doesn't listen on a predictable port then the docker run -p mechanism doesn't work well. If it needs to actively manage the host network then it needs to be given access to it (and it might be better run outside a container). If you've hard-coded localhost in your application, then in Docker your database isn't usually there (configuration via environment variables would be better).
From the Docker documentation:
--publish or -p flag. Publish a container's port(s) to the host.
--expose. Expose a port or a range of ports.
--link. Add link to another container. Is a legacy feature of Docker. It may eventually be removed.
I am using docker-compose with several networks. I do not want to publish any ports to the host, yet when I use expose, the port is then exposed to all the networks that container is connected to. It seems that after a lot of testing and reading I cannot figure out how to limit this to a specific network.
For example in this docker-compose file with where container1 joins the following three networks: internet, email and database.
services:
container1:
networks:
- internet
- email
- database
Now what if I have one specific port that I want to expose to ONLY the database network, so NOT to the host machine and also NOT to the email and internet networks in this example? If I would use ports: on container1 it is exposed to the host or I can bind it to a specific IP address of the host. *I also tried making a custom overlay network, giving the container a static IPv4 address and trying to set the ports in that format in ports: like - '10.8.0.3:80:80', but that also did not work because I think the binding can only happen to a HOST IP address. If i use expose: on container1 the port will be exposed to all three networks: internet, email and database.
I am aware I can make custom firewall ruling but it annoys me that I cannot write such simple config in my docker-compose file. Also, maybe something like 80:10.8.0.3:80 (HOST_IP:HOST_PORT:CONTAINER_IP:CONTAINER_PORT) would make perfect sense here (did not test it).*
Am I missing something or is this really not possible in Docker and Docker-compose?
Also posted here: https://github.com/docker/compose/issues/8795
No, container to container networking in docker is one-size-fits-many. When two containers are on the same network, and ICC has not been disabled, container-to-container communication is unrestricted. Given Docker's push into the developer workflow, I don't expect much development effort to change this.
This is handled by other projects like Kubernetes by offloading the networking to a CNI where various vendors support networking policies. This may be iptables rules, eBPF code, some kind of sidecar proxy, etc to implement it. But it has to be done as the container networking is setup, and docker doesn't have the hooks for you to implement anything there.
Perhaps you could hook into docker events and run various iptables commands for containers after they've been created. The application could also be configured to listen on the specific IP address for the network it trusts, but this requires injecting the subnet you trust and then looking up your container IP in your entrypoint, non-trivial to script up, and I'm not even sure it would work. Otherwise, this is solved by either restructuring the application so components that need to be on a less secure network are minimized, by hardening the sensitive ports, or switching the runtime over to something like Kubernetes with a network policy.
Things that won't help:
Removing exposed ports: this won't help since expose is just documentation. Changing exposed ports doesn't change networking between containers, or between the container and host.
Links: links are a legacy feature that adds entries to the host file when the container is created. This was replaced by creating networks with DNS resolution of other containers.
Removing published ports on the host: This doesn't impact container to container communication. The published port with -p creates a port forward from the host to the container, which you do want to limit, but containers can still communicate over a shared network without that published port.
The answer to this for me was to remove the -p command as that binds the container to the host and makes it available outside the host.
If you don't specify -p options. The container is available on all the networks it is connected to. On whichever port or ports the application is listening on.
It seems the -P forces the container on to the host and binds it to the port specified.
In your example if you don't use -p when staring "container1". "container1" would be available to the networks: internet, email, database with all its ports but not outside the host.
Why is it that Docker prohibits attaching a container to both the host and user defined bridge network?
Secondly, for deployments that require disabling IP forwarding on the host machine does docker recommend deploying docker containers with host networking only, since based on what i understand that seems to be the only option left.
Any insights on the above two?
Thanks
Why is it that Docker prohibits attaching a container to both the host and user defined bridge network?
Because there's no way to "attach" networks when a container is running in the host network namespace.
Docker attaches networks by adding virtual interfaces to a container's isolated network namespace. When running in the global network namespace, there's no sane way to do this: any new interfaces wouldn't be restricted to the container, and would potentially disrupt host networking.
Secondly, for deployments that require disabling IP forwarding on the host machine does docker recommend deploying docker containers with host networking only, since based on what i understand that seems to be the only option left.
That's probably the only easy option.
You could run a proxy service on the host that would expose services in Docker containers. You could potentially even automate that by monitoring the Docker for events and getting information about published ports. Otherwise you would need to manually implement the appropriate configuration.
If you tell docker-compose to scale a service, and do NOT expose its ports,
docker-compose scale dataservice=2
There will be two IPs in the network that the dns name dataservice will resolve to. So, services that reach it by hostname will load balance.
I would also like to do this to the edge proxy as well. The point would be that
docker-compose scale edgeproxy=2
Would cause edgeproxy to resolve to one of 2 possible IP Addresses.
But the semantics of exposing ports is wrong for this. If I expose:
8443:8443
Then it will try to bind each edgeproxy to be bound to host 8443. What I want is more like:
0.0.0.0:8443:edgeproxy:8443
Where when you try to come into the docker network via host 8443, it randomly selects an edgeproxy:8443 IP to bind the incoming TCP connection to.
Is there an alternative to just do a port-forward? I want a port that can get me in to talk to any ip that will resolve as edgeproxy.
This is provided by swarm mode. You can enable a single node swarm cluster with:
docker swarm init
And then deploy your compose file as a stack with:
docker stack deploy -c docker-compose.yml $stack_name
There are quite a few differences from docker compose including:
Swarm doesn't build images
You manage the target state with docker service commands, trying to stop a container with docker stop won't work since swarm will restart it
The compose file needs to be in a v3 syntax
Networks will be an overlay network, and not attachable by containers outside of swarm, by default
One of the main changes is that exposed ports are published on an ingress network managed by swarm mode, and connections are round robin load balanced to your containers. You can also define a replica count inside the compose file, eliminating the need to run a scale command.
See more at: https://docs.docker.com/engine/swarm/
How to use confluent/cp-kafka image in docker compose with exposing on localhost and my network container name kafka?
Do not link this as duplicate of:
Connect to docker kafka container from localhost and another docker container
Cannot produce message to kafka from service running in docker
These do not solve my issue because the methods they use are depreciated by confluent/cp-kafka and I want to connect on localhost and on the docker network.
In the configure script on confluent/cp-kafka they do this annoying task:
# By default, LISTENERS is derived from ADVERTISED_LISTENERS by replacing
# hosts with 0.0.0.0. This is good default as it ensures that the broker
# process listens on all ports.
if [[ -z "${KAFKA_LISTENERS-}" ]]
then
export KAFKA_LISTENERS
KAFKA_LISTENERS=$(cub listeners "$KAFKA_ADVERTISED_LISTENERS")
fi
It always sets whatever I give KAFKA_ADVERTISED_LISTENERS to 0.0.0.0! Using the docker network, doing
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9093,PLAINTEXT://kafka:9093
I expect the listeners to be either localhost:9092 or 0.0.0.0:9092 and some docker ip PLAINTEXT://172.17.0.1:9093 (whatever kafka resolves to on the docker network)
Currently I can get only one or the other to work. So using localhost, it only works on the host system, no docker containers can access it. Using kafka, it only works in the docker network, no host applications can access it. I want it to work with both. I am using docker compose so that I can have zookeeper, kafka, redis, and my application start up. I have other applications that will startup without docker.
Update
So when I set PLAINTEXT://localhost:9092 I can access kafka running docker, outside of docker.
When I set PLAINTEXT://kafka:9092 I cannot access kafka running docker, outside of docker.
This is expected, however doing this: PLAINTEXT://localhost:9092,PLAINTEXT://kafka:9093 I would expect to access kafka running docker, both inside and outside docker. The confluent/cp-kafka image is wiping out localhost and kafka. Setting them both to 0.0.0.0, then throwing an error that I set 2 different ports to the same ip...
Maybe I'm just clashing into some opinionated docker image and should look for a different image...
Maybe I'm just clashing into some opinionated docker image and should look for a different image...
The image is fine. You might want to read this explanation of the listeners.
tl;dr - you don't want to (and shouldn't?) use the same listener "protocol" in different networks.
Use the advertised.listeners, no need to edit the listeners
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
When PLAINTEXT://localhost:9093 is being loaded inside of the container, you need to add port mappings for 9093, which should be self explanatory, and you connect to localhost:9093 and it should work.
Then, if you also had PLAINTEXT://kafka:9092, that will only work within the Docker Compose network overlay, not externally to your DNS servers, because that's how Docker networking works. You should be able to run other applications as part of that Docker network with the --network flag, or link containers using Docker Compose
Keep in mind that if you're running on Mac, the recommended way (as per the Confluent docs) is to run these containers in Docker Machine, in a VM, where you can manage the external port mappings correctly using the --net=host flag of Docker. However, using the blog above, it all works fine on a Mac outside a VM.