If you tell docker-compose to scale a service, and do NOT expose its ports,
docker-compose scale dataservice=2
There will be two IPs in the network that the dns name dataservice will resolve to. So, services that reach it by hostname will load balance.
I would also like to do this to the edge proxy as well. The point would be that
docker-compose scale edgeproxy=2
Would cause edgeproxy to resolve to one of 2 possible IP Addresses.
But the semantics of exposing ports is wrong for this. If I expose:
8443:8443
Then it will try to bind each edgeproxy to be bound to host 8443. What I want is more like:
0.0.0.0:8443:edgeproxy:8443
Where when you try to come into the docker network via host 8443, it randomly selects an edgeproxy:8443 IP to bind the incoming TCP connection to.
Is there an alternative to just do a port-forward? I want a port that can get me in to talk to any ip that will resolve as edgeproxy.
This is provided by swarm mode. You can enable a single node swarm cluster with:
docker swarm init
And then deploy your compose file as a stack with:
docker stack deploy -c docker-compose.yml $stack_name
There are quite a few differences from docker compose including:
Swarm doesn't build images
You manage the target state with docker service commands, trying to stop a container with docker stop won't work since swarm will restart it
The compose file needs to be in a v3 syntax
Networks will be an overlay network, and not attachable by containers outside of swarm, by default
One of the main changes is that exposed ports are published on an ingress network managed by swarm mode, and connections are round robin load balanced to your containers. You can also define a replica count inside the compose file, eliminating the need to run a scale command.
See more at: https://docs.docker.com/engine/swarm/
Related
I'm trying to build and run a simple Docker container (using docker-compose to do this) on a GCP Instance (Ubuntu 20.04), and it seems that the container cannot access the internet, unless I run it using
docker run --net=host [...]
or use in my docker-compose.yml something like:
service:
build:
...
network: host
network_mode: host
...
I'm wondering why it is so, that a simple docker container on a standard GCP instance with Ubuntu 20.04 should require some specific configuration to access Internet, and why I see almost no mention of this while searching for this issue on the web.
Am I doing something wrong, is there a better way to do this?
See Container networking for Docker and the principle is applied consistently across other container runtimes too.
Using --net=host or network_mode: host binds container(s) to the host's network.
Rather than broadly publishing all of a container's or service's ports to the host network (and thus making them host public), you can be more precise using --publish=[HOST-PORT]:[CONTAINER-PORT] or ports to expose container ports as host ports (and potentially remap these too).
One (of several advantages) to the not-published-by-default behavior is that you must take a second step to publish a container's ports to a host where there is increased possibility that the service may be accessed (via its ports) by undesired actors.
I'm deploying a stack of services through the command:
docker stack deploy -c <docker-compose.yml> <stack-name>
And I'm mapping ports of one of these services on docker compose with ports: 8000:8000.
The network driver being used is overlay.
I can access these services via localhost:8000, via Peers IP(?).
When I inspect the network created, I can see the local IPs of each container (for instance, 10.0.1.2). But Where is the external IP of container (the one like 172.0. ...) ?
I am running these docker container on a virtual machine ubuntu.
How can I access the services running on containers from other nodes running on other networks? Isn't possible to access via hostIP:port?
If so, how do I get the host IP? When I do docker-machine IP I get "host is not running".
[EDIT: I wasn't doing port mapping between the host and the VM in virtualbox. Now it works!]
Whats the best way to communicate between containers on the same swarm?
Thanks
Whats the best way to communicate between containers on the same swarm? Through name discovery?
In general if you communicate between containers you should use the container/service name.
And for your other problem you probably wan't a reverse proxy like nginx or traefik.
From spark documentation I know that the ports that executors, i.e. workers (because by default there is just one executor per a worker) use for establishing connection with master are randomly determined, but how could I setup their range to publish those ports in docker. Also, if a worker establishes a connection with another container (which is not part of the distributed system), do I need to publish the port on which the worker would get returned data from the container (e.g. via an https request)?
Just to note, I do not use docker-compose.yml because I do not need the containers to be set as services and I want to add/remove containers when needed by increase/decrease in number of customers.
You should use the same docker network for all containers which will communicate with each other. Containers can reach others using container name (on all ports) just like if different hosts on a network.
Create a network (needed only once)
docker network create <network_name>
when you launch a container use --network to connect container to the network
docker run --network=<network_name> --name <container_name> <image>
You can also connect existing containers to networks
docker network connect <network_name> <container_name>
Reference:
https://docs.docker.com/engine/reference/commandline/network_create/
https://docs.docker.com/engine/reference/run/
I have not done any practical with the docker and container, But as per my knowledge.
As per the documents available online I did not get the details about the running two or more containers at the same time.
Docker allows container to map port address of container to the host machine.
Now, the question is can we run multiple container at the same time on docker? if yes then if two containers are mapped to same port number then how does the port is handled in this case?
Also out of curiosity, can two containers on docker communicate with each other?
Yes you can run multiple containers on a single host; docker is designed for exactly that.
You cannot map two containers of different images to the same port number; you get an error response if you try. However, if your containers run the same image (e.g.2 instances of a webapp) you could run them as a service, and have them exposed on the same port. Docker will load-balance the requests. You can read more about services here or follow the Get Started (Part 3, services) here
Yes, the containers on a single host can communicate with each other, by container name. For example if you have one container running MongoDB called mongo, and another one running Node.js called webserver, the webserver container can connect to the database by using the name mongo e.g. db.Connect("mongodb://mongo:27017/testdb").
We can run more one than one Docker at a time in a host but yes we will hit the limitation of binding the same port to the docker; so to resolve this we need to bind different port in the host to docker that is if you are running mongo-db then its default port is 27017 so we can run two mongo-db as -p 27017:27017 for Docker D1 and -p 27018:27017 for Docker D2 and 5000:27017 for docker D3; Like this you can bind different host port to map to 27017 for mongo-db port; Now your question is how to manage this ports from host then I would recommend you to use nginx for port managing in the host machine.
Coming to your next question all dockers are connected to default docker0 bridge network so we can connect to any of the dockers connected to default bridge 'docker0' network; If I am right it will come with ipaddress of 172.x.x.x network. Get inside to the docker and run 'ip addr' to see the ip-address assigned to the dockers and you can test connection by running ping command.
Yes two containers can run same time, they can also communicate with each other also, you can define your own network and they can communicate with each other. if two containers have their private ports, they are their internal ports, one container port does not collide with another container port. if you want to expose the port to host, then you have to publish the port(s).
Use case: haproxy container running with docker compose. I want to have the container discover which hosts are available in order to recreate haproxy config and reload it.
I know the there will be one or more containers named server1 and server2 available. From inside the haproxy container I can query dns for server1 and receive more than one IP address. Is that the only way to know when a new server1 cointainer becomes available or dies? I know I can use the docker api from python running inside a container that hast the docker host socket mapped to it, but I'm not sure that will work when running on swarm.
The perfect solution would be an api or command that let's me register an event handler that is called when a new container joins the network.
There is a solutions that you can use Registrator (https://github.com/gliderlabs/registrator), Consul and Consul Template.
Consul is a Service Discovery
Consul-Template watches Consul and updates HA Proxy config and reload it.
Registrator listens Docker Engine and update Consul if there is any container is up or down.
Please see the image:
For the full tutorial, you can refer to my blog (https://sonnguyen.ws/microservices-with-docker-swarm-and-consul/) to know how to implement it.