I'm trying to setup a 3 node Docker swarm cluster on Hetzner cloud, using wireguard VPN (setup on interface wg0) to build the local network between nodes. Networking works fine across nodes using VPN IP (ports 7946/tcp , 7946/udp and 4789/udp are open as reported here). I start docker swarm cluster with the following commands:
docker swarm init --advertise-addr wg0 --listen-addr wg0
docker swarm join --token SWMTKN-1-xxx --advertise-addr wg0 --listen-addr wg0 10.0.0.1:2377
If I try to run a service on this swarm, it seems to run correctly, every container can reach the other on different node and inspecting them, they join the ingress network and an overlay network created by me, as expected. The problem arises when I try to access from outside the service exposed port; it only works if I target the node where the container is running, so it seems that the routing mesh is not correctly working. I've not found any error in docker logs or syslog.
Note: I'm using docker 18.06.1-ce
I had this issue and made the following changes:
I moved my wireguard addresses from 10.0.* to 192.168.* (i have a feeling that swarm is allocating on top of these).
docker swarm init --advertise-addr 192.168.2.123 with the wireguard ip4 address of the master node.
That managed to fix it, and it still works after rebooting the master and worker nodes!
Related
I'm working on docker swarm. When I connect to a worker node I get this error:
Error response from daemon: rpc error: code = 14 desc = grpc: the
connection is unavailable
I have already stopped firewall and setenforce 0. What could be the problem?
If you are using VM then you can init docker swarm with alternative IP Address using docker "swarm init --advertise-addr :"
Example: docker swarm init --advertise-addr 192.168.99.100:2377
and then add the nodes to the swarm.
Example: docker swarm join --token --advertise-addr :
docker swarm join --token SWMTKN-1-RANDOMTOKEN 192.168.99.100:2377
Some people say that it works only with 2377 port.
Check yourself if it works for you as well.
If you are using the swarm init --advertise-addr <some ip>.Then you will get the join token but when I am going to add new swarm to the manager as a worker then i am getting the same error node is already part of swarm.
So take care while using vm ip address. It should be different from and exact match to the manager ip.
We are running two docker containers with network as host for each container, We are able to communicate to container from outside world, but we are not able to communicate between the containers.
Is there a way to configure docker network so that containers can communication among themselves as well as outside world.
When you configure a docker container to use "host" networking, you completely remove the docker networking stack, including the container to container networking and built in dns discovery. Everything you see in the container's network is identical to what you see on the host. So another container listening on localhost is visible on 127.0.0.1 from the host and from other containers with host networking.
This is not the recommended way to run docker containers in most scenarios. You typically create a docker network for a group of containers, start those containers on that docker network, and then refer to the other containers by their container name.
Found a solution to my question, I am able to communicate between containers using docker0 network (default network), and to map select ports of container to outside world I am using -p mapping to map from container port to host port.
as #BMitch suggested a separate network to connect only the containers that need inter container comm is better than using docker0 as all containers without any explicit network configuration connect to docker0.
Before asking this question I have read quiet of articles and stackoverflow questions but I couldn't get the right answer for my setup(perhaps it is already answered). Here is the architecture I have been struggling to get it to work.
I have three physical machines and I would like to setup the Docker swarm with multi-host networking so that I can run docker-compose.
For example:
Machine 1(Docker Swarm Manager and Contains Consoul)(192.168.5.11)
Machine 2(Docker Swarm Node)(192.168.5.12)
Machine 3 (Docker Swarm Node)(192.168.5.13)
And I need to run docker-compose from any other separate machine.
I have tried Docker article but in that article it is all setup under the same physical machine using docker-machine and virtual box. How can I achieve above in three remote machines. Any help appreciated.
The latest version of Docker has Swarm Mode built in, so you don't need Consul.
To set up on your boxes, make sure they all have docker version of 1.12 or higher and then you just need to initialise the swarm and join it.
On Machine 1 run:
docker swarm init --advertise-addr 192.168.5.11
The output from that will tell you the command to run on Machine 2 and 3 to join them to the swarm. You'll have a unique swarm token, and the command is something like:
docker swarm join \
--token SWMTKN-1-49nj1... \
192.168.5.11:2377
Now you have a 3-node swarm. Back on Machine 1 you can create a multi-host overlay network:
docker network create -d overlay my-app
And then you run workloads in the network by deploying services. If you want to use Compose with Swarm Mode, you need to use distributed application bundles - which are currently only in the experimental build of Docker.
I figured this needs an update, as docker compose files are supported in docker swarm
Initialize the swarm on Machine 1 using
docker swarm init --advertise-addr 192.168.5.11
Join the swarm from Machine 2 & 3 using
docker swarm join \
--token <swarm token from previous step> 192.168.5.11:2377 \
--advertise-addr eth0
eth0 is the network interface on machines 2 & 3, & could be different
based on your config. I found that without the --advertise-addr
option, containers couldn't talk to each other across hosts.
To list all the nodes in the swarm & see their status
docker node ls
After this, deploy the stack (group of services or containers) from a compose file
docker stack deploy -c <compose-file> my-app
This will create all the containers across multiple hosts
To list services (containers) on the swarm run docker service ls
See docker docs Getting started with swarm mode
Is there a way how I could get IPs of nodes joined in cluster?
In "old" swarm there is command that you can run on manager machine. docker exec -it <containerid> /swarm list consul://x.x.x.x:8500
To see a list of nodes, use:
docker node ls
Unfortunately they don't include IP's and ports in this output. You can run a docker node inspect $hostname on each one to get it's swarm ip/port. Then if you need to add more nodes to your cluster, you can use docker swarm join-token worker which does include the needed IP/port in it's output.
What docker node ls does provide is hostnames of each node in your swarm cluster. Unlike the standalone swarm, you do not connect your docker client directly to the swarm port. You now access it from one of the manager hosts in the same way you'd connect to that host before to init/join the swarm. After connecting to one of the manager hosts, you use docker service commands to control your running services.
Experimenting with Docker Swarm with Docker Desktop for Mac. I tried this:
docker-machine create -d virtualbox node-1
docker-machine create -d virtualbox node-2
docker-machine create -d virtualbox node-3
eval $(docker-machine env node-1)
docker swarm init \
--secret my-secret \
--auto-accept worker \
--listen-addr $(docker-machine ip node-1):2377
The last command (docker swarm init) returns this error:
Error response from daemon: could not choose an IP address to
advertise since this system has multiple addresses
I have no idea what's going on. Anyone have any idea how to debug?
First look for the public IP of your machine on your network
ifconfig
pick the physical one like 192.168.1.x (not docker0, that is a virtual IP internal to Docker)
docker swarm init --advertise-addr 192.1.68.1.x
(will default to port 2377)
Update 2017-05-24:
The prior answer was for an early state of swarm mode. The secret and auto-accept options have since been removed, and the advertise-addr option has been added. This can now by done with:
docker swarm init \
--advertise-addr $(docker-machine ip node-1)
The port will default to 2377. You can also use a network interface name instead of an IP address and swarm will lookup the IP address on that interface. The listener address is still an option but the default is to listen on all interfaces which is typically the preferred solution.
Original answer:
I haven't done this with docker-machine yet, but I do know that the new swarm is very sensitive to the entries in /etc/hosts. Make sure your ip and hostname are in that file, and only in a single place (not also mapped to loopback or any other internal addresses). As of RC3, they are also using the listener address for the advertise address, too, so make sure this hostname or ip can be referenced by all nodes in the swarm (pretty sure a fix is coming for that, if not already here).
To minimize the risk of issues between client and server versions, I'd also run the commands directly inside the virtualbox, rather than with docker-machine environment variables.
According to Docker´s guide: https://docs.docker.com/get-started/part4/#create-a-cluster
Getting an error about needing to use --advertise-addr?
Copy the IP address for your virtual machine by running docker-machine
ls, then run the docker swarm init command again, using that IP and
specifying port 2377 (the port for swarm joins) with --advertise-addr.
For example:
docker-machine ssh myvm1 "docker swarm init --advertise-addr
192.168.99.100:2377"
This works for me
docker swarm init --advertise-addr 127.0.0.1
Got the same error when using docker with envs to connect to the docker-machine-created machine.
After docker-machine ssh <machine-name>, and doing the docker swarm init locally on the machine, I got the message about --advertise-addr as well. The local command docker swarm init --listen-addr 192.168.99.100:2377 --advertise-addr 192.168.99.100:2377 then worked.
Check docker --version and make sure client and server are on the same version. If they are different, use the following command to pull the boot2docker version that matches with the docker client on your machine.
docker-machine create --driver virtualbox --virtualbox-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.12.0-rc4/boot2docker-experimental.iso manager1
Please ssh into the node 1 and then apply same command over there