I have three replicated containers running RabbitMQ in a Swarm Mode service. Each container acts as a RabbitMQ node to create a RabbitMQ cluster. The docker service create for this is as follows:
docker service create -e RABBITMQ_ERLANG_COOKIE='mysecretcookie' --replicas 4 --network rnet -p 15672:15672 -p 5672:5672 rabbitmq
I'm able to successfully create a RabbitMQ cluster using rabbitmqctl <hostname> join_cluster on each of the slave nodes thereafter.
My problem is that if a swarm node goes down, Docker starts a new container on a different node however the container has a different random hostname.
Since RabbitMQ uses the hostname to identify RabbitMQ cluster nodes, it doesn't recognise the new hostname on the new container so it assumes that the original node is indefinitely down.
I tried using the new templating feature in Docker 1.13 which allows you to create static hostnames by specifying --hostname="{{.Node.ID}}-{{.Service.Name}}" within docker service create. However, you cannot currently discover containers based on this custom hostname so a RabbitMQ cluster can't be created in this way.
I want to be able to have RabbitMQ automatically re-join a cluster node after Docker starts running a new container. Is this possible?
I want to be able to have RabbitMQ automatically re-join a cluster node after Docker starts running a new container. Is this possible?
I assume you mean "re-join a cluster node existing in the newly started docker container".
The answer - it's not. Because, by definition, it would not be re-joining but simply joining since, from RabbitMQ's perspective, this is simply a new machine (computer, vm, docker...) in the network.
Related
after I create a docker service using the below code
docker service create --name db --network era-networkkk -p 3306:3306 --mount type=bind,source=$(pwd)/data/mysql,destination=/var/lib/mysql schema
and when I check the services using
docker services ls
it shows the name as db
but when I use the command
docker ps
container name have some randomly generated numbers after the name
How can I solve this problem?
I think that behaviour is absolutely intended. What if your swarm is configured to start multiple containers of the same image on a single swarm node? These containers can't have the same name. So there has to be a suffix on the container names so there is no name collision. Why would you want to influence the container's names? Normally when working with clusters you are working on service level instead of container level.
I think the reason for this is that when you create a service you don't necessarily care what the container names are. You would usually create a service when docker is in swarm mode. With swarm mode you set up a cluster of nodes, I guess you only have one node for dev purposes. However when you have more than one cluster then the service would create as many containers as you specify with the --replicas option. Any requests to your application would then be load balanced across the containers in your cluster via the service.
Have a look at the docker documentation https://docs.docker.com/engine/swarm/how-swarm-mode-works/services/ it may help to clarify how all of this works.
From spark documentation I know that the ports that executors, i.e. workers (because by default there is just one executor per a worker) use for establishing connection with master are randomly determined, but how could I setup their range to publish those ports in docker. Also, if a worker establishes a connection with another container (which is not part of the distributed system), do I need to publish the port on which the worker would get returned data from the container (e.g. via an https request)?
Just to note, I do not use docker-compose.yml because I do not need the containers to be set as services and I want to add/remove containers when needed by increase/decrease in number of customers.
You should use the same docker network for all containers which will communicate with each other. Containers can reach others using container name (on all ports) just like if different hosts on a network.
Create a network (needed only once)
docker network create <network_name>
when you launch a container use --network to connect container to the network
docker run --network=<network_name> --name <container_name> <image>
You can also connect existing containers to networks
docker network connect <network_name> <container_name>
Reference:
https://docs.docker.com/engine/reference/commandline/network_create/
https://docs.docker.com/engine/reference/run/
I am using Docker version 17.12.1-ce.
I have set up a swarm with two nodes, and I have a stack running on the manager, while I am to instantiate new nodes on the worker (not within a service, but as stand-alone containers).
So far I have been unable to find a way to instantiate containers on the worker specifically, and/or to verify that the new container actually got deployed on the worker.
I have read the answer to this question which led me to run containers with the -e option specifying constraint:Role==worker, constraint:node==<nodeId> or constraint:<custom label>==<value>, and this github issue from 2016 showing the docker info command outputting just the information I would need (i.e. how many containers are on each node at any given time), however I am not sure if this is a feature of the stand-alone swarm, since docker info only the number of nodes, but no detailed info for each node. I have also tried with docker -D info.
Specifically, I need to:
Manually specify which node to deploy a stand-alone container to (i.e. not related to a service).
Check that a container is running on a specific swarm node, or check how many containers are running on a node.
Swarm commands will only care/show service-related containers. If you create one with docker run, then you'll need to use something like ssh node2 docker ps to see all containers on that node.
I recommend you do your best in a Swarm to have all containers as part of a service. If you need a container to run on nodeX, then you can create a service with a "node constraint" using labels and constraints. In this case you could restrict the single replica of that service to a node's hostname.
docker service create --constraint Node.Hostname==swarm2 nginx
To see all tasks on a node from any swarm manager:
docker node ps <nodename_or_id>
I am having a problem trying to implement the best way to add new container to an existing cluster while all containers run in docker.
Assuming I have a docker swarm, and whenever a container stops/fails for some reason, the swarm bring up new container and expect it to add itself to the cluster.
How can I make any container be able to add itself to a cluster?
I mean, for example, if I want to create a RabbitMQ HA cluster, I need to create a master, and then create slaves, assuming every instance of RabbitMQ (master or slave) is a container, let's now assume that one of them fails, we have 2 options:
1) slave container has failed.
2) master container has failed.
Usually, a service which have the ability to run as a cluster, it also has the ability to elect a new leader to be the master, so, assuming this scenerio is working seemlesly without any intervention, how would a new container added to the swarm (using docker swarm) will be able to add itself to the cluster?
The problem here is, the new container is not created with new arguments every time, the container is always created as it was deployed first time, which means, I can't just change it's command line arguments, and this is a cloud, so I can't hard code an IP to use.
Something here is missing.
Maybe trying to declare a "Service" in the "docker Swarm" level, will acctualy let the new container the ability to add itself to the cluster without really knowing anything the other machines in the cluster...
There are quite a few options for scaling out containers with Swarm. It can range from being as simple as passing in the information via a container environment variable to something as extensive as service discovery.
Here are a few options:
Pass in IP as container environment variable. e.g. docker run -td -e HOST_IP=$(ifconfig wlan0 | awk '/t addr:/{gsub(/.*:/,"",$2);print$2}') somecontainer:latest
this would set the internal container environment variable HOST_IP to the IP of the machine it was started on.
Service Discovery. Querying a known point of entry to determine the information about any required services such as IP, Port, ect.
This is the most common type of scale-out option. You can read more about it in the official Docker docs. The high level overview is that you set up a service like Consul on the masters, which you have your services query to find the information of other relevant services. Example: Web server requires DB. DB would add itself to Consul, the web server would start up and query Consul for the databases IP and port.
Network Overlay. Creating a network in swarm for your services to communicate with each other.
Example:
$ docker network create -d overlay mynet
$ docker service create –name frontend –replicas 5 -p 80:80/tcp –network mynet mywebapp
$ docker service create –name redis –network mynet redis:latest
This allows the web app to communicate with redis by placing them on the same network.
Lastly, in your example above it would be best to deploy it as 2 separate containers which you scale individually. e.g. Deploy one MASTER and one SLAVE container. Then you would scale each dependent on the number you needed. e.g. to scale to 3 slaves you would go docker service scale <SERVICE-ID>=<NUMBER-OF-TASKS> which would start the additional slaves. In this scenario if one of the scaled slaves fails swarm would start a new one to bring the number of tasks back to 3.
https://docs.docker.com/engine/reference/builder/#healthcheck
Docker images have a new layer for health check.
Use a health check layer in your containers for example:
RUN ./anyscript.sh
HEALTHCHECK exit 1 or (Any command you want to add)
HEALTHCHECK check the status code of command 0 or 1 and than result as
1. healthy
2. unhealthy
3. starting etc.
Docker swarm auto restart the unhealthy containers in swarm cluster.
I have 3 projects, that deploys on different hosts. Every project have it's own RabbitMQ container. But I need to create cluster with this 3 hosts, using the same vhost, but different user/login pair.
I was tried Swarm and overlay networks, but swarm is aimed to run solo containers and with compose it doesn't work. Also, I was tried docker-compose bundle, but this is not work as expected :(
I assumed that it would work something like this:
1) On manager node I create overlay network
2) In every compose file I extend networks config for rabbitmq container with my overlay network.
3) They work as expected and I don't publish to Internet rabbitmq port.
Any idea, how can I do this?
Your approach is right, but Docker Compose doesn't work with Swarm Mode at the moment. Compose just runs docker commands, so you could script up what you want instead. For each project you'd have a script like this:
docker network create -d overlay app1-net
docker service create --network app1-net --name rabbit-app1 rabbitmq:3
docker service create --network app1-net --name app1 your-app-1-image
...
When you run all three scripts on the manager, you'll have three networks, each network will have its own RabbitMQ service (just 1 container by default, use --replicas to run more than one). Within the network other services can reach the message queue by the DNS name rabbit-appX. You don't need to publish any ports, so Rabbit is not accessible outside of the Docker network.