I have a docker swarm setup with 4 nodes and deployed Pumba in global mode to the swarm setup. After that, I have run my application containers on the swarm with replicas on different nodes. I want to send kill or netem command to all the Pumba containers in all the nodes.
Right now the only way I am able to do it is either by specifying the command when creating the service:
docker service create --name PUMBA --mode=global --mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock gaiaadm/pumba:master pumba --random --interval 10s kill re2:"^customer-api*" --signal SIGTERM
Or by going in each host and doing a exec to the container and running the command:
docker exec -i $(docker ps| grep pumba|cut -d 'g' -f1 ) pumba netem --duration 60s delay --time 3000 --jitter 40 --distribution normal re2:"^${name[i]}*" > /dev/null 2>&1 &
I am creating a bash script for this. Is there a way I could pass the command to the service so that it's reflected in all of its global replicas?
I am able to send a new command to Pumba container using the following command
docker service update PUMBA --args "pumba --random --interval 5s kill re2:"^customer*" --signal SIGTERM"
Related
I have implemented docker native health check by adding HEALTHCHECK command in Docker file as shown below,
HEALTHCHECK --interval=60s --timeout=15s --retries=3 CMD ["/svc/app/healthcheck/healthCheck.sh"]
set the entry point for the container
CMD [".././run.sh"]
executing the docker run command as shown below,
docker run -d --net=host --pid=host --publish-all=true -p 7000:7000/udp applicationname:temp
healthCheck.sh is exiting with 1, when my application is not up and I can see the container status as unhealthy, but it is not getting restarted.
STATUS
Up 45 minutes (unhealthy)
Below are the docker and OS details:
[root#localhost log]# docker -v
Docker version 18.09.7, build 2d0083d
OS version
NAME="CentOS Linux"
VERSION="7 (Core)"
How to restart my container automatically when it becomes unhealthy?
Docker only reports the status of the healthcheck. Acting on the healthcheck result requires an extra layer running on top of docker. Swarm mode provides this functionality and is shipped with the docker engine. To enable:
docker swarm init
Then instead of managing individual containers with docker run, you would declare your target state with docker service or docker stack commands and swarm mode will manage the containers to achieve the target state.
docker service create -d --net=host applicationname:temp
Note that host networking and publishing ports are incompatible (they make no logical sense together), net requires two dashes to be a valid flag, and changing the pid namespace is not supported in swarm mode. Many other features should work similar to docker run.
https://docs.docker.com/engine/reference/commandline/service_create/
There is no auto restart mechanism for unhealth container currently, see this, but you can make a workaround as mentioned here:
docker run -d \
--name autoheal \
--restart=always \
-e AUTOHEAL_CONTAINER_LABEL=all \
-v /var/run/docker.sock:/var/run/docker.sock \
willfarrell/autoheal
It add docker unix domain socket to the monitor container, then it could monitor all unhealthy container and restart it for you if other container is not healthy.
I am running single docker container on two different ports using below command
docker run -p ${EXTERNAL_PORT_NUMBER}:${INTERNAL_PORT_NUMBER} -p ${EXTERNAL_PORT_NUMBER_SECOND}:${INTERNAL_PORT_NUMBER_SECOND} --network ${NETWORK} --name ${SERVICE_NAME} --restart always -m 1024M --memory-swap -1 -itd ${ORGANISATION}/${SERVICE_NAME}:${VERSION}
I can see the container is running fine
My question is How can I see the logs of this docker container.
Every time I do sudo docker logs database-service -f I can see the log of container running on 9003 port only.
How can I view the logs of container running on 9113
You are getting all the logs that was displayed on stdout or stderr in the container.
It has nothing to do with the processes which are exposed on different ports.
If 2 instance is running inside the container and both are showing there logs on system console then you will be getting both logs on the docker logs command for the container.
You can try multitail utility to tail more than one log files in docker exec command.
For that you have to install it in that container.
You can bind external volumes to container service logs and see the logs
docker run -v 'path_to_you_host_logs':'container_service_log_path'
docker run -v 'home/user/app/apache_access.log':
'/var/log/apache_access.log'
I have two worker nodes: worker1 and worker2 and one swarm manager. I'm running all the services in the worker nodes only. I need to run from the manager docker exec to access some of the containers created in the worker nodes but I keep getting that the service is not recognized. I know I can run docker exec in any of the worker nodes and it works fine but I dont want to have to find on which node the service is running and then ssh to the designated node to run docker exec command. Is there a way to do so in swarm or not?
Swarm mode does not currently have a way to run an exec on a running task. You need to find the container and run the exec on the host. You can configure the workers to have a TLS protected port they listen on, which would give you remote access (see docker's guide). And you can lookup the node for each task in a service by checking the output of a docker service ps $service_name.
If this helps, nowadays you can create the overlay network with --attachable flag to enable any container to join the network. This is great feature as it allows a lot of flexibility.
E.g.
$ docker network create --attachable --driver overlay my-network
$ docker service create --network my-network --name web --publish 80:80 nginx
$ docker run --network=my-network -ti alpine sh
$ wget -qO- web
<!DOCTYPE html>
<html>
<head>
....
I am running Docker Swarm 1.12 on 3 CoreOS machines via Vagrant.
What is the best way to start a Solr cloud on the cluster?
Do I need Zookeeper?
I have gotten as far as this:
docker service create --mode=global --name solr -p 8983:8983 solr:5.3.1 bash -c "/opt/solr/bin/solr start -f -c"
But then the Cloud is empty because it does not know about the other 2 machines, how can I use Swarm's power here?
Documentation
The container image documentation describes how interconnect Zookeeper as a backing store for a distributed Solr setup:
You can also run a distributed Solr configuration, with Solr nodes in
separate containers, sharing a single ZooKeeper server:
Run ZooKeeper, and define a name so we can link to it:
$ docker run --name zookeeper -d -p 2181:2181 -p 2888:2888 -p 3888:3888 jplock/zookeeper
Run two Solr nodes, linked to the zookeeper container:
$ docker run --name solr1 --link zookeeper:ZK -d -p 8983:8983 \
solr \
bash -c '/opt/solr/bin/solr start -f -z $ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT'
$ docker run --name solr2 --link zookeeper:ZK -d -p 8984:8983 \
solr \
bash -c '/opt/solr/bin/solr start -f -z $ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT'
Running distributed Solr on Docker Swarm
Starting with a 3 node swarm:
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
2wkpmybf4wni8wia1s46lm2ml * node1 Ready Active Leader
ajrnf5oibgm7b12ayy0hg5i32 node3 Ready Active
bbe8n1hybhruhhrhmswn7fjmd node2 Ready Active
Create a network
$ docker network create --driver overlay my-net
Start a zookeeper service and wait for it to start
$ docker service create --name zookeeper --replicas 1 --network my-net jplock/zookeeper
Start 2 solr instances configured to connect to the DNS address "zookeeper". For more information on swarm mode networking you can read the documentation
$ docker service create --name solr --replicas 2 --network my-net -p 8983:8983 \
solr \
bash -c '/opt/solr/bin/solr start -f -z zookeeper:2181'
The web UI will be available on any node in the cluster
http://192.168.99.100:8983/
http://192.168.99.101:8983/
http://192.168.99.102:8983/
If you check the services you'll notice the containers are spread across all 3 nodes in the cluster. This is the default scheduling behaviour
$ docker service ps zookeeper
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
3fhipbsd4jdazmx8d7zum0ohp zookeeper.1 jplock/zookeeper node1 Running Running 7 minutes ago
$ docker service ps solr
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
bikggwpyz5q6vdxrpqwevlwsr solr.1 solr node2 Running Running 43 seconds ago
cutbmjsmcxrmi1ld75eox0s9m solr.2 solr node3 Running Running 43 seconds ago
I finished this documentation:
https://docs.docker.com/swarm/install-w-machine/
It works fine.
Now I tried to setup this EC2 instances by following this documentation:
https://docs.docker.com/swarm/install-manual/
I am in Step 4. Set up a discovery backend
I cannot understand the steps what I need to do further.
I created 5 nodes in EC2: manager0, manager1, consul0, node0, node1. Now I need to know how to setup service discovery with swarm.
In that document they ask us to connect manager0 and consul0 then ifconfig, then they given as etc0 instance. I don't know where this is coming from.
Ultimately I need to know where (in which node?) to run this command:
$ docker run -d -p 8500:8500 --name=consul progrium/consul -server -bootstrap
Any suggestion for me How to clear this step?
Consul will run on the consul0 server you created. So basically you first need to be able to run docker on worker0 and worker1 remotely, this is step 3. A better way of doing this is editing the daemon directly with the command:
sudo echo 'DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock"' > /etc/default/docker`
Then restart docker. Afterwards you will find that you can run docker remotely from master0, master1 or any other instance behind your firewall with docker commands that start with:
docker -H $WORKER0_IPADDRESS:2375
For example if your workers ip address was 1.2.3.4 this would run the docker ps command remotely:
docker -H 1.2.3.4:2375 ps
This is what swarm runs on. Then start up your consul server with the command you want to run, you got that one right and thats it you wont do anything else with the consul0 server except use its IP address when you run your swarm commands.
So if $CONSUL0 represented the IP address of your consul server this is how you would set up the rest of swarm. If you ran each of them on the local machine of each node:
On consul0:
docker run -d -p 8500:8500 --restart=unless-stopped --name=consul progrium/consul -server -bootstrap
On master0 and master1:
docker run --name=master -d -p 4000:4000 swarm manage -H :4000 --replication --advertise $(hostname -i):4000 consul://$CONSUL0:8500
On worker0 and worker1:
docker run -d --name=worker swarm join --advertise=$(hostname -i):2375 consul://$CONSUL0:8500/