Managing Docker containers from Ansible plays - docker

I am in the process of writing my first ever Ansible playbook and am in need of a bit of steering. I have a simple network that consists of 3 VMs:
ansible01 - my Ansible server (Ubuntu)
db01 - a DB (again, Ubuntu)
myapp01 - an Ubuntu VM hosting a Java app
I have configured my /etc/ansible/hosts file like so:
[databases]
db01.example.com
[app_servers]
myapp01.example.com
myapp02.example.com
I have configured SSH correctly, and I can run ansible all ping -m and Ansible is able to ping the DB and app server nodes. So far so good.
I’m trying to write three (3) Docker-related playbooks that will accomplish the following:
Ensure that Docker is running on all [databases] nodes as well as all [app_servers] nodes; if it is not installed and running, then install Docker engine and start running it. If it is installed but not running, restart it.
Stop/start/restart all containers running for a specific type of node (“role"?!?). For instance, I’d like to tell Ansible that I want to restart all containers running on all [app_servers] nodes.
Stop/start/restart an arbitrary container running on an arbitrary node. For instance, perhaps myapp01 has 2 containers running on it, fizz and buzz. I’d like to be able to tell Ansible to restart (specifically) myapp01’s fizz container, but not its buzz container, nor any myapp02 containers.
I believe these belong in three separate playbooks (correct me if I’m wrong or if there’s a better way). I took a stab at them. The first is setup_docker.yml:
- name: ensure docker engine is installed and running
docker:
name: *
state: started
Then for restarting all [databases], in restart_app_servers.yml:
- name: restart app servers
docker:
name: app_servers
state: restarted
And for restarting an arbitrary container on a single node (restart_container.yml):
- name: restart a specific container
docker:
name: %name_of_container_and node%
state: restarted
But several problems here:
In setup_docker.yml, how do I specify that all node types ([databases] and [app_servers]) should be affected? I know that asterisk (“*”) isn’t correct.
In restart_app_servers.yml, what is the proper value for the name field? How do I actually tell Ansible to restart all app_server nodes?
In restart_container.yml, how do I “inject” (pass in as arguments/variables) the node's and container’s names? Ideally I’d like to run this playbook against any node and any container.
Anything else jumping out at you as wrong?
Thanks in advance!

I think you have Plays and Playbooks mixed up in meaning here. The three things you have specified above, setup_docker.yml, restart_app_servers.yml, and restart_container.yml appear to be Plays. I recommend creating a Docker role which contains the tasks you have detailed here.
To address your problems:
In setup_docker.yml, how do I specify that all node types ([databases] and [app_servers]) should be affected? I know that asterisk (“*”) isn’t correct.
This is done at the Playbook level. You can specify which hosts you want to be effected by which tasks, e.g:
#docker.yml
- hosts: all
user: {{ privileged_user }}
gather_facts: false
roles:
- install_docker
Then in your install_docker role, you would have something along the lines of:
- name: Add docker apt keys
apt_key: keyserver=keyserver.ubuntu.com id=36A1D7869245C8950F966E92D8576A8BA88D21E9
- name: update apt
apt_repository: repo='deb https://get.docker.com/ubuntu docker main' state=present
- name: Install docker
apt: pkg=lxc-docker update_cache=yes
In restart_app_servers.yml, what is the proper value for the name field? How do I actually tell Ansible to restart all app_server nodes?
I'm assuming you mean you wish to restart all Docker containers on each of the nodes which belong to the app-server group?
I would keep an inventory of all of the container names for each group (since this example is relatively simple). e.g:
#group_vars/app-server
all_containers: [ 'container1', 'container2', 'container3',.. 'containern' ]
From here you can use this inventory in your Play to restart each container. In your Playbook:
#restart_app_containers.yml
- hosts: app_server
user: {{ privileged_user }}
gather_facts: false
roles:
- restart_app_servers
Then in the Play itself:
#restart_app_servers.yml
- name: restart app servers
docker:
name: {{ item }}
state: restarted
with_items: all_containers
In restart_container.yml, how do I “inject” (pass in as arguments/variables) the node's and container’s names? Ideally I’d like to run this playbook against any node and any container.
For this portion you would need to reference your container directly which you need to act against. This can be done with Dynamic Inventory, e.g
#sample.yml
- hosts: Tag_name_{{ public_name }}
user: {{ privileged_user }}
gather_facts: false
roles:
- example
In the event you are on AWS. The hosts dictionary would vary by infrastructure.
Then in your actual play you listed, you can pass in the specific variable. Since it's a single container on a single host, you could do this via the command line:
ansible-playbook -i $INVENTORY_FILE -e container_name=$CONTAINER_NAME restart_single_container_on_single_host.yml
Where your Play would look something like:
- name: restart a specific container
docker:
name: {{ container_name }}
state: restarted

Related

Migrating Docker Compose to Kubernetes Volume Mount Isn't Supported

I am trying to change my existing deployment logic/switch to kubernetes (My server is in gcp and till now I used docker-compose to run my server.) So I decided to start by using kompose and generating services/deployments using my existing docker-compose file. After running
kompose --file docker-compose.yml convert
#I got warnings indicating Volume mount on the host "mypath" isn't supported - ignoring path on the host
After a little research I decided to use the command below to "fix" the issue
kompose convert --volumes hostPath
And what this command achieved is -> It replaced the persistent volume claims that were generated with the first command to the code below.
volumeMounts:
- mountPath: /path
name: certbot-hostpath0
- mountPath: /somepath
name: certbot-hostpath1
- mountPath: /someotherpath
name: certbot-hostpath2
- hostPath:
path: /path/certbot
name: certbot-hostpath0
- hostPath:
path: /path/cert_challenge
name: certbot-hostpath1
- hostPath:
path: /path/certs
name: certbot-hostpath2
But since I am working in my local machine
kubectl apply -f <output file>
results in The connection to the server localhost:8080 was refused - did you specify the right host or port?
I didn't want to connect my local env with gcp just to generate the necessary files, is this a must? Or can I move this to startup-gcp etc
I feel like I am in the right direction but I need a confirmation that I am not messing something up.
1)I have only one compute engine(VM instance) and lots of data in my prod db. "How do I"/"do I need to" make sure I don't lose any data in db by doing something?
2)In startup-gcp after doing everything else (pruning docker images etc) I had a docker run command that makes use of docker/compose 1.13.0 up -d. How should I change it to switch to kubernetes?
3)Should I change anything in nginx.conf as it referenced to 2 different services in my docker-compose (I don't think I should since same services also exist in kubernetes generated yamls)
You should consider using Persistent Volume Claims (PVCs). If your cluster is managed, in most cases it can automatically create the PersistentVolumes for you.
One way to create the Persistent Volume Claims corresponding to your docker compose files is using Move2Kube(https://github.com/konveyor/move2kube). You can download the release and place it in path and run :
move2kube translate -s <path to your docker compose files>
It will then interactively allow you configure PVCs.
If you had a specific cluster you are targeting, you can get the specific storage classes supported by that cluster using the below command in a terminal where you have set your kubernetes cluster as context for kubectl.
move2kube collect
Once you do collect, you will have a m2k_collect folder, which you can then place it in the folder where your docker compose files are. And when you run move2kube translate it will automatically ask you whether to target this specific cluster, and also option to choose the storage class from that cluster.
1)I have only one compute engine(VM instance) and lots of data in my
prod db. "How do I"/"do I need to" make sure I don't lose any data in
db by doing something?
Once the PVC is provisioned you can copy the data to the PVC by using kubectl cp command into a pod where the pvc is mounted.
2)In startup-gcp after doing everything else (pruning docker images
etc) I had a docker run command that makes use of docker/compose
1.13.0 up -d. How should I change it to switch to kubernetes?
You can potentially change it to use helm chart. Move2Kube, during the interactive session, can help you create helm chart too. Once you have the helm chart, all you have to do is "helm upgrade -i
3)Should I change anything in nginx.conf as it referenced to 2
different services in my docker-compose (I don't think I should since
same services also exist in kubernetes generated yamls)
If the services names are name in most cases it should work.

How to close docker environment from the outside world?

I have defined the following Jelastic configuration for my environment:
env:
topology:
nodes:
- nodeGroup: bl
nodeType: nginx-dockerized
tag: 1.14.2
displayName: Node balancing
count: 1
fixedCloudlets: 1
cloudlets: 4
env:
DOCKER_EXPOSED_PORT: 22,80,443
- image: jenkins/jenkins:lts
count: 1
cloudlets: 16
nodeGroup: cp
- nodeGroup: sqldb
Now, I want the users of my environment to access my docker application only through the load balancing node. From Jelastic's dashboard, I can't seem to configure any firewall rules for the cp node group. How can I close any connection to the Jenkins node from the outside world and only keep it open from the nginx node?
As we can see, your environment has a custom Docker as a cp layer, that's why UI Firewall is not available for your cp node group. More details are in Container Firewall Rules Management article.
Nevertheless, even if your cp layer is not a custom Docker but Jelastic certified dockerized template, UI Firewall will be available but you will not be able to close direct access to this node via Shared Load Balancer anyway, due to internal limitations which will be improved in future releases. Some info you can find here
Custom Docker, unlike other types of templates, has a full root access, so you can easily configure firewall with help of a command line.

Access Kubernetes pod's log files from inside the pod?

I'm currently migrating a legacy server to Kubernetes, and I found that kubectl or dashboard only shows the latest log file, not the older versions. In order to access the old files, I have to ssh to the node machine and search for it.
In addition to being a hassle, my team wants to restrict access to the node machines themselves, because they will be running pods from many different teams and unrestricted access could be a security issue.
So my question is: can I configure Kubernetes (or a Docker image) so that these old (rotated) log files are stored in some directory accessible from inside the pod itself?
Of course, in a pinch, I could probably just execute something like run_server.sh | tee /var/log/my-own.log when the pod starts... but then, to do it correctly, I'll have to add the whole logfile rotation functionality, basically duplicating what Kubernetes is already doing.
So there are a couple of ways to and scenarios for this. If you are just interested in the log of the same pod from before last restart, you can use the --previous flag to look at logs:
kubectl logs -f <pod-name-xyz> --previous
But since in your case, you are interested in looking at log files beyond one rotation, here is how you can do it. Add a sidecar container to your application container:
volumeMounts:
- name: varlog
mountPath: /tmp/logs
- name: log-helper
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/*.log']
volumeMounts:
- name: varlog
mountPath: /tmp/logs
volumes:
- name: varlog
hpostPath: /var/log
This will allow the directory which has all logs from /var/log directory from host to /tmp/log inside the container and the command will ensure that content of all files is flushed. Now you can run:
kubectl logs <pod-name-abc> -c count-log-1
This solution does away with SSH access, but still needs access to kubectl and adding a sidecar container. I still think this is a bad solution and you consider of one of the options from the cluster level logging architecture documentation of Kubernetes such as 1 or 2

Ansible service restart does not work

I use Ansible in order to provision a Docker container with Vagrant.
My Ansible file contains the following section to start nginx:
- name: nginx started
service:
name: nginx
state: restarted
From what I understand, this section should restart nginx in all cases, but when I connect via SSH to the container, nginx is not running (the whole Ansible provision process succeeds, there is no log for nginx in /var/log/nginx/error.log). It starts correctly when I manually type the following command: sudo service nginx start.
However, it works when I replace the section above by:
- name: nginx restarted
command: service nginx restart
It seems the issue is not limited to nginx, and also happens with other services like syslog-ng.
Any idea why using Ansible service does not work ? (docker 17.10.0, Vagrant 2.0.1, Ansible 2.4.0)
Ansible service module try to guess the underlying init system.
In your case of the phusion/baseimage docker image, it finds /sbin/initctl, so the module simply launch /sbin/initctl stop nginx; /sbin/initctl start nginx inside the container which does nothing as the init system is changed in this image (my_init).
So the problem is the inconsistent init system state of the image that ansible doesn't detect correctly.
The solutions are:
write a my_init ansible module (service module try first to use the {{ ansible_service_mgr }} module [code])
remove initctl from the image, so ansible will not detect any init system and will use the service command (maybe raise an issue in phusion/baseimage-docker)
use the command module to explicitly use service command as you finally did
Please take a look into this ansible issue I think it may be related.
According to them:
...this is not service specific. The restarted/reloaded options are
currently simple options that call the underlying init systems
scripts, which (if successful) return success...
So, probably that's why even in Ansible documentation you can find an example of what you were trying to achieve:
# Example action to restart service httpd, in all cases
- service:
name: httpd
state: restarted
But for nginx doesn't work as it may not be supported.
Appart from your solution:
- name: nginx restarted
command: service nginx restart
You can achieve that by using:
- name: restart nginx
service: name=nginx state=restarted

Setting a policy for RabbitMQ as a part of Dockerfile process

I'm trying to make a Dockerfile based on the RabbitMQ repository with a customized policy set. The problem is that I can't useCMD or ENTRYPOINT since it will override the base Dockerfile's and then I have to come up with my own and I don't want to go down that path. Let alone the fact if I don't use RUN, it will be a part of run time commands and I want this to be included in the image, not just the container.
Other thing I can do is to use RUN command but the problem with that is the RabbitMQ server is not running at build time and also there's no --offline flag for the set_policycommand of rabbitmqctl program.
When I use docker's RUN command to set the policy, here's the error I face:
Error: unable to connect to node rabbit#e06f5a03fe1f: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#e06f5a03fe1f]
rabbit#e06f5a03fe1f:
* connected to epmd (port 4369) on e06f5a03fe1f
* epmd reports: node 'rabbit' not running at all
no other nodes on e06f5a03fe1f
* suggestion: start the node
current node details:
- node name: 'rabbitmq-cli-136#e06f5a03fe1f'
- home dir: /var/lib/rabbitmq
- cookie hash: /Rw7u05NmU/ZMNV+F856Fg==
So is there any way I can set a policy for the RabbitMQ without writing my own version of CMD and/or ENTRYPOINT?
You're in a slightly tricky situation with RabbitMQ as it's mnesia data path is based on the host name of the container.
root#bf97c82990aa:/# ls -1 /var/lib/rabbitmq/mnesia
rabbit#bf97c82990aa
rabbit#bf97c82990aa-plugins-expand
rabbit#bf97c82990aa.pid
For other image builds you could seed the data files, or write a script that RUN calls to launch the application or database and configure it. With RabbitMQ, the container host name will change between image build and runtime so the image's config won't be picked up.
I think you are stuck with doing the config on container creation or at startup time.
Options
Creating a wrapper CMD script to do the policy after startup is a bit complex as /usr/lib/rabbitmq/bin/rabbitmq-server runs rabbit in the foreground, which means you don't have access to an "after startup" point. Docker doesn't really do background processes so rabbitmq-server -detached isn't much help.
If you were to use something like Ansible, Chef or Puppet to setup the containers. Configure a fixed hostname for the containers startup. Then start it up and configure the policy as the next step. This only needs to be done once, as long as the hostname is fixed and you are not using the --rm flag.
At runtime, systemd could complete the config to a service with ExecStartPost. I'm sure most service managers will have the same feature. I guess you could end up dropping messages, or at least causing errors at every start up if anything came in before configuration was finished?
You can configure the policy as described here.
Docker compose:
rabbitmq:
image: rabbitmq:3.7.8-management
container_name: rabbitmq
volumes:
- ~/rabbitmq/data:/var/lib/rabbitmq:rw
- ./rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
- ./rabbitmq/definitions.json:/etc/rabbitmq/definitions.json
ports:
- "5672:5672"
- "15672:15672"

Resources