I use Ansible in order to provision a Docker container with Vagrant.
My Ansible file contains the following section to start nginx:
- name: nginx started
service:
name: nginx
state: restarted
From what I understand, this section should restart nginx in all cases, but when I connect via SSH to the container, nginx is not running (the whole Ansible provision process succeeds, there is no log for nginx in /var/log/nginx/error.log). It starts correctly when I manually type the following command: sudo service nginx start.
However, it works when I replace the section above by:
- name: nginx restarted
command: service nginx restart
It seems the issue is not limited to nginx, and also happens with other services like syslog-ng.
Any idea why using Ansible service does not work ? (docker 17.10.0, Vagrant 2.0.1, Ansible 2.4.0)
Ansible service module try to guess the underlying init system.
In your case of the phusion/baseimage docker image, it finds /sbin/initctl, so the module simply launch /sbin/initctl stop nginx; /sbin/initctl start nginx inside the container which does nothing as the init system is changed in this image (my_init).
So the problem is the inconsistent init system state of the image that ansible doesn't detect correctly.
The solutions are:
write a my_init ansible module (service module try first to use the {{ ansible_service_mgr }} module [code])
remove initctl from the image, so ansible will not detect any init system and will use the service command (maybe raise an issue in phusion/baseimage-docker)
use the command module to explicitly use service command as you finally did
Please take a look into this ansible issue I think it may be related.
According to them:
...this is not service specific. The restarted/reloaded options are
currently simple options that call the underlying init systems
scripts, which (if successful) return success...
So, probably that's why even in Ansible documentation you can find an example of what you were trying to achieve:
# Example action to restart service httpd, in all cases
- service:
name: httpd
state: restarted
But for nginx doesn't work as it may not be supported.
Appart from your solution:
- name: nginx restarted
command: service nginx restart
You can achieve that by using:
- name: restart nginx
service: name=nginx state=restarted
Related
I built my container image, but when I try to deploy it from the gcloud command line or the Cloud Console, I get the following error: "Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable."
In your code, you probably aren't listening for incoming HTTP requests, or you're listening for incoming requests on the wrong port.
As documented in the Cloud Run container runtime contract, your container must listen for incoming HTTP requests on the port that is defined by Cloud Run and provided in the $PORT environment variable.
If your container fails to listen on the expected port, the revision health check will fail, the revision will be in an error state and the traffic will not be routed to it.
For example, in Node.js with Express, you should use :
const port = process.env.PORT || 8080;
app.listen(port, () => {
console.log('Hello world listening on port', port);
});
In Go:
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
In python:
app.run(port=int(os.environ.get("PORT", 8080)),host='0.0.0.0',debug=True)
One of the other reason may be the one which I observed. Docker images may not have the required code to run the application.
I had a Node application written in TypeScript. In order to dockerize the application all I need to do is compile the code tsc and run docker build but I though that gcloud builds submit will be taking care of that and picking the compiled code as the Dockerfile suggested in conjunction to the .dockerignore and will build my source code and submit to the repository.
But what all it did was to copy my source code and submitted to the Cloud Build and there as per the Dockerfile it dockerized my source code as compared to dockerizing the compiled code.
So remember to include a build step in Dockerfile if you are doing a source code in a language with require compilation.
Remember that enabling the build step in the Dockerfile will increase the image size every time you do a image push to the repository. It is eating the space over there and google is going to charge you for that.
Another possibility is that the docker image ends with a command that takes time to complete. By the time deployment starts the server is not yet running and the health check will hit a blank.
What kind of command would that be ? Usually any command that runs the server in dev mode. For Scala/SBT it would be sbt run or in Node it would be something like npm run dev. In short make sure to run only on the packaged build.
I was exposing a PORT in dockerfile , remove that automatically fixed my problem. Google injects PORT env variable so the project will pick up that Env variable.
We can also specify the port number used by the image from the command line.
If we are using Cloud Run, we can use the following:
gcloud run deploy --image gcr.io/<PROJECT_ID>/<APP_NAME>:<APP_VERSION> --max-instances=3 --port <PORT_NO>
Where
<PROJECT_ID> is the project ID
<APP_NAME> is the app name
<APP_VERSION> is the app version
<PORT_NO> is the port number
The Cloud Run is generating default yaml file which has hard-coded default port in it:
spec:
containerConcurrency: 80
timeoutSeconds: 300
containers:
- image: us.gcr.io/project-test/express-image:1.0
ports:
- name: http1
containerPort: 8080
resources:
limits:
memory: 256Mi
cpu: 1000m
So, we need to expose the same 8080 port or change the containerPort in yaml file and redeploy.
Here is more about that:
A possible solution could be:
build locally
push the image on google cloud
deploy on google run
With commands:
docker build -t gcr.io/project-name/image-name
docker push gcr.io/project-name/image-name
gcloud run deploy tag-name --image gcr.io/project-name/image-name
When creating a Jenkins Docker container, it is very useful to able to connect to the Docker daemon. In that way, I can start docker commands inside the Jenkins container.
For example, after starting the Jenkins Docker container, I would like to 'docker exec -it container-id bash' and start 'docker ps'.
On Linux you can use bind-mounts on /var/run/docker.sock. On Windows this seems not possible. The solution is by using 'named pipes'. So, in my docker-compose.yml file I tried to create a named pipe.
version: '2'
services:
jenkins:
image: jenkins-docker
build:
context: ./
dockerfile: Dockerfile_docker
ports:
- "8080:8080"
- "50000:50000"
networks:
- jenkins
volumes:
- jenkins_home:/var/jenkins_home
- \\.\pipe\docker_engine:\\.\pipe\docker_engine
# - /var/run/docker.sock:/var/run/docker.sock
# - /path/to/postgresql/data:/var/run/postgresql/data
# - etc.
Starting docker-compose with this file, I get the following error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is
the docker daemon running?
How can I setup the docker-compose file so that I can use the docker.sock (or Docker) inside the started container?
On Linux you can use something like volumes: /var/run/docker.sock:/var/run/docker.sock. This does not work in a Windows environment. When you add this folder (/var) to Oracle VM Virtualbox, it won't get any IP forever. And on many posts
You can expose the daemon on tcp://localhost:2375 without TLS in the settings. This way you can configure Jenkins to use the Docker API instead of the socket. I encourage you to read this article by Nick Janetakis about "Understanding how the Docker Daemon and the Docker CLI work together".
And then there are several Docker plugins for Jenkins that allows this connection:
Also, you can find additional information in the Docker plugin documentation on wiki.jenkins.io:
def dockerCloudParameters = [
connectTimeout: 3,
containerCapStr: '4',
credentialsId: '',
dockerHostname: '',
name: 'docker.local',
readTimeout: 60,
serverUrl: 'unix:///var/run/docker.sock', // <-- Replace here by the tcp address
version: ''
]
EDIT 1:
I don't know if it is useful, but the Docker Daemon on Windows is located to C:\ProgramData\docker according to the Docker Daemon configuration doc.
EDIT 2:
You need to say explicitly the container to use the host network because you want to expose both Jenkins and Docker API.
Following this documentation, you only have to add --network=host (or network_mode: 'host' in docker-compose) to your container/service. For further information, you can read this article to understand what is the purpose of this network mode.
First try was to start a Docker environment using "Docker Quickstart terminal". This is a good solution when running Docker commands within that environment.
When installing a complete CI/CD Jenkins environment via Docker means that WITHIN the Jenkins Docker container you need to access the Docker daemon. After trying many solutions, reading many posts, this did not work. #Paul Rey, thank you very much for trying all kinds of routes.
A good solution is to get an Ubuntu Virtual Machine and install it via the Oracle VM Virtualbox. It is then VERY IMPORTANT to install Docker via this official description.
Before installing Docker, of course you need to install Curl, Git, etc.
Is it possible to execute commands on a docker swarm cluster hosted in cloud from my local mac? If yes, how?
I want to execute command such as following on docker swarm from my local:
docker create secret my-secret <address to local file>
docker service create --name x --secrets my-secret image
Answer to the question can be found here.
What one needs to do for ubuntu machine is define daemon.json file at path /etc/docker with following content:
{
"hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]
}
The above configuration is unsecured and shouldn't be used if server is publicly hosted.
For secured connection use following config:
{
"tls": true,
"tlscert": "/var/docker/server.pem",
"tlskey": "/var/docker/serverkey.pem",
"hosts": ["tcp://x.x.x.y:2376", "unix:///var/run/docker.sock"]
}
Details for generating certificate can be found here as mentioned by #BMitch.
One option is to provide direct access to the docker daemon as suggested in the previous answers, but that requires setting up TLS certificates and keys, which can itself be tricky and time consuming. Docker machine can automate that process, when docker machine has been used to create the nodes.
I had the same problem, in that I wanted to create secrets on the swarm without uploading the file containing the secret to the swarm manager. Also, I wanted to be able to run the deploy stackfile (e.g. docker-compose.yml) without the hassle of first uploading the stackfile.
I wanted to be able to create the few servers I needed on e.g. DigitalOcean, not necessarily using docker machine, and be able to reproducibly create the secrets and run the stackfile. In environments like DigitalOcean and AWS, a separate set of TLS certificates is not used, but rather the ssh key on the local machine is used to access the remote node over ssh.
The solution that worked for me was to run the docker commands using individual commands over ssh, which allows me to pipe the secret and/or stackfile using stdin.
To do this, you first need to create the DigitalOcean droplets and get docker installed on them, possibly from a custom image or snapshot, or simply running the commands to install docker on each droplet. Then, join the droplets into a swarm: ssh into the one that will be the manager node, type docker swarm init (possibly with the --advertise-addr option if there is more than one IP on that node, such as when you want to keep intra-swarm traffic on the private network) and get back the join command for the swarm. Then ssh into each of the other nodes and issue the join command, and your swarm is created.
Then, export the ssh command you will need to issue commands on the manager node, like
export SSH_CMD='ssh root#159.89.98.121'
Now, you have a couple of options. You can issue individual docker commands like:
$SSH_CMD docker service ls
You can create a secret on your swarm without copying the secret file to the swarm manager:
$SSH_CMD docker create secret my-secret - < /path/to/local/file
$SSH_CMD docker service create --name x --secrets my-secret image
(Using - to indicate that docker create secret should accept the secret on stdin, and then piping the file to stdin using ssh)
You can also create a script to be able to reproducibly run commands to create your secrets and bring up your stack with secret files and stackfiles only on your local machine. Such a script might be:
$SSH_CMD docker secret create rabbitmq.config.01 - < rabbitmq/rabbitmq.config
$SSH_CMD docker secret create enabled_plugins.01 - < rabbitmq/enabled_plugins
$SSH_CMD docker secret create rmq_cacert.pem.01 - < rabbitmq/cacert.pem
$SSH_CMD docker secret create rmq_cert.pem.01 - < rabbitmq/cert.pem
$SSH_CMD docker secret create rmq_key.pem.01 - < rabbitmq/key.pem
$SSH_CMD docker stack up -c - rabbitmq_stack < rabbitmq.yml
where secrets are used for the certs and keys, and also for the configuration files rabbitmq.config and enabled_plugins, and the stackfile is rabbitmq.yml, which could be:
version: '3.1'
services:
rabbitmq:
image: rabbitmq
secrets:
- source: rabbitmq.config.01
target: /etc/rabbitmq/rabbitmq.config
- source: enabled_plugins.01
target: /etc/rabbitmq/enabled_plugins
- source: rmq_cacert.pem.01
target: /run/secrets/rmq_cacert.pem
- source: rmq_cert.pem.01
target: /run/secrets/rmq_cert.pem
- source: rmq_key.pem.01
target: /run/secrets/rmq_key.pem
ports:
# stomp, ssl:
- 61614:61614
# amqp, ssl:
- 5671:5671
# monitoring, ssl:
- 15671:15671
# monitoring, non ssl:
- 15672:15672
# nginx here is only to show another service in the stackfile
nginx:
image: nginx
ports:
- 80:80
secrets:
rabbitmq.config.01:
external: true
rmq_cacert.pem.01:
external: true
rmq_cert.pem.01:
external: true
rmq_key.pem.01:
external: true
enabled_plugins.01:
external: true
(Here, the rabbitmq.config file sets up the ssh listening ports for stomp, amqp, and the monitoring interface, and tells rabbitmq to look for the certs and key within /run/secrets. Another alternative for this specific image would be to use the environment variables provided by the image to point to the secrets files, but I wanted a more generic solution that did not require configuration within the image)
Now, if you want to bring up another swarm, your script will work with that swarm once you have set the SSH_CMD environment variable, and you need neither set up TLS nor copy your secret or stackfiles to the swarm filesystem.
So, this doesn't solve the problem of creating the swarm (whose existence was presupposed by your question), but once it is created, using an environment variable (exported if you want to use it in scripts) will allow you to use almost exactly the commands you listed, prefixed with that environment variable.
This is the easiest way of running commands on remote docker engine:
docker context create --docker host=ssh://myuser#myremote myremote
docker --context myremote ps -a
docker --context myremote create secret my-secret <address to local file>
docker --context myremote service create --name x --secrets my-secret image
or
docker --host ssh://myuser#myremote ps -a
You can even set the remote context as default and issue commands as if it is local:
docker context use myremote
docker ps # lists remote running containers
In this case you don't even need to have docker engine installed, just docker-ce-cli.
You need to use key based authentication for this do work (you should already be using it). Other options include setting up tls cert based socket, or ssh tunnels.
Also, consider setting up ssh control socket to avoid re-authenting on each command, so your commands will run faster, as it was local.
To connect to a remote docker node, you should setup TLS on both the docker host and client signed from the same CA. Take care to limit what keys you sign with this CA since it is used to control access to the docker host.
Docker has documented the steps to setup a CA and create/install the keys here: https://docs.docker.com/engine/security/https/
Once configured, you can connect to the newer swarm mode environments using the same docker commands you run locally on the docker host just by changing the value of $DOCKER_HOST in your shell.
If you start from scratch, you can create the manager node using a generic docker-machine driver. Afterwards you will be able to connect to that docker engine from your local machine with the help of docker-machine env command.
I'm trying to make a Dockerfile based on the RabbitMQ repository with a customized policy set. The problem is that I can't useCMD or ENTRYPOINT since it will override the base Dockerfile's and then I have to come up with my own and I don't want to go down that path. Let alone the fact if I don't use RUN, it will be a part of run time commands and I want this to be included in the image, not just the container.
Other thing I can do is to use RUN command but the problem with that is the RabbitMQ server is not running at build time and also there's no --offline flag for the set_policycommand of rabbitmqctl program.
When I use docker's RUN command to set the policy, here's the error I face:
Error: unable to connect to node rabbit#e06f5a03fe1f: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#e06f5a03fe1f]
rabbit#e06f5a03fe1f:
* connected to epmd (port 4369) on e06f5a03fe1f
* epmd reports: node 'rabbit' not running at all
no other nodes on e06f5a03fe1f
* suggestion: start the node
current node details:
- node name: 'rabbitmq-cli-136#e06f5a03fe1f'
- home dir: /var/lib/rabbitmq
- cookie hash: /Rw7u05NmU/ZMNV+F856Fg==
So is there any way I can set a policy for the RabbitMQ without writing my own version of CMD and/or ENTRYPOINT?
You're in a slightly tricky situation with RabbitMQ as it's mnesia data path is based on the host name of the container.
root#bf97c82990aa:/# ls -1 /var/lib/rabbitmq/mnesia
rabbit#bf97c82990aa
rabbit#bf97c82990aa-plugins-expand
rabbit#bf97c82990aa.pid
For other image builds you could seed the data files, or write a script that RUN calls to launch the application or database and configure it. With RabbitMQ, the container host name will change between image build and runtime so the image's config won't be picked up.
I think you are stuck with doing the config on container creation or at startup time.
Options
Creating a wrapper CMD script to do the policy after startup is a bit complex as /usr/lib/rabbitmq/bin/rabbitmq-server runs rabbit in the foreground, which means you don't have access to an "after startup" point. Docker doesn't really do background processes so rabbitmq-server -detached isn't much help.
If you were to use something like Ansible, Chef or Puppet to setup the containers. Configure a fixed hostname for the containers startup. Then start it up and configure the policy as the next step. This only needs to be done once, as long as the hostname is fixed and you are not using the --rm flag.
At runtime, systemd could complete the config to a service with ExecStartPost. I'm sure most service managers will have the same feature. I guess you could end up dropping messages, or at least causing errors at every start up if anything came in before configuration was finished?
You can configure the policy as described here.
Docker compose:
rabbitmq:
image: rabbitmq:3.7.8-management
container_name: rabbitmq
volumes:
- ~/rabbitmq/data:/var/lib/rabbitmq:rw
- ./rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
- ./rabbitmq/definitions.json:/etc/rabbitmq/definitions.json
ports:
- "5672:5672"
- "15672:15672"
I am in the process of writing my first ever Ansible playbook and am in need of a bit of steering. I have a simple network that consists of 3 VMs:
ansible01 - my Ansible server (Ubuntu)
db01 - a DB (again, Ubuntu)
myapp01 - an Ubuntu VM hosting a Java app
I have configured my /etc/ansible/hosts file like so:
[databases]
db01.example.com
[app_servers]
myapp01.example.com
myapp02.example.com
I have configured SSH correctly, and I can run ansible all ping -m and Ansible is able to ping the DB and app server nodes. So far so good.
I’m trying to write three (3) Docker-related playbooks that will accomplish the following:
Ensure that Docker is running on all [databases] nodes as well as all [app_servers] nodes; if it is not installed and running, then install Docker engine and start running it. If it is installed but not running, restart it.
Stop/start/restart all containers running for a specific type of node (“role"?!?). For instance, I’d like to tell Ansible that I want to restart all containers running on all [app_servers] nodes.
Stop/start/restart an arbitrary container running on an arbitrary node. For instance, perhaps myapp01 has 2 containers running on it, fizz and buzz. I’d like to be able to tell Ansible to restart (specifically) myapp01’s fizz container, but not its buzz container, nor any myapp02 containers.
I believe these belong in three separate playbooks (correct me if I’m wrong or if there’s a better way). I took a stab at them. The first is setup_docker.yml:
- name: ensure docker engine is installed and running
docker:
name: *
state: started
Then for restarting all [databases], in restart_app_servers.yml:
- name: restart app servers
docker:
name: app_servers
state: restarted
And for restarting an arbitrary container on a single node (restart_container.yml):
- name: restart a specific container
docker:
name: %name_of_container_and node%
state: restarted
But several problems here:
In setup_docker.yml, how do I specify that all node types ([databases] and [app_servers]) should be affected? I know that asterisk (“*”) isn’t correct.
In restart_app_servers.yml, what is the proper value for the name field? How do I actually tell Ansible to restart all app_server nodes?
In restart_container.yml, how do I “inject” (pass in as arguments/variables) the node's and container’s names? Ideally I’d like to run this playbook against any node and any container.
Anything else jumping out at you as wrong?
Thanks in advance!
I think you have Plays and Playbooks mixed up in meaning here. The three things you have specified above, setup_docker.yml, restart_app_servers.yml, and restart_container.yml appear to be Plays. I recommend creating a Docker role which contains the tasks you have detailed here.
To address your problems:
In setup_docker.yml, how do I specify that all node types ([databases] and [app_servers]) should be affected? I know that asterisk (“*”) isn’t correct.
This is done at the Playbook level. You can specify which hosts you want to be effected by which tasks, e.g:
#docker.yml
- hosts: all
user: {{ privileged_user }}
gather_facts: false
roles:
- install_docker
Then in your install_docker role, you would have something along the lines of:
- name: Add docker apt keys
apt_key: keyserver=keyserver.ubuntu.com id=36A1D7869245C8950F966E92D8576A8BA88D21E9
- name: update apt
apt_repository: repo='deb https://get.docker.com/ubuntu docker main' state=present
- name: Install docker
apt: pkg=lxc-docker update_cache=yes
In restart_app_servers.yml, what is the proper value for the name field? How do I actually tell Ansible to restart all app_server nodes?
I'm assuming you mean you wish to restart all Docker containers on each of the nodes which belong to the app-server group?
I would keep an inventory of all of the container names for each group (since this example is relatively simple). e.g:
#group_vars/app-server
all_containers: [ 'container1', 'container2', 'container3',.. 'containern' ]
From here you can use this inventory in your Play to restart each container. In your Playbook:
#restart_app_containers.yml
- hosts: app_server
user: {{ privileged_user }}
gather_facts: false
roles:
- restart_app_servers
Then in the Play itself:
#restart_app_servers.yml
- name: restart app servers
docker:
name: {{ item }}
state: restarted
with_items: all_containers
In restart_container.yml, how do I “inject” (pass in as arguments/variables) the node's and container’s names? Ideally I’d like to run this playbook against any node and any container.
For this portion you would need to reference your container directly which you need to act against. This can be done with Dynamic Inventory, e.g
#sample.yml
- hosts: Tag_name_{{ public_name }}
user: {{ privileged_user }}
gather_facts: false
roles:
- example
In the event you are on AWS. The hosts dictionary would vary by infrastructure.
Then in your actual play you listed, you can pass in the specific variable. Since it's a single container on a single host, you could do this via the command line:
ansible-playbook -i $INVENTORY_FILE -e container_name=$CONTAINER_NAME restart_single_container_on_single_host.yml
Where your Play would look something like:
- name: restart a specific container
docker:
name: {{ container_name }}
state: restarted