I am migrating away from Docker Cloud to pure Docker Swarm setup. One thing that I am missing is the nice way of how the containers got the hostname set as $SERVICE_NAME-$SLOT_NUMBER.
Is it possible to dynamically set the hostname/container name in a swarm stack service?
In your stack/compose file, use this format:
services:
thaservice:
...
hostname: "{{.Service.Name}}-{{.Task.Slot}}"
Link to documentation for available template variables: https://docs.docker.com/engine/swarm/services/#create-services-using-templates
Related
I have been researching a little bit about docker compose.
From what I understand,
services:
api:
build: ./api
db:
image: <someimage>
With something like this(adding some other missing options here), I should be able to access the db container from the web container using 'db' as the hostname.
This works on my local machine. However, I would like to know if this will still work on something like an ECS cluster.
Do I need to make any further changes in the code itself?
Example -> I might have this as an env variable in my api:
DB_URL=mysql://admin:12345#db/mydb
Do I need to change anything when I deploy it to an ECS cluster or will docker compose take care of it?
I have seen people using links, and depends_on, but I am not quite clear on what it all does yet. I understand that depends_on just tells docker that it has to wait for another container to start up first, but links don't seem to do anything in local.
In my github actions workflow I am getting this error (ECONNREFUSED) while running my jest test script. The test uses axios to connect to my api which is running in a container bootstrapped via docker-compose (created during the github workflow itself). That network has just has 2 containers: the api, and postgres. So my test script is, I am assume, on the "host network" (github workflow), but it couldn't reach the docker network via the containers' mapped ports.
I then skipped jest test entirely and just tried to directly ping the containers. Didn't work.
I then modified the workflow to inspect the default docker network that should have been created:
UPDATE 1
I've narrowed down the issue as follows. When I modified the compose file to rely on the default network (i no longer have a networks: in my compose file):
So it looks as though the containers were never attached to the default bridge network.
UPDATE 2
It looks like I just have the wrong paradigm. After reading this: https://help.github.com/en/actions/configuring-and-managing-workflows/about-service-containers
I realise this is not how GA expects us to instantiate containers at all. Looks like I should be using services: nodes inside the workflow file, not using containers from my own docker-compose files. 🤔 Gonna try that...
So the answer is:
do not use docker-compose to build your own custom containers. GA does not support this yet.
Use services: in your workflow .yml file to launch your containers, which must be public docker images. If your container is based on a private image or custom dockerfile, it's not supported yet by GA.
So instead of "docker-compose up" to bootstrap postgres + my api for integration testing, I had to:
Create postgres as a service container in my github workflow .yml
Change my test command in package.json to:
first start the api as background process (because I can't create my own docker image from it 🙄) then
invoke my test framework next (as the foreground process)
so npm run start & npm run <test launch cmds>. This worked.
There are several possibilities here.
Host Network
Since you are using docker compose, when you start the api container, publish the endpoint that the api is listening on to the host machine. You can achieve this by doing:
version: 3
services:
api:
...
ports:
- "3010:3010"
in your docker-compose.yml. This will publish the ports, similar to doing docker run ... ---publish localhost:3010:3010. See reference here: https://docs.docker.com/compose/compose-file/#ports
Compose network
By default, docker-compose will create a network called backend-net_default. Containers created by this docker-compose.yml will have access to other containers via this network. The host name to access other containers on the network is simply the name of the service. For example, your tests could access the api endpoint using the host api (assuming that is the name of your api service), e.g.:
http://api:3010
The one caveat here is that the tests must be launched in a container that is managed by that same docker-compose.yml, so that it may access the common backend-net_default network.
There are a lot of applications which I launch on my workstation using docker-compose up.
Reasons:
They don't have an installer, or I don't want to use it
They require a dedicated storage engine to be present
They require a build process step
They are created by me and I want them to be easily launched on any workstation
e.t.c
So what I usually end up with the following file-structure:
myAppDir
- docker-compose.yml
- Dockerfile (not always)
- someConfigFile
And my docker-compose.yml is something like this:
(It can contain 2 or 3 services, but I provide the simplest form that I use)
version: '3.7'
services:
mysql:
image: mysql:5.7.29
restart: always
volumes:
- ./mysqld.cnf:/etc/mysql/mysql.conf.d/mysqld.cnf
environment:
- MYSQL_ROOT_PASSWORD=xyz
ports:
- 3306:3306
Then when I need to launch the application I just perform:
docker-compose up # (or with --build)
Recently I tried to add:
deploy:
resources:
limits:
cpus: '0.50'
memory: 200M
and got a message:
Some services (mysql) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use docker stack deploy to deploy to a swarm.
So I tried:
docker stack deploy mystack --compose-file docker-compose.yml
and got message:
Ignoring unsupported options: restart
this node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again
This seems more complex that docker-compose up.
I saw that I can use --compatibility flag e.g.
docker-compose --compatibility up
But the word compatibility means to me that I should soon switch to a new way of launching my apps locally.
My question is: What is the new procedure that I should follow for launching apps on my workstation using a docker and a descriptor file, in order to support options present in Compose file v3?
If you want to specify memory limits and similar constraints for local containers, you need to use a version 2 Compose file. This is called out in the documentation for the deploy: resources: section. docker/compose#4513 has some reasonably clear statements that Compose file version 2 is more targeted at local setups and version 3 more at Swarm installations, and that Docker intends to keep supporting both file versions.
Docker has put many options and functions specific to their Swarm cluster-installation mode into the core product. Anything that mentions a "stack", for example, is specific to a Swarm setup. One consequence of Swarm and plain-Docker things being combined together is that the deploy: Docker Compose options only have an effect in Swarm mode. The documentation for the deploy: key notes:
This only takes effect when deploying to a swarm with docker stack deploy, and is ignored by docker-compose up and docker-compose run.
My question is: What is the new procedure that I should follow for launching apps on my workstation using a docker and a descriptor file, in order to support options present in Compose file v3?
Docker compose V3 is meant to be used with Docker Swarm deployments, therefore you need to run your Docker in Swarm mode, otherwise just keep using the V2 and it's simpler interface for localhost developments.
For example restart is ignored because that responsibility belongs now to the Docker Swarm, not to Docker itself.
Using the compatibility flag it's kind of converting at runtime your V3 compose file into a V2 compose file.
So in short just use V3 if you want to run Docker in Swarm mode to take advantage of all its new features, aka it's kind of a Kubernetes in Docker land.
docker and docker-compose seem to be interacting with the same dockerFile, what is the difference between the two tools?
The docker cli is used when managing individual containers on a docker engine. It is the client command line to access the docker daemon api.
The docker-compose cli can be used to manage a multi-container application. It also moves many of the options you would enter on the docker run cli into the docker-compose.yml file for easier reuse. It works as a front end "script" on top of the same docker api used by docker, so you can do everything docker-compose does with docker commands and a lot of shell scripting. See this documentation on docker-compose for more details.
Update for Swarm Mode
Since this answer was posted, docker has added a second use of docker-compose.yml files. Starting with the version 3 yml format and docker 1.13, you can use the yml with docker-compose and also to define a stack in docker's swarm mode. To do the latter you need to use docker stack deploy -c docker-compose.yml $stack_name instead of docker-compose up and then manage the stack with docker commands instead of docker-compose commands. The mapping is a one for one between the two uses:
Compose Project -> Swarm Stack: A group of services for a specific purpose
Compose Service -> Swarm Service: One image and it's configuration, possibly scaled up.
Compose Container -> Swarm Task: A single container in a service
For more details on swarm mode, see docker's swarm mode documentation.
docker manages single containers
docker-compose manages multiple container applications
Usage of docker-compose requires 3 steps:
Define the app environment with a Dockerfile
Define the app services in docker-compose.yml
Run docker-compose up to start and run app
Below is a docker-compose.yml example taken from the docker docs:
services:
web:
build: .
ports:
- "5000:5000"
volumes:
- .:/code
- logvolume01:/var/log
links:
- redis
redis:
image: redis
volumes:
logvolume01: {}
A Dockerfile is a text document that contains all the commands/Instruction a user could call on the command line to assemble an image.
Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. By default, docker-compose expects the name of the Compose file as docker-compose.yml or docker-compose.yaml. If the compose file has a different name we can specify it with -f flag.
Check here for more details
docker or more specifically docker engine is used when we want to handle only one container whereas the docker-compose is used when we have multiple containers to handle. We would need multiple containers when we have more than one service to be taken care of, like we have an application that has a client server model. We need a container for the server model and one more container for the client model. Docker compose usually requires each container to have its own dockerfile and then a yml file that incorporates all the containers.
I created simple compose config to try Postgres BDR replication.
I expect containers to have host names as service names I defined and I expect one container to be able to resolve and reach another with this hostname. I expect it to be true because of that:
https://docs.docker.com/compose/networking/
My config:
version: '2'
services:
bdr1:
image: bdr
volumes:
- /var/lib/postgresql/data1:/var/lib/postgresql/data
ports:
- "5001:5432"
bdr2:
image: bdr
volumes:
- /var/lib/postgresql/data2:/var/lib/postgresql/data
ports:
- "5002:5432"
But in reality both containers get rubbish hostnames and are not reachable by container names:
Creating network "bdr_default" with the default driver
Creating bdr_bdr1_1
Creating bdr_bdr2_1
Attaching to bdr_bdr1_1, bdr_bdr2_1
bdr1_1 | Hostname: 938e0585fee2
bdr2_1 | Hostname: 7153165f4d5b
Is it a bug, or I did something wrong?
I use Ubuntu 14.04.4 LTS, Docker version 1.10.1, build 9e83765, docker-compose version 1.6.0, build d99cad6
docker-compose gives you the option of scaling services up or down, meaning you can launch multiple instances of the same service. That is at least one reason why the hostnames are not just service names. You will notice that if you scale bdr1 to 2 instance, you will then have bdr_bdr1_1 and bdr_bdr1_2 containers.
You can work around this inside the containers that were started up by docker-compose in at least two ways:
If a service refers to other service, you can use links section, for example make bdr1 link to bdr2. In this case when you are inside bdr1 you can call host bdr2 by name. I have not tried what happens when you scale up bdr2 in this case.
You can force the hostname of a container internally to the name you want by using the hostname section. For example if you add hostname: bdr1 to bdr1, then you can internally connect to bdr1, which is itself.
You can possibly achieve similar result with the networks section, but I have not yet used it myself so I don't know for sure.
The hostname inside the container should be the short container id, so this is correct (note there was a bug with Compose 1.6.0 and the short container id, so you should use at least version 1.6.2). Also /etc/hosts is not longer used, there is now an embedded dns server that handles resolving names to container ip addresses.
The container is discoverable by other containers with 3 names: the container name, the container short id, and the service name.
However, the other container may not be available immediately when the first one starts. You can use depends_on to set the order.
If you are testing the discovery, try using ping, and make sure to retry , because the name may not resolve immediately.