I have an application that performs elaboration over a data feed. The process is divided into tasks so I structured a docker-compose.yml file like this:
task1-service:
image: task1-image
task2-service:
image: task2-image
task3-service:
image: task3-image
Each task-service is triggered by the end of the previous and triggers the next, then it can exit. So there's no point to keep each service running.
I wonder if there's a solution to keep them all stopped, and start each service on demand when needed.
I don't know if docker compose is the correct solution, but I like the idea ok keeping the system described into one yml file. Anyway, other solutions are appreciated.
Thanks
Is possible to solve your approach in different ways, and one of them is with docker-compose.
First, you can start one concrete service (taskX-service) using docker-compose up -d <service_name>
You have more details in docker-compose up for only certain containers
Second, docker-compose also allows you to configure dependencies between containers. If you want to run them in order, you can specify it in depends_on: structure.
For example, to execute tasks 1, 2, and 3 in order you could use:
task1-service:
image: task1-image
task2-service:
image: task2-image
depends_on:
- task1-service
task3-service:
image: task3-image
depends_on:
- task2-service
Furthermore, this docker-compose.yml is compatible with first I said:
docker-compose up -d task1-service
docker-compose up -d task2-service (also launchs task1-service)
docker-compose up -d task3-service (also launchs task2 and 1 service)
If you don't specify any container, with docker-compose down stops all containers in compose file.
I hope it's useful for you.
Related
Given the following Docker Compose file....
version: '3.8'
services:
producer:
image: producer
container_name: producer
depends_on: [db]
build:
context: ./producer
dockerfile: ./Dockerfile
db:
image: some-db-image
container_name: db
When I do docker-compose up producer obviously the db service gets started too. When I CTRL+C both services are stopped. This is expected and fine.
But sometimes, the db service is started before, on a different shell and so doing docker-compose up producer understands that db is running and only starts producer. But when I hit CTRL+C, both producer and db are stopped even though db was not started as part of this docker compose up command.
Is there a way to avoid getting the dependencies services stopped when stopping its "parent" ?
When running just docker-compose up, the CTRL+C command always stops all running services in the current compose scope. It doesn't care about depends_on.
You would need to spin it up with detach option -d, like
docker-compose up -d producer
Then you can do
docker stop producer
And db service should still be running.
As I understand your question: You want to stop a container A which depends on another container B. But when stopping A, you don't want docker-compose to stop B.
Docker-compose stops the dependent containers ('B' in this case) when 'A' is stopped.
How I would approach this:
Split up the docker-compose files into A and B
In docker-compose for A create a health check testing (and waiting) for container B to be alive.
Since this is a database, you could do this with a dummy query.
Then you still have dependency, but not the docker-compose connection of stopping dependant containers.
You can't simply do that with CTRL+C.
Your docker-compose file and the services defined in it are treated as a project. You may notice that all containers, networks and volumes are prefixed with the name of the directory where the docker-compose file is located by default. This is the project name. It can be changed via an environment variable or the -p flag of the docker-compose command.
What docker-compose does is it keeps track of all the resources for a given project.
In your case there are two services: db and producer. Whenever you run docker-compose up, both of them start up. They both end up being part of the same project. The same applies when you only start one of the services (e.g. with docker-compose up db). You can later start the other service and it will still be part of the same project.
One more thing to note here: Whenever you run docker-compose without the -d (detached) flag, you get attached to the whole project, meaning whenever you hit CTRL+C, you'll stop all services. It does not matter if the last compose command started only one of the services or if they depend on each other. Attaching to the project and hitting CTRL+C will stop them.
A possible solution to your problem would be the following:
Start up your services via docker-compose up -d (both db and producer will get created). They are now in detached mode. If you still want to check the logs in real time (kinda like attaching), use docker-compose logs -f. Now, however, if you want to stop only one of the services you can simply do docker-compose stop $SVC_NAME (where $SVC_NAME is either db or producer) and this will keep the other one running. This way, whatever happens to your terminal session, your services won't stop, unless you explicitly tell them to.
Is there a way to avoid getting the dependencies services stopped when stopping its "parent" ?
Yes.
Using the new version docker compose instead of docker-compose might solve your problem Reference.
Simple example
Assuming now you are using the new version, your process could be something like this.
docker-compose.yml
version: "3.8"
services:
db:
build: .
producer:
build: .
depends_on: [db]
extra:
build: .
Dockerfile
FROM node:alpine
WORKDIR /app
COPY . .
ENTRYPOINT [ "/bin/sh", "script.sh" ]
script.sh
while :; do sleep 1; done
Suppose db has started before with
$ docker compose up -d db.
Then later,
$ docker compose up -d producer.
Now you can stop only producer with
$ docker compose stop producer.
You can check if db is still running with
$ docker compose ps.
Notice the use of -d flag for detached mode, as pointed out in another answer, so you don't need to kill the process with CTRL+C. Also, using detached flag allows you to check the services that are running with docker compose ps.
A similar issue as yours was reported and fixed a while ago, as you can see here.
I was not able to reproduce the behavior you observe with a complete minimal example. Namely, when running docker compose stop producer, the underlying db is not stopped AFAICT.
Anyway, you may be interested in an alternative command that is a bit more flexible than docker compose up, regarding how to run "one-off commands": docker compose run.
The typical use cases are as follows:
docker compose run db bash → run the db service, replacing the default CMD with bash
docker compose run -d db → run the db service in the background (detach mode)
docker compose run --service-ports producer → run the service producer and its dependencies (unless they were run with docker compose up), enabling the ports mapping.
So for your specific use case, you could run:
docker compose up -d db
docker compose run --service-ports producer
I have a docker-compose project which (simplified) looks like:
version: '3'
services:
main:
image: main-image
depends_on:
- my_service
my_service:
image: very-big-image
The functionality supplied by my-service is also available in the cloud. Running it locally is faster, but demands extreme CPU and disk resources. Therefore, I will sometimes want to run just main, and other times will want to run both containers.
I don't want to duplicate the docker-compose code; the full container definitions are much more complex than the simplified version above.
I thought I could put each service in its own .yml file and then do docker-compose -f main.yml -f my_service.yml up, but that fails on the depends-on line, which seems to require the dependency to be in the same file.
For now, I have an ugly solution. I split into two (with the second files starting with the depends_on line. I then use a shell script that knows which configuration I want to run, and either runs the first file or concatenates the two files together. This works, I guess; but is ugly.
What is the right way to do this?
Try this:
docker-compose start <container_name>
This command only starts a selected container.
What is the right way to do this?
I think you've already identified it: split the service definitions between two files, so that sometimes you can start both:
docker-compose -f main.yml -f service.yml up
And sometimes you can start just the service container:
docker-compose -f service.yml up
This works fine, and the depends_on key does not require that all definitions be in the same file. The dictionaries are merged before docker-compose tries to resolve dependencies.
For example, if I have in main.yml:
---
version: "3"
services:
main:
image: alpine
command: sleep 30
depends_on:
- service
And in service.yml:
---
version: "3"
services:
service:
image: alpine
command: sleep 30
I can start just the service container like this:
$ docker-compose -f service.yml up
Starting compose_service_1 ... done
Attaching to compose_service_1
Or I can start both the main and service containers:
$ docker-compose -f main.yml -f service.yml up
Starting compose_service_1 ... done
Starting compose_main_1 ... done
Attaching to compose_service_1, compose_main_1
If you're seeing behavior that appears to contradict this, it would help if you were to update your question with minimal examples that reproduce the problem, and include any error output from running docker-compose.
Our team is new to running a micro-service ecosystem and I am curious how one would achieve conditionally loading docker containers from a compose, or another variable-cased script.
An example use-case would be.
Doing front-end development that depends on a few different services. We will label those DockerA/D
Dependency Matrix
Feature1 - DockerA
Feature2 - DockerA and DockerB
Feature3 - DockerA and DockerD
I would like to be able to run something like the following
docker-compose --feature1
or
magic-script -service DockerA -service DockerB
Basically, I would like to run the command to conditionally start the APIs that I need.
I am already aware of using various mock servers for UI development, but want to avoid them.
Any thoughts on how to configure this?
You can stop all services after creating them and then selectively starting them one by one. E.g.:
version: "3"
services:
web1:
image: nginx
ports:
- "80:80"
web2:
image: nginx
ports:
- "8080:80"
docker-compose up -d
Creating network "composenginx_default" with the default driver
Creating composenginx_web2_1 ... done
Creating composenginx_web1_1 ... done
docker-compose stop
Stopping composenginx_web1_1 ... done
Stopping composenginx_web2_1 ... done
Now any service can be started using, e.g.,
docker-compose start web2
Starting web2 ... done
Also, using linked services, there's the scale command that can change the number of running services (can add containers without restart).
My situation. Trying to run a docker-compose structure several times on the same box. This is my docker-compose.yml:
version: '3'
services:
code:
image: organization:java-maven
links:
- mysql:mysql
volumes:
- "${PWD}:/home/ubuntu/src"
mysql:
image: organization:mysql
Running this twice with docker-compose run code mvn clean test creates two containers of code and one container of mysql.
Now, I want one code to be linked to one mysql, and another code linked to another mysql.
How do I accomplish this? This is supposed to run on jenkins slaves and the maven executions cannot share mysql.
I've miserably failed trying with the "-e KEY=VALUE" option for docker-compose run together with container_namein the docker compose file.
Not sure how to approach this, please help, thank you.
So, I focused too much on using directives to alter container names manually. The solution was much easier.
docker-compose -p anything run code mvn clean test
docker-compose -p anything_else run code mvn clean test
So, this is the project name solution. Docker compose will use the value given with the option -p as a prefix when creating container names. That means no collision.
Very handy!
For more reading: documentation around project-name option
This question is coming from an issue on the Docker's repository:
https://github.com/docker/compose/issues/942
I can't figure it out how to create a data container (no process running) with docker compose.
UPDATE: Things have changed in the last years. Please refer to the answer from #Frederik Wendt for a good and up-to-date solution.
My old answer: Exactly how to do it depends a little on what image you are using for your data-only-container. If your image has an entrypoint, you need to overwrite this in your docker-compose.yml. For example this is a solution for the official MySql image from docker hub:
DatabaseData:
image: mysql:5.6.25
entrypoint: /bin/bash
DatabaseServer:
image: mysql:5.6.25
volumes_from:
- DatabaseData
environment:
MYSQL_ROOT_PASSWORD: blabla
When you do a docker-compose up on this, you will get a container like ..._DatabaseData_1 which shows a status of Exited when you call docker ps -a. Further investigation with docker inspect will show, that it has a timestamp of 0. That means the container was never run. Like it is stated by the owner of docker compose here.
Now, as long as you don't do a docker-compose rm -v, your data only container (..._DatabaseData_1) will not loose its data. So you can do docker-compose stop and docker-compose up as often as you like.
In case you like to use a dedicated data-only image like tianon/true this works the same. Here you don't need to overwrite the entrypoint, because it doesn't exist. It seems like there are some problems with that image and docker compose. I haven't tried it, but this article could be worth reading in case you experience any problems.
In general it seems to be a good idea to use the same image for your data-only container that you are using for the container accessing it. See Data-only container madness for more details.
The other answers to this question are quite out of date, and data volumes have been supported for some time now. Example:
version: "3.9"
services:
frontend:
image: node:lts
volumes:
- myapp:/home/node/app
volumes:
myapp:
See
https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose for details and options.
A data only container (DOC) is a container that is created only to serve as a volume provider. The container itself has no function other than that other containers can mount it's volume by using the volumes_from directive.
The DOC has to run only once to create the volume. Other containers can reference the volumes in it even if it's stopped.
The OP Question:
The docker-compose.yml starts the DOC every time you do a docker-compose up. OP asks for an option to only create container and volume, and not run it, using some sort of an create_only: true option.
As mention in the issue from the OP's question:
you either create a data container with the same name as the one specified in the docker-compose.yml, and run docker-compose up --no-recreate (the one specified in docker-compose.yml won't be recreated).
or you run a container with a simple command which never returns.
Like: tail -f /dev/null