This is a section of my docker-compose file:
version: "3.2"
services:
cachable_service:
image: my/cachable_service:x.x
build:
context: .
dockerfile: cachable_service_Dockerfile
service_in_development:
image: my/service_in_development:latest
build:
context: .
dockerfile: Dockerfile
depends_on:
- cachable_service
How do I force only the image for the service_in_development to be always build from scratch so I get my latest code in there?
I've looked into the cache_from option in the docker-compose file reference, but I couldn't find any clear explanation on what exactly it does.
Clarifications:
In this question, I was asking about an option to force rebuild only specific services using an option in the docker-compose.yml file itself. But, if a commandline option let's me do this, I'd be happy with that as well for now.
Also, I'm talking about forcing a 'rebuild' of images. Not just 'recreation' of service.
You can use command below to rebuild all images:
docker-compose up --build --force-recreate
to force rebuild one service:
docker-compose up --build --force-recreate service-name-here
More useful parameters in documentation
Usage: up [options] [--scale SERVICE=NUM...] [SERVICE...]
Options:
-d, --detach Detached mode: Run containers in the background,
print new container names. Incompatible with
--abort-on-container-exit.
--no-color Produce monochrome output.
--quiet-pull Pull without printing progress information
--no-deps Don't start linked services.
--force-recreate Recreate containers even if their configuration
and image haven't changed.
--always-recreate-deps Recreate dependent containers.
Incompatible with --no-recreate.
--no-recreate If containers already exist, don't recreate
them. Incompatible with --force-recreate and -V.
--no-build Don't build an image, even if it's missing.
--no-start Don't start the services after creating them.
--build Build images before starting containers.
--abort-on-container-exit Stops all containers if any container was
stopped. Incompatible with -d.
-t, --timeout TIMEOUT Use this timeout in seconds for container
shutdown when attached or when containers are
already running. (default: 10)
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
--remove-orphans Remove containers for services not defined
in the Compose file.
--exit-code-from SERVICE Return the exit code of the selected service
container. Implies --abort-on-container-exit.
--scale SERVICE=NUM Scale SERVICE to NUM instances. Overrides the
`scale` setting in the Compose file if present.
Related
We have various docker-compose files that create a service infrastructure. My question is if it is possible to only create the volumes defined in the compose file without creating the containers?
It might be possible without starting any services.
command can be provided like this.
docker-compose up --no-start --no-deps
Or it's possible to specify individual volume.
docker-compose up --no-deps -d --volume <volume-name>
You can use a docker-compose.override.yml file with invalid entrypoints or different image (use a small template image) for those containers, that way they'll die immediately.
docker-compose.override.yml:
<conatiner-name>:
image: hello-world
command: "/hello"
restart: "no"
Given the following Docker Compose file....
version: '3.8'
services:
producer:
image: producer
container_name: producer
depends_on: [db]
build:
context: ./producer
dockerfile: ./Dockerfile
db:
image: some-db-image
container_name: db
When I do docker-compose up producer obviously the db service gets started too. When I CTRL+C both services are stopped. This is expected and fine.
But sometimes, the db service is started before, on a different shell and so doing docker-compose up producer understands that db is running and only starts producer. But when I hit CTRL+C, both producer and db are stopped even though db was not started as part of this docker compose up command.
Is there a way to avoid getting the dependencies services stopped when stopping its "parent" ?
When running just docker-compose up, the CTRL+C command always stops all running services in the current compose scope. It doesn't care about depends_on.
You would need to spin it up with detach option -d, like
docker-compose up -d producer
Then you can do
docker stop producer
And db service should still be running.
As I understand your question: You want to stop a container A which depends on another container B. But when stopping A, you don't want docker-compose to stop B.
Docker-compose stops the dependent containers ('B' in this case) when 'A' is stopped.
How I would approach this:
Split up the docker-compose files into A and B
In docker-compose for A create a health check testing (and waiting) for container B to be alive.
Since this is a database, you could do this with a dummy query.
Then you still have dependency, but not the docker-compose connection of stopping dependant containers.
You can't simply do that with CTRL+C.
Your docker-compose file and the services defined in it are treated as a project. You may notice that all containers, networks and volumes are prefixed with the name of the directory where the docker-compose file is located by default. This is the project name. It can be changed via an environment variable or the -p flag of the docker-compose command.
What docker-compose does is it keeps track of all the resources for a given project.
In your case there are two services: db and producer. Whenever you run docker-compose up, both of them start up. They both end up being part of the same project. The same applies when you only start one of the services (e.g. with docker-compose up db). You can later start the other service and it will still be part of the same project.
One more thing to note here: Whenever you run docker-compose without the -d (detached) flag, you get attached to the whole project, meaning whenever you hit CTRL+C, you'll stop all services. It does not matter if the last compose command started only one of the services or if they depend on each other. Attaching to the project and hitting CTRL+C will stop them.
A possible solution to your problem would be the following:
Start up your services via docker-compose up -d (both db and producer will get created). They are now in detached mode. If you still want to check the logs in real time (kinda like attaching), use docker-compose logs -f. Now, however, if you want to stop only one of the services you can simply do docker-compose stop $SVC_NAME (where $SVC_NAME is either db or producer) and this will keep the other one running. This way, whatever happens to your terminal session, your services won't stop, unless you explicitly tell them to.
Is there a way to avoid getting the dependencies services stopped when stopping its "parent" ?
Yes.
Using the new version docker compose instead of docker-compose might solve your problem Reference.
Simple example
Assuming now you are using the new version, your process could be something like this.
docker-compose.yml
version: "3.8"
services:
db:
build: .
producer:
build: .
depends_on: [db]
extra:
build: .
Dockerfile
FROM node:alpine
WORKDIR /app
COPY . .
ENTRYPOINT [ "/bin/sh", "script.sh" ]
script.sh
while :; do sleep 1; done
Suppose db has started before with
$ docker compose up -d db.
Then later,
$ docker compose up -d producer.
Now you can stop only producer with
$ docker compose stop producer.
You can check if db is still running with
$ docker compose ps.
Notice the use of -d flag for detached mode, as pointed out in another answer, so you don't need to kill the process with CTRL+C. Also, using detached flag allows you to check the services that are running with docker compose ps.
A similar issue as yours was reported and fixed a while ago, as you can see here.
I was not able to reproduce the behavior you observe with a complete minimal example. Namely, when running docker compose stop producer, the underlying db is not stopped AFAICT.
Anyway, you may be interested in an alternative command that is a bit more flexible than docker compose up, regarding how to run "one-off commands": docker compose run.
The typical use cases are as follows:
docker compose run db bash → run the db service, replacing the default CMD with bash
docker compose run -d db → run the db service in the background (detach mode)
docker compose run --service-ports producer → run the service producer and its dependencies (unless they were run with docker compose up), enabling the ports mapping.
So for your specific use case, you could run:
docker compose up -d db
docker compose run --service-ports producer
I am bringing up my project dependencies using docker-compose. So far this used to work
docker-compose up -d --no-recreate;
However today I tried running the project again after couple of weeks and I was greeted with error message
Creating my-postgres ... error
ERROR: for my-postgres Cannot create container for service postgres: b'Conflict. The container name "/my-postgres" is already in use by container "dbd06bb1d99eda6f075ea688df16e8b355e559e1759f084dee8f3cddfc535b0b". You have to remove (or rename) that container to be able to reuse that name.'
ERROR: for postgres Cannot create container for service postgres: b'Conflict. The container name "/my-postgres" is already in use by container "dbd06bb1d99eda6f075ea688df16e8b355e559e1759f084dee8f3cddfc535b0b". You have to remove (or rename) that container to be able to reuse that name.'
ERROR: Encountered errors while bringing up the project.
My docker-compose.yml file is
postgres:
container_name: my-postgres
image: postgres:latest
ports:
- "15432:5432"
Docker version is
Docker version 19.03.1, build 74b1e89
Docker compose version is
docker-compose version 1.24.1, build 4667896b
Intended behavior of this call is to:
make the container if it does not exist
start the container if it exists
just chill and do nothing if the container is already started
Docker Compose normally assigns a container name based on its current project name and the name of the services: block. Specifying container_name: explicitly overrides this; but, it means you can’t launch multiple copies of the same Compose file with different project names (from different directories) because the container name you’ve explicitly chosen won’t be used.
You almost never care what the container name is explicitly. It only really matters if you’re trying to use plain docker commands to manipulate Compose-managed containers; it has no impact on inter-service communication. Just delete the container_name: line.
(For similar reasons you can almost always delete hostname: and links: sections if you have them with no practical impact on your overall system.)
In my case I moved the project in an other directory.
When I tryed to run docker-compose up it failed because of some conflicts.
With command docker system prune I resolved them.
It's caused by being in a different directory than when you last ran docker-compose up. One option is to change back to the original directory. Or if you've configured it as a systemd service you can use systemctl.
Well...the error message seems pretty straightforward to me...
The container name "/my-postgres" is already in use by container
If you just want to restart where you left, you should use docker-compose start.
Otherwise, just clean up your workspace before running it :
docker-compose down
docker-compose up -d
Remove --no-recreate flag from your docker-compose command. And execute the command again.
$docker-compose up -d
--no-recreate is using for preventing accedental updates.
If there are existing containers for a service, and the service’s configuration or image was changed after the container’s creation, docker-compose up picks up the changes by stopping and recreating the containers. To prevent Compose from picking up changes, use the --no-recreate flag.
official docker docs.Link
I had similar issue
dcdown --remove-orphans
That worked for me.
I docker compose, it is possible to define the context as follows
version: '3'
services:
node1:
build: node1
image: node1
container_name: node1
node2:
build: node2
image: node2
container_name: node2
Where build refers to directory containing Dockerfile and the build resources. By using the command docker-compose up -d --build it is possible to rebuild images and restart changed containers. But using the following docker-compose file, is it possible to only build images without starting them, and preferably choose the images to build or build all?
You can build specific services without starting them like so:
docker-compose build node1
To build them all:
docker-compose build
You can also try docker-compose up --no-start
It will perform build , create network , create volume without starting containers.
Usage: up [options] [--scale SERVICE=NUM...] [SERVICE...]
Options:
-d, --detach Detached mode: Run containers in the background,
print new container names. Incompatible with
--abort-on-container-exit.
--no-color Produce monochrome output.
--quiet-pull Pull without printing progress information
--no-deps Don't start linked services.
--force-recreate Recreate containers even if their configuration
and image haven't changed.
--always-recreate-deps Recreate dependent containers.
Incompatible with --no-recreate.
--no-recreate If containers already exist, don't recreate
them. Incompatible with --force-recreate and -V.
--no-build Don't build an image, even if it's missing.
--no-start Don't start the services after creating them.
I have a docker-compose file that spins up, among several other, a couchdb container (https://hub.docker.com/r/klaemo/couchdb/); and the couchdb container spews out a lot of output when I do the docker-compose up. Is there a way to suppress that output so I see only other containers' s output?
Maybe
I can run the couchdb in daemon mode somehow?
or
I can override the default command somehow and redirect output to a tmp file?
I am not sure how to do any of the two, and I want to do that within the compose file itself, not by changing my compose file callup command. Any help?
Here is the minimal compose file:
couchdb:
container_name: couchdb
image: klaemo/couchdb:2.0.0
ports:
- "5984:5984"
and I call that from a makefile with : docker-compose up --abort-on-container-exit --force-recreate && docker-compose down
Note that Docker containers log to stdout and stderr for a reason. It allows a consistent log interface for commands like docker logs to use and for logging drivers to pick up information from containers. In a large container eco system, it's easier if everything works the same.
Runtime
At runtime there are a couple of options.
You can background the couchdb container and start the others in the foreground.
docker-compose up -d couchdb
docker-compose up other container names
You can start everything in the background, and only view the logs for particular containers
docker-compose start # or docker-compose up -d
docker-compose logs -f other container names
Build time
To permanently modify logging you could change CouchDB's log config in an image build
couchdb:
container_name: couchdb
image: me/klaemo-couchdb:2.0.0
build:
context: .
dockerfile: Dockerfile.couchdb
ports:
- "5984:5984"
Dockerfile.couchdb
FROM klaemo/couchdb:2.0.0
COPY couchdb.ini /opt/couchdb/etc/local.ini
couchdb.ini needs to contain all the original config settings from the containers /opt/couchdb/etc/local.ini, updating some the log settings from stderr to a file:
[log]
file = /opt/couchdb/log/couch.log
level = info
You can also set log levels specifically for a module
[log_level_by_module]
couch_httpd = info
couch_replicator = info
couch_query_servers = error
You probably want to mount the /opt/couchdb/log directory as a volume from the container host so you are not writing data into the current container instance all the time.