I have a docker-compose.yml set up like this:
app:
build:
dockerfile: ./docker/app/Dockerfile.dev
image: test/test:${ENV}-test-app
...
Dockerfile called here has this line present:
...
RUN ln -s ../overrides/${ENV}/plugins ../plugins
...
And there is also a script I am running to get the whole environment up (it is dependant upon several containers so I tried to omit irrelevant info).
It is a bash script and running the following:
ENV=$1 docker-compose -p $1 up -d --force-recreate --build app
What I wanted to achieve is that i can run two app containers at the same time, and this works as follows:
sh initializer.sh foo -> creates foo-test-app container
sh initializer.sh bar -> creates bar-test-app container
Now the issue I'm having is that even when I have --force-recreate flag present two images created actually are seen as the same image with two different tags.
And what this does when I inspect the containers is that both containers have a symbolic link to:
overrides/foo/plugins
It doesn't notice when I create the new container to re-do that part. How can I fix it?
Also if I sh to one container and change the symbolic link, it is automatically changed in the other container as well.
$ENV in your dockerfile is not the same as the one in your compose file.
When you run docker-compose up, it can be roughly seen as a docker build followed by a docker run. So Docker builds the image, layer by layer, at that stage there is not env called ENV. Only at docker run will $ENV be used.
Environment variables at build stage can be used though, they are passed via ARG
// compose.yml
build:
context: frontend
args:
- BUILD_ENV=${BUILD_ENV}
// dockerfile
ARG BUILD_ENV
RUN ./node_modules/.bin/ng build --$BUILD_ENV
You can do this to solve your problem however this will create one image per project, which you may not want. Or you can do it in an entrypoint script.
I have found the answer to be in project flag when creating my containers. So this is what I did:
docker-compose -p foo up -d
docker-compose -p bar up -d
This would bring containers up as 2 separate projects.
Link to documentation
Related
For all my images I have a latest tag for production, and test for testing. I also have 2 docker-compose files that only differ in that one uses latest on all packages (except nginx), while the other uses test. Is there a way to set this via a CLI variable, so that I don't have to keep those two files in sync manually all the time?
We can handle this with an environment variable.
docker-compose.yml:
services:
...
my-service-one:
image: my/service-one:$TAG
...
my-service-two:
image: my/service-two:$TAG
...
Then we can add an .env file (for production) with content
TAG=latest
and a test.env file (for testing) with content
TAG=test
When we run docker compose up, file .env will be used by default. If we want to start our test deployment, we can run docker compose --env-file test.env up.
You can simply run these commands for each compose file
TAG=latest docker-compose -p latestapp -f compose-for-latest.yaml up -d
and
TAG=test docker-compose -p testapp -f compose-for-test.yaml up -d
of course, don't forget to specify where you want to use your Tag variable in the compose files using ${TAG} or $TAG (BOTH ARE VALID)
If you are a LINUX user this should work fine.
if you are a WINDOWS user and the commands didn't work you can use GIT BASH instead.
I'm using docker compose to build my application using docker.
Version of docker-compose is 2.2
I have all the containers running well at the moment where one of the container has nginx running.
I need to change some configuration on this container.
The way I need to do (because of special scenario) is, to update the config inside the container.
Then I commit the container to build a new image.
docker commit <container> <image-name>
Now I have new image with tag latest.
What I want is to use this image when I run, docker-compose down && docker-compose up --build next time.
docker-compose down && docker-compose up --build -d
With --build option, docker-compose will go through the steps in Dockerfile and run those and all my changes will be reverted.
Question:
Is there anyway that I can tell docker-compose to use the newly created image as cache and ignore Dockerfile for this one container?
Solution Tried:
I have tried with docker-compose-override and using option cache-from and it's not working.
docker-compose.override.yml
container:
build:
cache_from:
- new-image:latest
Thanks in advance.
I don't understand why you would want to build an image from docker-compose even though you have already built it by docker-commit.
Now I have new image with tag latest.
What I want is to use this image when I run, docker-compose down && docker-compose up
If you have already built image, skip the build phase in docker-compose. Just specify which image should be used like so:
container:
image: new-image:latest
container_name: "Foo bar"
.....(other options)
Image
Specify the image to start the container from. Can either be a
repository/tag or a partial image ID.
image: redis
image: ubuntu:14.04
image: tutum/influxdb
image: example-registry.com:4000/postgresql
image: a4bc65fd
If the image does
not exist, Compose attempts to pull it, unless you have also specified
build, in which case it builds it using the specified options and tags
it with the specified tag.
If you have any other images that you build from inside docker-compose run:
docker-compose build && docker-compose up
If not simple docker-compose up will suffice.
Supposed I have multiple containers deployed
init
service1
service2
db
web
test
The init container runs into completion and then shutdowns by itself. That is his job, which is to do some
pre-configuration stuffs then exit.
When running locally, I dont have any issues running this in my desktop work environment.
My issue is when it is deployed in my CI pipeline. When my init container finished up...it shutdowns the whole
docker-compose network.
Even if I explicitly set the --exit-code-from into my test container.
docker-compose up --exit-code-from test
The end result is that I am not able to run my test cases to its completion because everything is being shutdown by the init container that exits.
Anybody has hints what can I do?
This is interesting. Is it possible to include the compose file? Maybe you have a depends_on defined, and the version of docker used by your CI pipeline handles it differently from the one on your dev environment.
At any rate, you'd want to stop using --exit-code-from, it apparently implies --abort-on-container-exit.
From https://docs.docker.com/compose/reference/up/:
--abort-on-container-exit Stops all containers if any container was
stopped. Incompatible with -d.
--exit-code-from SERVICE Return the exit code of the selected service
container. Implies --abort-on-container-exit.
I ran into the same issue when trying to run Cypress together with MongoDB seeding container and a replica set starter container. The 2 mongo-related containers would exit quickly after doing their job, thus triggering the unintuitive --abort-on-container-exit implied by --exit-code-from cypress.
For me the simplest solution was to use the tail -f /dev/null hack. The idea is that if you run this command after whatever the containers that you don't want to exit are finished with their actual jobs, they will hang until another container triggers the --abort-on-container-exit and pulls with it the entire docker-compose setup down.
Note that this is not a univsal answer: the downside of this approach is that you have to find out what the original CMD is in containers that you don't have control over.
For example, let's take the mongo-seeding project and their Dockerfile. In order to keep the container alive after doing its job, I'd like to make my own Dockerfile in which I'll pull that image and define a custom ENTRYPOINT that will first run the CMD from the original definition of the mongo-seeding image and then run tail -f /dev/null to keep the container alive. In their Dockerfile I can see that the CMD is simply seed and I can assume it won't change in the future (good design) so my ENTRYPOINT script entry.sh can just look like this:
#!/bin/sh
seed
tail -f /dev/null
And my Dockerfile:
FROM pkosiec/mongo-seeding:3.6.0
ENTRYPOINT [ "/app/scripts/entry.sh" ]
Plus the relevant service in docker-compose with volumes mount for completeness:
mongo-seed:
build:
context: ./mongoSeed
volumes:
- ./mongoSeed/data:/app/data
- ./mongoSeed/scripts:/app/scripts
working_dir: /app/data
depends_on:
- mongodb
Which makes the container do its job and then hang until Cypress exits and causes the entire docker-compose setup to stop.
For clarification, when I say base image, I mean the parent image that has all the common configurations, so that the children based on it don't need to download the dependencies individually.
From my understanding, docker-compose.yml files are the run-time configurations, while Dockerfiles are the build-time configurations. However, there is a build option using docker-compose, and I was wondering how I could use this to build a base image.
As of right now, I use a shellscript that runs other shellscripts. One builds all my images, from a base image that it also creates. The other runs them as containers with the necessary configurations. However, the base image is never ran as a container.
Currently, the shellscript I hope to change into a docker-compose file, looks like so:
echo "Creating docker network net1"
docker network create net1
echo "Running api as a container with port 5000 exposed on net1"
docker run --name api_cntr --net net1 -d -p 5000:5000 api_img
echo "Running redis service with port 6379 exposed on net1"
docker run --name message_service --net net1 -p 6379:6379 -d redis
echo "Running celery worker on net1"
docker run --name celery_worker1 --net net1 -d celery_worker_img
echo "Running flower HUD on net1 with port 5555 exposed"
docker run --name flower_hud --net net1 -d -p 5555:5555 flower_hud_img
The shellscript that makes the images, is as follows:
echo "Building Base Image"
docker build -t base ../base-image
echo "Building api image from Dockerfile"
docker build -t api_img ../api
echo "Building celery worker image"
docker build -t celery_worker_img ../celery-worker
echo "Building celery worker HUD"
docker build -t flower_hud_img ../flower-hud
My questions comes down to one thing, can I create this Base image without ever running it in a container with docker-compose. (All the Dockerfiles start with FROM base:latest other than the base itself). I'm looking to make it as easy as possible for other people, so that they only have to run a single command.
EDIT: I am using version 3, and acording to the docs, build: is ignored, and docker-compose only accepts pre-built images.
Yes, kind of. Use it like this:
version: '2'
services:
wls-admin:
container_name: wls-admin
image: weblogic-domain
build:
context: wls-admin
args:
- ADMIN_PORT=${WLS_ADMIN_PORT}
- CLUSTER_NAME=${WLS_CLUSTER_NAME}
- PRODUCTION_MODE=dev
networks:
- wls-network
image clause here makes docker-compose build generate docker image named weblogic-domain for this service. This image can be re-used by other services' Dockerfiles, even in the same build process.
Doing a bit more research based on #amiasato 's anser, it looks as if there is a replicated key, which you can set to 0 like so:
version: "3"
services:
base-image:
build:
context: .
dockerfile: Dockerfile-base
deploy:
mode: replicated
replicas: 0
See https://docs.docker.com/compose/compose-file/compose-file-v3/#replicas
Just a minor addition to Kanedias' answer. If you choose to follow his approach (which was my choice), you can avoid instantiating a container for the base image with the --scale flag from the docker-compose up command:
docker-compose up --scale wls-admin=0
From the up command documentation:
--scale SERVICE=NUM Scale SERVICE to NUM instances. Overrides the
`scale` setting in the Compose file if present.
One important thing to note is that the scale setting in the docker-compose.yml was removed in v3, so there is actually nothing to override in v3.
Instead of running docker-compose, you can implement a script, witch builds image with specific tag docker build ... -t your_tag, then runs docker-compose. In children dockerfiles you can use FROM your_tag.
As per the documentation the build option of a service takes a directory as an argument which contains the famous Dockerfile. There is no way to build a base image and then the actual image of the service.
Docker is a environment in which your application runs. When you are creating a base image, it should have things which are not going to change often. Then you need to build baseiamge once and upload to your repository and use FROM baseimage:latest in the Dockerfile.
For example, if you are building a python application you can create it from python and install requirements:
FROM python:3.6
COPY requirements.txt .
RUN pip install -r requirements.txt
here, python:3.6 is the base image which is not going to change often and thus you need not build it every time you are running docker compose commands.
From the shellscript that makes the images, we can see that you have different dockerfiles in different directories. You can use that to create a docker-compose.yml file. The build settings are used to tell docker that how should it build the image.
You can use those dockerfiles in your compose file in this manner:
version: '3'
services:
api_cntr:
image: api_img
build:
context: ./api
container_name:api_cntr
ports:
- 5000:5000
Here, I have assumed that your docker-compose.yml file is placed in a folder which also contains a directory called base-image. And base-image has a dockerfile which is used to build the image.
This can be structure of one of your service. In similar manner, you can create other services also. And while usig docker-compose you will not need to specify a network for each, because all services declared within a docker-compose.yml file are part of an isolated network.
I am trying to get my head around the COMMAND option in docker compose. In my current docker-compose.yml i start the prosody docker image (https://github.com/prosody/prosody-docker) and i want to create a list of users when the container is actually started.
The documentation of the container states that a user can be made using environment options LOCAL, DOMAIN, and PASSWORD, but this is a single user. I need a list of users.
When reading some stuff around the internet it seemed that using the command option i should be able to execute commands in a starting or running container.
xmpp:
image: prosody/prosody
command: prosodyctl register testuser localhost testpassword
ports:
- "5222:5222"
- "127.0.0.1:5347:5347"
But this seems not to work, i checked to running container using docker exec -it <imageid> bash but the user is not created.
Is it possible to execute a command on a started container using docker-compose or are there other options?
The COMMAND instruction is exactly the same as what is passed at the end of a docker run command, for example echo "hello world" in:
docker run debian echo "hello world"
The command is interpreted as arguments to the ENTRYPOINT of the image, which in debian's case is /bin/bash. In the case of your image, it gets passed to this script. Looking at that script, your command will just get passed to the shell. I would have expected any command you pass to run successfully, but the container will exit once your command completes. Note that the default command is set in the Dockerfile to CMD ["prosodyctl", "start"] which is presumably a long-running process which starts the server.
I'm not sure how Prosody works (or even what it is), but I think you probably want to either map in a config file which holds your users, or set up a data container to persist your configuration. The first solution would mean adding something like:
volumes:
- my_prosodoy_config:/etc/prosody
To the docker-compose file, where my_prosody_config is a directory holding the config files.
The second solution could involve first creating a data container like:
docker run -v /etc/prosody -v /var/log/prosody --name prosody-data prosody-docker echo "Prosody Data Container"
(The echo should complete, leaving you with a stopped container which has volumes set up for the config and logs. Just make sure you don't docker rm this container by accident!)
Then in the docker-compose file add:
volumes_from:
- prosody-data
Hopefully you can then add users by running docker exec as you did before, then running prosodyctl register at the command line. But this is dependent on how prosody and the image behave.
CMD is directly related to ENTRYPOINT in Docker (see this question for an explanation). So when changing one of them, you also have to check how this affects the other. If you look at the Dockerfile, you will see that the default command is to start prosody through CMD ["prosodyctl", "start"]. entrypoint.sh just passes this command through as Adrian mentioned. However, your command overrides the default command, so your prosody demon is never started. Maybe you want to try something like
xmpp:
image: prosody/prosody
command: sh -c prosodyctl register testuser localhost testpassword && prosodyctl start
ports:
- "5222:5222"
- "127.0.0.1:5347:5347"
instead. More elegant and somehow what the creator seems to have intended (judging from the entrypoint.sh script) would be something like
xmpp:
image: prosody/prosody
environment:
- LOCAL=testuser
- DOMAIN=localhost
- PASSWORD=testpassword
ports:
- "5222:5222"
- "127.0.0.1:5347:5347"
To answer your final question: no, it is not possible (as of now) to execute commands on a running container via docker-compose. However, you can easily do this with docker:
docker exec -i prosody_container_name prosodyctl register testuser localhost testpassword
where prosody_container_name is the name of your running container (use docker ps to list running containers).