I'm using Docker Compose with Docker Config.
The config is created ahead of time with docker config create conf.yml conf.yml
The compose file specifies the configs:
version: '3.3'
configs:
conf.yml:
external: true
services:
api:
image: <image_link>
deploy:
replicas: 1
ports:
- "5002:80"
configs:
- source: conf.yml
target: /etc/conf/conf.yml
mode: 0440
I then deploy it to a docker swarm stack with docker stack deploy
Now I rotate the config according to this example, I end up with conf2.yml
That means the next time I run docker stack deploy (through our CI), the source file will be invalid.
I could re-create conf.yml then call docker service update but it's a lot of manual work for a configuration file.
Do you have any advice for a more robust handling of config files? Note that the configuration files are not in the repo and not stored in the CI runner / environment variables either.
Seems like the best solution is to edit the docker-compose file with the new config version and re-deploy.
Related
I use docker-compose to run containers to build different components as part of my CI/CD pipeline. Recently I have started noticing that the build pipeline fails sometimes while trying to download build dependencies from the internet in run time, as sometimes these links are down.
To fix this I want to test running the docker container which builds these applications without any internet connection, I want to include all the required dependencies which are downloaded at runtime as part of the docker image.
Can someone let me what change should I make in my docker-compose so that the container starts without any network access?
Here is a sample of my docker-compose file:
version: "3.7"
services:
build-component-1:
image: base-image:1
network_mode: host
volumes:
- "$PWD:/build-dir"
command: ./build.sh build_app1
working_dir: /build-dir
privileged: true
environment:
- USER=root
- VAR1=$VAR1
env_file:
- version.txt
Found the answer here - https://docs.docker.com/compose/compose-file/compose-file-v3/
I have to replace network_mode: host with network_mode: none in the docker-compose file.
I have existing docker-compose.yml file that runs on my Docker CE standalone server.
I would like to deploy this same configuration using the AWS ECS service. The documentation of the ecs-cli tool states that Docker Compose files can be used. Other (simpler) container configs have worked with my existing files.
With my configuration, this errors with:
ERRO[0000] Unable to open ECS Compose Project error="External option
is not supported"
FATA[0000] Unable to create and read ECS Compose Project
error="External option is not supported"
I am using "external" Docker volumes, so that they are auto-generated as required and not deleted when a container is stopped or removed.
This is a simplification of the docker-compose.yml file I am testing with and would allow me to mount the volume to a running container:
version: '3'
services:
busybox:
image: busybox:1.31.1
volumes:
- ext_volume:/path/in/container
volumes:
ext_volume:
external: true
Alternatively, I have read in other documentation to use the ecs-params.yml file in the same directory to pass in variables. Is this a replacement to my docker-compose.yml file? I had expected to leave it's syntax unchanged.
Working config (this was ensuring the container stays running, so I could ssh in and view the mounted drive):
version: '3'
services:
alpine:
image: alpine:3.12
volumes:
- test_docker_volume:/path/in/container
command:
- tail
- -f
- /dev/null
volumes:
test_docker_volume:
And in ecs-params.yml:
version: 1
task_definition:
services:
alpine:
cpu_shares: 100
mem_limit: 28000000
docker_volumes:
- name: test_docker_volume
scope: "shared"
autoprovision: true
So, I'm running into an issue. Say you have a simple Docker Compose file like this:
version: "3.7"
services:
web:
image: repo.hostname.com/web:latest
environment:
port: 8080
ports:
- 8080:8080
Then, I'd run the following command to apply it:
docker stack deploy --compose-file path/to/compose.yml
Now, here's my problem. Once I've created the services via stack deploy, how do I UPDATE an existing service via the compose file?
If I just change the environment variable of "port" from "8080" to "8000" and rerun stack deploy with the new compose file, it doesn't pick up the change.
And, no, I can't use Kubernetes for reasons that are way out of the scope of this post.
I am trying to setup a Docker-based Jenkins instance. Essentially, I run the jenkins/jenkins:lts image as a container and mount a data volume to persist the data Jenkins will create.
Now, what I would like to do is share the host's ssh keys with this Jenkins instance. It's probably due to my limited Docker knowledge, but my problem is I don't know how I can mount additional files/directories to my volume and Jenkins requires that I put ssh keys within var/jenkins_home/.ssh.
I tried naively creating the directories in Dockerfile and then mounting them with docker-compose. It failed, as you might expect, since the volume is the one containing Jenkins' home directory data, not the Jenkins container itself.
I have the following docker-compose.yml (not working, for the reasons mentioned above):
version: '3.1'
services:
jenkins:
restart: always
build: ./jenkins
environment:
VIRTUAL_HOST: ${NGINX_VIRTUAL_HOST}
VIRTUAL_PORT: 8080
JAVA_OPTS: -Djenkins.install.runSetupWizard=false
TZ: America/New_York
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- jenkins_data:/var/jenkins_home
networks:
- web
- proxy
healthcheck:
test: ["CMD", "curl --fail http://${NGINX_VIRTUAL_HOST}/ || exit 1"]
interval: 1m
timeout: 10s
retries: 3
secrets:
- host_ssh_key
volumes:
jenkins_data:
networks:
web:
driver: bridge
proxy:
external:
name: nginx-proxy
secrets:
host_ssh_key:
file: ~/.ssh/id_rsa
My question is: is there anyway I could get this secret within my data volume?
I know this is a fairly old thread but a lot of people get stuck on this including me and the answer is simply not true. You can indeed use secrets with docker-compose without using Swarm provided it's a local machine or the secrets file is mounted on the host. Not saying this is secure or desirable, just that it can be done. One of the best explanations of the several ways this is possible is this blog;
Using Docker Secrets during Development
Below is an example of parts of a docker compose file used to add an api key to a Spring application. The key are then available at /run/secrets/captcha-api-key inside the Docker container. Docker compose "fakes" it by literally binding the file as a mount which then can be accessed in whatever way. It's not secure as in the file is still there, visible to all with access to /run/secrets but it's definitely doable as a work-around. Great for dev servers but would not do it in production though;
version: '3.6'
services:
myapp:
image: mmyapp
restart: always
secrets:
- captcha-api-key
secrets:
captcha-api-key:
file: ./captcha_api_key.txt
EDIT: Besides that, one can simply just run a one-node swarm which is just a tiny bit more on the resources and use secrets the way they are intended. Provided the images are already built, "docker stack deploy mydocker-composefile.yml mystackname" will do mostly the same as old docker compose did. Note though that the yml file must be written in 3 or higher specification.
Here is a short but concise write-up on compose vs swarm; The Difference Between Docker Compose And Docker Stack
mount the secret like given and try.
secrets:
- source: host_ssh_key
target: /var/jenkins_home/.ssh/id_rsa
mode: 0600
It can't be done. Secrets will only work with docker swarm; docker-compose is unable to use secrets.
More details in this GitHub issue.
Let's say we have the following docker-compose.yml:
version: '3'
services:
db:
image: "postgres"
ports:
- "5432:5432"
environment:
- POSTGRES_PASSWORD=mysecretpassword
web:
build: web
depends_on: [ db ]
ports:
- "80:80"
The first service, db, just runs a container with the official postgres image from Docker Hub.
The second service, web, first builds a new image based on the Dockerfile in a folder also called web, then runs a container with that image.
While developing, we now can (repeatedly) make changes to whatever is in the web folder, then run docker-compose up --build to run our app locally.
Let's say we now want to deploy to production. My understanding is that docker-compose.yml can now be used to "define a stack in Docker's swarm mode" (see this answer, for instance). However, for the build step of the web service, Docker's compose file documentation states that
This option is ignored when deploying a stack in swarm mode with a (version 3) Compose file. The docker stack command accepts only pre-built images.
It probably wouldn't be a great idea to build the image on the production machine anyways, as this would leave build artifacts (source code) behind; this should happen on a build server.
My question is, is there a recommended way to modify docker-compose.yml en route to production to swap out build: web with image: <id> somehow?
Nothing on Use Compose in production on that. Is there something wrong with my approach in general?
docker-compose.yml should only contain canonical service definitions.
Anything that's specific to the build environment (e.g. dev vs prod) should be declared in a separate file docker-compose.override.yml. Each build environment can have its own version of that file.
The build: web declaration doesn't belong into docker-compose.yml, as it's only supposed to run locally (and possibly on a build server), not in production.
Therefore, in the example above, this is what docker-compose.yml should look like:
version: '3'
services:
db:
image: "postgres"
ports:
- "5432:5432"
environment:
- POSTGRES_PASSWORD=mysecretpassword
web:
depends_on: [ db ]
ports:
- "80:80"
And this would be the default docker-compose.override.yml for local development:
version: '3'
services:
web:
build: web
Running docker-compose up --build -d will now build the latest code changes and launch our app locally.
There could also be another version docker-compose.override.build.yml, targeting a build/CI server:
version: '3'
services:
web:
build: web
image: mydockeruser/web
Running docker-compose -f docker-compose.yml -f docker-compose.override.build.yml push will build the latest code changes and push the image to its registry/repository.
Finally, there could be another version docker-compose.override.prod.yml:
version: '3'
services:
web:
image: mydockeruser/web
Deploying to production (just to a single Docker host, not a cluster) can now be as simple as copying over only docker-compose.yml and docker-compose.override.prod.yml and running docker-compose -f docker-compose.yml -f docker-compose.override.prod.yml up -d.
The correct way to do it (i.e. the way I do it :P) is to have different docker-compose files; for example, docker-compose.dev.yml and docker-compose.prod.yml. You can then push your production-ready image to a repository, say Docker Hub, and reference that image in docker-compose.prod.yml's web service. All the while you can use the dev docker-compose file (the one with the build option) for local development.
Also, in case you've thought about this, you cannot use env variables as keys in docker-compose (see here). So there is no way to conditionally set either image or build options.