I wonder how can i pass external secrets into services spawned by docker-compose. I do the following:
I create new secret
printf "some secret value goes here" | docker secret create wallet_password -
My docker-compose.yml:
version: "3.4"
services:
test:
image: alpine
command: 'cat /run/secrets/wallet_password'
secrets:
- wallet_password
secrets:
wallet_password:
external: true
Then I run:
docker-compose -f services/debug/docker-compose.yml up -d --build
and
docker-compose -f services/debug/docker-compose.yml up
I get the following response:
WARNING: Service "test" uses secret "wallet_password" which is external. External secrets are not available to containers created by docker-compose.
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
Starting debug_test_1 ...
Starting debug_test_1 ... done
Attaching to debug_test_1
test_1 | cat: can't open '/run/secrets/wallet_password': No such file or directory
Sooo.... is there any way of passing external secret into container spawned by docker-compose?
Nope.
External secrets are not available to containers created by docker-compose.
The error message sums it up pretty nicely. Secrets are a swarm mode feature, the secret is stored inside of the swarm manager engine. That manager does not expose those secrets to externally launched containers. Only swarm services with the secret can run containers with the secret loaded.
You can run a service in swarm mode that extracts the secret since it's just a file inside the container and the application inside the container can simply cat out the file contents. You can also replicate the functionality of secrets in containers started with compose by mounting a file as a volume in the location of the secret. For that, you'd want to have a separate compose file since the volume mount and secret mount would conflict with each other.
You need to run a swarm. This is how it goes:
Create a swarm:
docker swarm init
Create your secrets (as many as you need):
docker secret create <secret_name> <secret_content>
Check all the available secrets with:
docker secret ls
Now, use the docker-compose as precursor for the service:
docker stack deploy --compose-file <path_to_compose> <service_name>
Be aware that you'll find your secrets in a plain text file located at /run/secrets/<secret_name>.
Related
The project I am working on consists of multiple components, written in C#/.NET6, and deployed as docker containers on a Linux host. Each component has his own git repository on Gitlab and its Gitlab pipeline is building the docker container to the Gitlab 'container registry'. For instance, one component is called "runtime", another one "services", etc.
All docker containers are defined in in a docker-compose.yml file: the suite is started with a "docker compose up" command.
I created a 'integration test' project to check the data exchange between the running containers. I've a lot of complex Linux shell scripts to prepare the mock data and so on for the tests. I've a bunch of tests written in Python running in a py-env on the Linux host and also some other tests written in C# running in a dedicated docker container.
I've actually different test scenarios or test group: each group has his own docker-compose-integration_$group.yml file to set up e.g. the mocked services.
All of this is run with
docker compose -f docker-compose.yml -f docker-compose-integration_$group.yml up -d
In multiple services defined in the docker compose file, I set up docker volumes to be able to check the data generated by the containers within my tests. For instance, the following is an extract of my docker-compose-integration_4.yml file for the 4th group of tests using the C# written tests running in a dedicated 'integration-tests-dotnet' container:
runtime:
extends:
file: ./docker-compose.yml
service: runtime
volumes:
- ./runtime/config_4.ini}:/etc/runtime/config.ini
- ${OUTPUT:-./output}/runtime/:/runtime/output/
integration-tests-dotnet:
volumes:
# share config for current group same as for runtime.
- ./runtime/config_4.ini}:/etc/runtime/config.ini
# share output folder from runtime.
- ./testData/:/opt/testData/
- ./output/runtime/:/opt/runtime/output/
- ./output/services/:/opt/services/output/
# share report file generated from the tests.
- ./output/integration/:/app/output/
Everything is running nicely on a Linux machine, or on WSL2 in my Windows PC (or on the Mac of a colleague).
The integration-test project has his own Gitlab pipeline.
Now, we would like to be able to run the integration tests within the Giltab pipeline, i.e. run "docker compose" from a Gitlab runner.
I do have already a 'docker in docker' capable runner, and added such a job in my .gitlab-ci.yml
run-integration-tests:
stage: integration-tests
variables:
DOCKER_TLS_CERTDIR: ''
DOCKER_HOST: tcp://localhost:2375/
services:
- name: docker:20.10.22-dind
command: ["--tls=false"]
tags:
- dind
image: $CI_REGISTRY_IMAGE:latest
This job is starting properly, BUT fails with the volume sharing.
This question Docker in Docker cannot mount volume raised already the issue with volume sharing by using the shared docker socket. The docker volume is shared from the HOST (i.e. on my runner). But the data are unknown from my host: they are only meant to be shared between the integration-test and the other containers.
As Olivier wrote in that question, for a
host: H
docker container running on H: D
docker container running in D: D2
the docker compose with volume sharing is equivalent to
docker run ... -v <path-on-D>:<path-on-D2> ...
while I only something equivalent to the following can run:
docker run ... -v <path-on-H>:<path-on-D2> ...
But I have no data on H to share, I just want between D and D2!
Is the volume sharing on HOST limitation the same using my docker-in-docker runner as the shared socket?
If so, it seems I need to rework the infrastructure and the concept of volume sharing used here.
Some suggests a Docker data volume containers.
Maybe I should use more of the names volumes.
Mayber tmpfs volumes? I need to check the data AFTER some container are exited, but I don't know if a container still running but in "exited" status has sill the tmpfs volume activated.
Is my analysis correct?
Any other suggestions?
I am playing around with a single container docker image. I would like to store my db password as a secret without using compose (having probs with that and Gradle for now). I thought I could still use secrets even without compose but when I try I get...
$ echo "helloSecret" | docker secret create helloS -
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Why do I need to use swarm mode just to use secrets? Why can't I use them without a cluster?
You need to run swarm mode for secrets because that's how docker implemented secrets. The value of secrets is that workers never write the secret to disk, the secret is on a need-to-know basis (other workers do not receive the secret until a task is scheduled there), and on managers encrypt that secret on disk. The storage of the secret on the manager uses the raft database.
You can easily deploy a single node swarm cluster with the command docker swarm init. From there, docker-compose up gets changed to docker stack deploy -c docker-compose.yml $stack_name.
Secrets and configs in swarm mode provide a replacement for mounting single file volumes into containers for configuration. So without swarm mode on a single node, you can always make the following definition:
version: '2'
services:
app:
image: myapp:latest
volumes:
- ./secrets:/run/secrets:ro
Or you can separate the secrets from your app slightly by loading those secrets into a named volume. For that, you could do something like:
tar -cC ./secrets . | docker run -i -v secrets:/secrets busybox tar -xC /secrets
And then mount that named volume:
version: '2'
volumes:
secrets:
external: true
services:
app:
image: myapp:latest
volumes:
- secrets:/run/secrets:ro
Check out this answer: https://serverfault.com/a/936262 as provided by user sel-en-ium :-
You can use secrets if you use a compose file. (You don't need to run
a swarm).
You use a compose file with docker-compose: there is documentation for
"secrets" in a docker-compose.yml file.
I switched to docker-compose because I wanted to use secrets. I am
happy I did, it seems much more clean. Each service maps to a
container. And if you ever want to switch to running a swarm instead,
you are basically already there.
Unfortunately the secrets are not loaded into the container's
environment, they are mounted to /run/secrets/
Is it possible to execute commands on a docker swarm cluster hosted in cloud from my local mac? If yes, how?
I want to execute command such as following on docker swarm from my local:
docker create secret my-secret <address to local file>
docker service create --name x --secrets my-secret image
Answer to the question can be found here.
What one needs to do for ubuntu machine is define daemon.json file at path /etc/docker with following content:
{
"hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]
}
The above configuration is unsecured and shouldn't be used if server is publicly hosted.
For secured connection use following config:
{
"tls": true,
"tlscert": "/var/docker/server.pem",
"tlskey": "/var/docker/serverkey.pem",
"hosts": ["tcp://x.x.x.y:2376", "unix:///var/run/docker.sock"]
}
Details for generating certificate can be found here as mentioned by #BMitch.
One option is to provide direct access to the docker daemon as suggested in the previous answers, but that requires setting up TLS certificates and keys, which can itself be tricky and time consuming. Docker machine can automate that process, when docker machine has been used to create the nodes.
I had the same problem, in that I wanted to create secrets on the swarm without uploading the file containing the secret to the swarm manager. Also, I wanted to be able to run the deploy stackfile (e.g. docker-compose.yml) without the hassle of first uploading the stackfile.
I wanted to be able to create the few servers I needed on e.g. DigitalOcean, not necessarily using docker machine, and be able to reproducibly create the secrets and run the stackfile. In environments like DigitalOcean and AWS, a separate set of TLS certificates is not used, but rather the ssh key on the local machine is used to access the remote node over ssh.
The solution that worked for me was to run the docker commands using individual commands over ssh, which allows me to pipe the secret and/or stackfile using stdin.
To do this, you first need to create the DigitalOcean droplets and get docker installed on them, possibly from a custom image or snapshot, or simply running the commands to install docker on each droplet. Then, join the droplets into a swarm: ssh into the one that will be the manager node, type docker swarm init (possibly with the --advertise-addr option if there is more than one IP on that node, such as when you want to keep intra-swarm traffic on the private network) and get back the join command for the swarm. Then ssh into each of the other nodes and issue the join command, and your swarm is created.
Then, export the ssh command you will need to issue commands on the manager node, like
export SSH_CMD='ssh root#159.89.98.121'
Now, you have a couple of options. You can issue individual docker commands like:
$SSH_CMD docker service ls
You can create a secret on your swarm without copying the secret file to the swarm manager:
$SSH_CMD docker create secret my-secret - < /path/to/local/file
$SSH_CMD docker service create --name x --secrets my-secret image
(Using - to indicate that docker create secret should accept the secret on stdin, and then piping the file to stdin using ssh)
You can also create a script to be able to reproducibly run commands to create your secrets and bring up your stack with secret files and stackfiles only on your local machine. Such a script might be:
$SSH_CMD docker secret create rabbitmq.config.01 - < rabbitmq/rabbitmq.config
$SSH_CMD docker secret create enabled_plugins.01 - < rabbitmq/enabled_plugins
$SSH_CMD docker secret create rmq_cacert.pem.01 - < rabbitmq/cacert.pem
$SSH_CMD docker secret create rmq_cert.pem.01 - < rabbitmq/cert.pem
$SSH_CMD docker secret create rmq_key.pem.01 - < rabbitmq/key.pem
$SSH_CMD docker stack up -c - rabbitmq_stack < rabbitmq.yml
where secrets are used for the certs and keys, and also for the configuration files rabbitmq.config and enabled_plugins, and the stackfile is rabbitmq.yml, which could be:
version: '3.1'
services:
rabbitmq:
image: rabbitmq
secrets:
- source: rabbitmq.config.01
target: /etc/rabbitmq/rabbitmq.config
- source: enabled_plugins.01
target: /etc/rabbitmq/enabled_plugins
- source: rmq_cacert.pem.01
target: /run/secrets/rmq_cacert.pem
- source: rmq_cert.pem.01
target: /run/secrets/rmq_cert.pem
- source: rmq_key.pem.01
target: /run/secrets/rmq_key.pem
ports:
# stomp, ssl:
- 61614:61614
# amqp, ssl:
- 5671:5671
# monitoring, ssl:
- 15671:15671
# monitoring, non ssl:
- 15672:15672
# nginx here is only to show another service in the stackfile
nginx:
image: nginx
ports:
- 80:80
secrets:
rabbitmq.config.01:
external: true
rmq_cacert.pem.01:
external: true
rmq_cert.pem.01:
external: true
rmq_key.pem.01:
external: true
enabled_plugins.01:
external: true
(Here, the rabbitmq.config file sets up the ssh listening ports for stomp, amqp, and the monitoring interface, and tells rabbitmq to look for the certs and key within /run/secrets. Another alternative for this specific image would be to use the environment variables provided by the image to point to the secrets files, but I wanted a more generic solution that did not require configuration within the image)
Now, if you want to bring up another swarm, your script will work with that swarm once you have set the SSH_CMD environment variable, and you need neither set up TLS nor copy your secret or stackfiles to the swarm filesystem.
So, this doesn't solve the problem of creating the swarm (whose existence was presupposed by your question), but once it is created, using an environment variable (exported if you want to use it in scripts) will allow you to use almost exactly the commands you listed, prefixed with that environment variable.
This is the easiest way of running commands on remote docker engine:
docker context create --docker host=ssh://myuser#myremote myremote
docker --context myremote ps -a
docker --context myremote create secret my-secret <address to local file>
docker --context myremote service create --name x --secrets my-secret image
or
docker --host ssh://myuser#myremote ps -a
You can even set the remote context as default and issue commands as if it is local:
docker context use myremote
docker ps # lists remote running containers
In this case you don't even need to have docker engine installed, just docker-ce-cli.
You need to use key based authentication for this do work (you should already be using it). Other options include setting up tls cert based socket, or ssh tunnels.
Also, consider setting up ssh control socket to avoid re-authenting on each command, so your commands will run faster, as it was local.
To connect to a remote docker node, you should setup TLS on both the docker host and client signed from the same CA. Take care to limit what keys you sign with this CA since it is used to control access to the docker host.
Docker has documented the steps to setup a CA and create/install the keys here: https://docs.docker.com/engine/security/https/
Once configured, you can connect to the newer swarm mode environments using the same docker commands you run locally on the docker host just by changing the value of $DOCKER_HOST in your shell.
If you start from scratch, you can create the manager node using a generic docker-machine driver. Afterwards you will be able to connect to that docker engine from your local machine with the help of docker-machine env command.
Version 3.1 of the docker-compose.yml specification introduces support for secrets.
I tried this:
version: '3.1'
services:
a:
image: tutum/hello-world
secret:
password: the_password
b:
image: tutum/hello-world
$ docker-compose up returns:
Unsupported config option for services.secret: 'password'
How can we use the secrets feature in practice?
You can read the corresponding section from the official documentation.
To use secrets you need to add two things into your docker-compose.yml file. First, a top-level secrets: block that defines all of the secrets. Then, another secrets: block under each service that specifies which secrets the service should receive.
As an example, create the two types of secrets that Docker will understand: external secrets and file secrets.
1. Create an 'external' secret using docker secret create
First thing: to use secrets with Docker, the node you are on must be part of a swarm.
$ docker swarm init
Next, create an 'external' secret:
$ echo "This is an external secret" | docker secret create my_external_secret -
(Make sure to include the final dash, -. It's easy to miss.)
2. Write another secret into a file
$ echo "This is a file secret." > my_file_secret.txt
3. Create a docker-compose.yml file that uses both secrets
Now that both types of secrets are created, here is the docker-compose.yml file that will read both of those and write them to the web service:
version: '3.1'
services:
web:
image: nginxdemos/hello
secrets: # secrets block only for 'web' service
- my_external_secret
- my_file_secret
secrets: # top level secrets block
my_external_secret:
external: true
my_file_secret:
file: my_file_secret.txt
Docker can read secrets either from its own database (e.g. secrets made with docker secret create) or from a file. The above shows both examples.
4. Deploy your test stack
Deploy the stack using:
$ docker stack deploy --compose-file=docker-compose.yml secret_test
This will create one instance of the web service, named secret_test_web.
5. Verify that the container created by the service has both secrets
Use docker exec -ti [container] /bin/sh to verify that the secrets exist.
(Note: in the below docker exec command, the m2jgac... portion will be different on your machine. Run docker ps to find your container name.)
$ docker exec -ti secret_test_web.1.m2jgacogzsiaqhgq1z0yrwekd /bin/sh
# Now inside secret_test_web; secrets are contained in /run/secrets/
root#secret_test_web:~$ cd /run/secrets/
root#secret_test_web:/run/secrets$ ls
my_external_secret my_file_secret
root#secret_test_web:/run/secrets$ cat my_external_secret
This is an external secret
root#secret_test_web:/run/secrets$ cat my_file_secret
This is a file secret.
If all is well, the two secrets we created in steps 1 and 2 should be inside the web container that was created when we deployed our stack.
Given you have a service myapp and a secrets file secrets.yml:
Create a compose file:
version: '3.1'
services:
myapp:
build: .
secrets:
secrets_yaml
Provision a secret using this command:
docker secret create secrets_yaml secrets.yml
Deploy your service using this command:
docker deploy --compose-file docker-compose.yml myappstack
Now your app can access the secret file at /run/secrets/secrets_yaml. You can either hardcode this path in your application or create a symbolic link.
The different question
This answer is probably to the question "how do you provision your secrets to your docker swarm cluster".
The original question "how do you manage secret values with docker compose" implies that the docker-compose file contains secret values. It doesn't.
There's a different question: "Where do you store the canonical source of the secrets.yml file". This is up to you. You can store it in your head, print on a sheet of paper, use a password manager, use a dedicated secrets application/database. Heck, you can even use a git repository if it's safely secured itself. Of course, never store it inside the system you're securing with it :)
I would recommend vault. To store a secret:
# create a temporary secret file
cat secrets.yml | vault write secret/myappsecrets -
To retrieve a secret and put it into your docker swarm:
vault read -field=value secret/myappsecrets | docker secret create secrets_yaml -
Of course, you can use docker cluster itself as a single source of truth for you secrets, but if your docker cluster breaks, you'd lost your secrets. So make sure to have a backup elsewhere.
The question nobody asked
The third question (that nobody asked) is how to provision secrets to developers' machines. It might be needed when there's an external service which is impossible to mock locally or a large database which is impossible to copy.
Again, docker has nothing to do with it (yet). It doesn't have access control lists which specify which developers have access to which secrets. Nor does it have any authentication mechanism.
The ideal solution appears to be this:
A developer opens some web application.
Authenticates using some single sign on mechanism.
Copies some long list of docker secret create commands and executes them in the terminal.
We have yet to see if such an application pops up.
You can also specify secrets stored locally in a file using file: key in secrets object. Then you don't have to docker secret create them yourself, Compose / docker stack deploy will do it for you.
version: '3.1'
secrets:
password:
file: ./password
services:
password_consumer:
image: alpine
secrets:
- password
Reference: Compose file version 3 reference: Secrets
One question was raised here in the comments, why should I initialize a swarm if I only need secrets? And my answer is that secrets is created for the swarm, where you have more than one node and you want to manage and share secrets in a secure way. But if you have one node, this will not (almost) add any extra security if someone can access your host machine where you have the one node swarm, as secrets can be retrieved from the running containers, or directly on the host if the secret is created from a file, like a private key.
Check this blog: https://www.docker.com/blog/docker-secrets-management/
And read the comments:
"Thank you very much for the introductory article. The steps are mentioned to view the contents of secrets in container will not work when the redis container is created on a worker node."
Is that the exact indentation of your docker-compose.yml file? I think secret secrets should be nested under a (i.e. one of the services), not directly under services section.
I guess the keyword is secrets not secret. That is at least what I understand from reading the schema.
The keyword is secrets instead of secret.
It should also properly indented under service a.
docker and docker-compose seem to be interacting with the same dockerFile, what is the difference between the two tools?
The docker cli is used when managing individual containers on a docker engine. It is the client command line to access the docker daemon api.
The docker-compose cli can be used to manage a multi-container application. It also moves many of the options you would enter on the docker run cli into the docker-compose.yml file for easier reuse. It works as a front end "script" on top of the same docker api used by docker, so you can do everything docker-compose does with docker commands and a lot of shell scripting. See this documentation on docker-compose for more details.
Update for Swarm Mode
Since this answer was posted, docker has added a second use of docker-compose.yml files. Starting with the version 3 yml format and docker 1.13, you can use the yml with docker-compose and also to define a stack in docker's swarm mode. To do the latter you need to use docker stack deploy -c docker-compose.yml $stack_name instead of docker-compose up and then manage the stack with docker commands instead of docker-compose commands. The mapping is a one for one between the two uses:
Compose Project -> Swarm Stack: A group of services for a specific purpose
Compose Service -> Swarm Service: One image and it's configuration, possibly scaled up.
Compose Container -> Swarm Task: A single container in a service
For more details on swarm mode, see docker's swarm mode documentation.
docker manages single containers
docker-compose manages multiple container applications
Usage of docker-compose requires 3 steps:
Define the app environment with a Dockerfile
Define the app services in docker-compose.yml
Run docker-compose up to start and run app
Below is a docker-compose.yml example taken from the docker docs:
services:
web:
build: .
ports:
- "5000:5000"
volumes:
- .:/code
- logvolume01:/var/log
links:
- redis
redis:
image: redis
volumes:
logvolume01: {}
A Dockerfile is a text document that contains all the commands/Instruction a user could call on the command line to assemble an image.
Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. By default, docker-compose expects the name of the Compose file as docker-compose.yml or docker-compose.yaml. If the compose file has a different name we can specify it with -f flag.
Check here for more details
docker or more specifically docker engine is used when we want to handle only one container whereas the docker-compose is used when we have multiple containers to handle. We would need multiple containers when we have more than one service to be taken care of, like we have an application that has a client server model. We need a container for the server model and one more container for the client model. Docker compose usually requires each container to have its own dockerfile and then a yml file that incorporates all the containers.