I have setup an instance of Hashicorp Vault. I have successfully written and read secrets to and from it. Getting Vault up and running and is the easy part. Now, how do I use Vault as a store to replace the .env file in docker-compose.yml? How do I read secrets from Vault in all of my docker-compose files?
Even more difficult: how do I dynamically generate keys to access access the secrets in Vault, then use those keys in my docker-compose.yml files, without editing those files each time I restart a stack? How is that process automated? In short, just exactly how can I leverage Hashicorp Vault to secure the secrets that are otherwise exposed in the .env files?
I have read all of their literature and blog posts, and haven't been able to find anything that outlines that process. I am stuck and any tips will be greatly appreciated.
Note: This is not a question about running a Hashicorp Vault container with docker-compose, I have successfully done that already.
Also Note: I cannot modify the containers themselves; I can only modify the docker-compose.yml file
You would need to query the vault API to populate either your .env file or in the entrypoint of your container. My preference would be the container entrypoint at worst, and ideally directly in your application. The reason is because vault secrets could be short lived, and any container running for longer than that period would need to refresh it's secrets.
If you go with the worst case of doing this in the entrypoint, there are a few tools that come to mind. confd from Kelsey Hightower, and gomplate.
confd can run as a daemon and restart your app inside the container when the configuration changes. My only concern is that it is an older and less maintained project.
gomplate would be run by your entrypoint to expand a template file with the needed values. That file could just be an env.sh that you then source into your environment if you needed env vars. Or you can run it within your command line as a subshell, e.g.
your-app --arg "$(gomplate ...sometemplate...)"
If you only use these tools to set the value once and then start your app, make sure to configure a healthcheck and/or graceful exit your app when the credentials expire. Then run your container with orchestration (Kubernetes/Swarm Mode) or set a restart policy so that it restarts after any credentials expire to get the new credentials.
Related
We are pre building the Shopware 6 code (composer install, storefront + admin build), but not the theme build and copy it into a docker container.
What is the best way to generate or supply the JWT secrets when running such a prebuilt container.
Normally we would do a
bin/console secrets:generate-keys
bin/console system:generate-jwt-secret
on the first installation.
But can this secrets also be kept in an ENV variable to avoid the need for a persitent /var volume?
You can override secrets locally as described here.
So in theory:
Run secrets:generate-keys to generate keys once.
Run secrets:decrypt-to-local to get the secrets added to your env file.
Run secrets:encrypt-from-local on deployment to set secrets from your env your file.
The newer docker compose (vs docker-compose) allows you to set secrets in the build section. This is nice because if you do secrets at runtime then the file is readable by anyone that can get into the container by reading /run/secrets/<my_secret>.
Unfortunately, it appears that it's only possible to pass the secrets via either the environment or a file. Doing it via the environment doesn't seem like a great idea because someone on the box could read the /proc/<pid>/environment while the image is being built to snag the secrets. Doing it via a file on disk isn't good because then the secret is being stored on disk unencrypted.
It seems like the best way to do this would be with something like
docker swarm init
$(read -sp "Enter your secret: "; echo $REPLY) | docker secret create my_secret -
docker compose build --no-cache
docker swarm leave --force
Alas, it appears that Docker can't read from the swarm for build time secrets for some unknown reason.
What is the best way to do this? This seems to be a slight oversight, along the lines of docker secrete create not having a way to prompt for the value instead of having to resort to to hacks like above to keep the secret out of your bash history.
UPDATE: This is for SWARM/Remote docker systems, not targeted on local build time secrets. (I realised you were asking for those primarily and just mentioned swarm in the second part of the question. I believe it still holds good advice for some so ill leave the answer undeleted.
Docker Swarm can only read runtime-based secrets you create with the docker secret create command and must already exist on the cluster when deploying stack. We had been in the same situation before. We solved the "issue" using docker contexts. You can create an SSH-based docker context which points to a manager (we just use the first one). Then on your LOCAL device (we use Win as the base platform and WSL2/Linux VM for the UNIX part), you can simply run docker commands with inline --context property. More on context on official docs. For instance: docker --context production secret create .... And so on.
I have .env file for my docker-compose, and was able to run using "docker-compose up"
Now I pushed to cloud registry, and want to Cloud Run
How can I supply the various environemnt variables?
I did create secrets in secret manager, but how can I integrate both, so that my container starts reading all those needed secrets?
Note: My docker-compose is an app with database, but I can split them as 2 containers, if needed, but they still need secrets
Edit: Added secret references.
EDIT:
I am unable to run my container
If env file X=x , and docker-compose environemnt app.prop=${X}
then should I create secret X or x?
Is Cloud run using Dockerfile or docker-compose? I image pushed is built from docker-compose only. Sorry I am getting confused (not assuming trivial things as it is not working)
It is not possible to use docker-compose on Cloud Run, as it is designed for individual stateless containers. My suggestion is to create an image from your application service, upload the image to Google Container Registry in order to use it for your Cloud Run service, and connect it to Cloud SQL following the attached documentation. You can store database credentials with Secret Manager and pass it to your Cloud Run service as environment variables (check this documentation).
I am running an OPC UA Server in a Docker container. The OPC UA Server is connecting to a Cloud Service via a ID and a secret that is stored in a config-File. Furthermore the OPC UA Server holds SSH certificates for authentication.
I see a problem when releasing the image to a work group, because everyone would have access to my personal login, and to SSH certificates that were supposed to be unique to the Host that is running the image.
What would be the appropriate way to inject the certificates and the config-Files into a image, without building the whole thing again?
You have two main ways to pass configuration information into a container at runtime:
Use an environment variable for simple string values.
Use a volume. You can use a pure Docker volume, but a bind-mounted volume is often useful for things like key stores. Bind-mounted volumes share a file or directory from the host's filesystem into a specific location on the container filesystem.
Either way, you may need to inject the value into the right place in your config file. Sometimes config files can have variables from the environment - if not, then you can make your container entrypoint run a script to update the configuration file, and then exec your true entry point.
I put the docker in swarm mode and did the following
echo "'admin'" | docker secret create password -
docker service create \
--network="host" \
--secret source=password,target=password \
-e PASSWORD='/run/secrets/password' \
<image>
I was not able to pass the password secret created via environment variable through docker service.
Please help me out where I am going wrong.
You are misunderstanding the concept of docker secrets.
The whole point of creating secrets is avoiding putting sensitive information into environment variables.
In your example the PASSWORD environment variable will simply carry the value /run/secrets/password which is a file name and not the password admin.
A valid usacase of docker secrets would be, that your docker-image reads the password from that file.
Checkout the docs here especially the example about MySQL:
the environment variables MYSQL_PASSWORD_FILE and MYSQL_ROOT_PASSWORD_FILE to point to the files /run/secrets/mysql_password and /run/secrets/mysql_root_password. The mysql image reads the password strings from those files when initializing the system database for the first time.
In short: your docker image should read the content of the file /run/secrets/password
There is no standard here.
Docker docs discourages using environment variables, but there is confusion whether it is setting password directly as string in "environment" section or other usage of environment variables within container.
Also using string instead of secret when same value might be used in multiple services requires checking and changing it in multiple places instead of one secret value.
Some images, like mariadb, is using env variables with _FILE suffix to populate suffixless version of variable with secret file contents. This seems to be ok.
Using Docker should not require to redesign application architecture only to support secrets in files. Most of other orchestration tools, like Kubernetes, supports putting secrets into env variables directly. Nowadays it is rather not considered as bad practice. Docker Swarm simply lacks good pracitces and proper examples for passing secret to env variable.
IMHO best way is to use entrypoint as a "decorator" to prepare environment from secrets.
Proper entrypoint script can be written as almost universal way of processing secrets, because we can pass original image entrypoint as argument to our new entrypoint script so original image "decorator" is doing it's own work after we prepare container with our script.
Personally I am using following entrypoint with images containing /bin/sh:
https://github.com/DevilaN/docker-entrypoint-example