If I understand it correctly, in order to use secrets properly I need to use docker swarm.
Once I did a 'docker swarm init', portainer noticed the difference and put back everything in the swarm : running containers, existing stacks, etc..
However after adding a secret to the secrets section now available in portainer, a stack I am trying to setup cannot find the corresponding secret.
Here is the compose : https://pastebin.com/H1wnBLjy
Here is the secrets page :
And if I try running ls /run/secrets/ in the container I get this error :
Error response from daemon: Container xxx is restarting, wait until the container is running
The logs keep repeating this :
Loading configuration from /wiki/config.yml... OK
DB_PASS_FILE is defined. Will use secret from file.
Failed to read Docker Secret File using path defined in DB_PASS_FILE env variable!
ENOENT: no such file or directory, open '/run/secrets/db_passwd'
I tried removing the containers then setting them up again, restarting them, nothing works so far.
For info, it is run on a Swarm 20.10.7 with portainer 2.6.3, on a debian buster host.
What have I done wrong ?
Thanks for your help.
Well my bad, thing is if an env var is not explicitely defined it cannot be used.
I thought adding _FILE to any env was enough to make it understands that it will be passed through a file, but I learned that it is not the case.
So secrets are available in general, but if not defined in the image it is a no-go.
Related
The newer docker compose (vs docker-compose) allows you to set secrets in the build section. This is nice because if you do secrets at runtime then the file is readable by anyone that can get into the container by reading /run/secrets/<my_secret>.
Unfortunately, it appears that it's only possible to pass the secrets via either the environment or a file. Doing it via the environment doesn't seem like a great idea because someone on the box could read the /proc/<pid>/environment while the image is being built to snag the secrets. Doing it via a file on disk isn't good because then the secret is being stored on disk unencrypted.
It seems like the best way to do this would be with something like
docker swarm init
$(read -sp "Enter your secret: "; echo $REPLY) | docker secret create my_secret -
docker compose build --no-cache
docker swarm leave --force
Alas, it appears that Docker can't read from the swarm for build time secrets for some unknown reason.
What is the best way to do this? This seems to be a slight oversight, along the lines of docker secrete create not having a way to prompt for the value instead of having to resort to to hacks like above to keep the secret out of your bash history.
UPDATE: This is for SWARM/Remote docker systems, not targeted on local build time secrets. (I realised you were asking for those primarily and just mentioned swarm in the second part of the question. I believe it still holds good advice for some so ill leave the answer undeleted.
Docker Swarm can only read runtime-based secrets you create with the docker secret create command and must already exist on the cluster when deploying stack. We had been in the same situation before. We solved the "issue" using docker contexts. You can create an SSH-based docker context which points to a manager (we just use the first one). Then on your LOCAL device (we use Win as the base platform and WSL2/Linux VM for the UNIX part), you can simply run docker commands with inline --context property. More on context on official docs. For instance: docker --context production secret create .... And so on.
I used command "docker pull mysql:5.7.28" which showed image and container correctly in docker desktop but when trying to run the container it showed exited and error was MYSQL_ROOT_PASSWORD required.
So I need to edit MYSQL_ROOT_PASSWORD in yaml file to resolve this issue.
Now the problem is simple I have not used docker-compose file to setup the container and unable to find option in docker desktop to set up this variable.
You can set the environment variable when you run the container with docker run - see, e.g. "Start a mysql server instance" on https://hub.docker.com/_/mysql.
An alternative would be to create a docker-compose.yml and set the environment variable there (the reference for what you can put in Compose files is here).
There might be a way to set environment variables in Docker Desktop, but I don't use it, so I don't know. The documentation should tell you, though.
how to configure docker containers proxy ?
First of all,
I tried to use the way that setted '/etc/systemd/system/docker.service.d/http-proxy.conf' (https://docs.docker.com/config/daemon/systemd/#httphttps-proxy) and it really works for docker daemon, but it doesn't work for docker containers, it seems this way just take effect for some command like 'docker pull'
Secondary,
I have a lot of docker containers, I don't want to use 'docker run -e http_proxy=xxx... ' command every time when I start a container.
So I guess if there is such a way automatically load the global configuration file when the container starts, I googled it and got it to set the file '~/.docker/config.json'(How to configure docker container proxy?, this way still does not work for me.
(
my host machine system is centos7, here is my docker -v:
Docker version 1.13.1, build 6e3bb8e/1.13.1
)
I feel that it may be related to my docker version or the docker started by the systemd service, so ~/.docker/config.json does not take effect.
Finally ,
I just hope that modifying configuration files will allow all my containers to automatically configure environment variables when it start (that is auto set environment variables 'http_proxy=http://HostIP:8118 https_proxy=http://HostIP:8118' when a container start, like Dockerfile param ENV) . I want to know if there is such a way? And if this way can be realised I can make the container use the host's proxy, after all, my host's agent is working properly.
But I was wrong, I tried to run a container,then set http_proxy=http://HostIP:8118 and https_proxy=http://HostIP:8118, but when I use the command 'wget facebook.com' and I got 'Connecting to HostIP:8118... failed: No route to host.', But, the host machine(centos7) can successfully execute the wget, And I can successfully ping the host in the container. I don't know why it might be related to firewalls and the 8118 port.
It is Over,
OMG.. I have no other way, can anyone help me?
==============================
ps:
You can see from the screenshot below, I actually want to install goa and goagen but report an error, maybe because of network reasons, I want to open the agent to try, so...only have the above problem.
1.my go docker container
enter image description here
go docker wget
2.my host
my host wget
You need version 17.07 or more recent to automatically pass the proxy to containers you start using the config.json file. The 1.13 releases are long out of support.
This is well documented from docker:
https://docs.docker.com/network/proxy/
I have a Jenkins instance setup using Googles Jenkins on Kubernetes solution. I have not changed any of the settings of the Kubernetes Pod.
When I trigger a new job I am successfully able to get everything up and running until the point of my tests.
My tests use docker-compose. First I make sure to install docker (1.5-1+b1) and docker-compose (1.8.0-2) on the instance (I know I can optimize this by using an image that already includes these, but I am still just in proof-of-concept).
When I run the docker-compose up command everything works and the services start their initialization scripts. However, the mounts are empty. I have verified that the files exist on the Jenkins slave, and the mount is created inside the docker service when I run docker-compose, however they are empty.
Some information:
In order to get around file permissions I am using /tmp as the Jenkins Workspace. I am using SCM to pull my files (successfully) and in the docker-compose file I specify version: '2' and the mount paths with absolute paths. The volume section of the service that fails looks like this:
volumes:
- /tmp/automation:/opt/automation
I changed the command that is run in the service to ls /opt/automation and the result is an empty directory.
What am I missing? I just want to mount a directory into my docker-compose service. This works perfectly from Windows, Ubuntu, and Centos devices. Why won't it work using the Kubernetes instance?
I found the reason it fails here:
A Docker container in a Docker container uses the parent HOST's Docker daemon and hence, any volumes that are mounted in the "docker-in-docker" case is still referenced from the HOST, and not from the Container.
Therefore, the actual path mounted from the Jenkins container "does not exist" in the HOST. Due to this, a new directory is created in the "docker-in-docker" container that is empty. Same thing applies when a directory is mounted to a new Docker container inside a Container.
So it seems like it will be impossible to mount something from the outer docker into the inner docker. And another solution must be found.
I have monolithic application that i am trying to containerize. The foler structure is like this:
--app
|
|-file.py <-has a variable foo that is passed in
--configs
|
|-variables.py <- contains foo variable
Right now, I have the app in a container and the configs in a container. When I try to start up the app container, it fails because a dependency on the config container variable.
What am i doing wrong? And how should I approach this issue. Should the app and config be in one big container for now?
I was thinking docker-compose could solve this issue. Thoughts?
The variables.py file could be (in) a volume accessed by the app container that you import from the config container with --volumes-from config option to docker run. With Docker Compose you would use the volumes_from directive.
Less recommended way -
Run the Config Container first, it will have its own docker.sock.
You can mount the above Docker Socket in first app Container via -v "/var/run/docker.sock:/var/run/docker.sock", which will let the App container access the Config container, but I think this will need privileged access.
This is similar to Docker in Docker concept.
You can also consider design changes to your application by serving that foo variable over HTTP, which will result in much simpler solution. You can use simple web server and urllib3 module in Python to have a simple solution which will serve the variable via Internal Docker Networking.