I've been struggling with this concept. To start I'm new to docker and self teaching myself (slowly). I am using a docker swarm instance and trying to leverage docker secrets for a simple username and password to an exiting rocker/rstudio image. I've set up the reverse proxy and can successfully use https to access the R studio via my browser. Now when I pass the variables at path /run/secrets/user and /run/secrets/pass to the environment variables it doesn't work. Its essentially think the path is the actual username and password. I need the environment variables to actually pull the values (in this case user=test, pass=test123 as set up using the docker secret command). I've looked around and a bit of a loss on how to accomplish this. I know some have mentioned leveraging a custom entrypoint shell script and I'm a bit confused on how to do this. Here is what I've tried
Rebuild a brand new image using the existing r image with a dockerfile that adds entrypoint.sh to the image -> it can't find the entrypoint.sh doc
added entrypoint: entrypoint.sh as a part of my docker compose. Same issue.
I'm trying to use docker stack to build the containers. The stack gets built but the containers keep restarting to the point they are unusable.
Here are my files
Dockerfile
FROM rocker/rstudio
COPY entry.sh /
RUN chmod +x /entry.sh
ENTRYPOINT ["entry.sh"]
Here is my docker-compose.yaml
version: '3.3'
secrets:
user:
external: true
pass:
external: true
services:
rserver:
container_name: rstudio
image: rocker/rstudio:latest (<-- this is the output of the build using rocker/rstudio and Dockerfile)
secrets:
- user
- pass
environment:
- USER=/run/secrets/user
- PASSWORD=/run/secrets/pass
volumes:
- ./rstudio:/home/user/rstudio
ports:
- 8787:8787
restart: always
entrypoint: /entry.sh
Finally here is the entry.sh file that I found on another thread
#get your envs files and export envars
export $(egrep -v '^#' /run/secrets/* | xargs)
#if you need some specific file, where password is the secret name
#export $(egrep -v '^#' /run/secrets/password| xargs)
#call the dockerfile's entrypoint
source /docker-entrypoint.sh
In the end it would be great to use my secret user and pass and pass those to the environment variable so that I can authenticate into an R studio instance. If I just put a username and password in plain text under environment it works fine.
Any help is appreciated. Thanks in advance
Related
UPDATE
It appears the problem is specifically related to the RUN command in the Dockerfile. If I remove it, the build works fine and the environment variables are clearly being picked up since the password gets applied and I can connect using it. Not sure why the login fails in the RUN command, I've seen many examples using similar code.
I'm working on a very basic docker compose file to setup a dev environment for an app and I started with the database server, which is MS SQL. Here's what the docker-compose.yml file looks like:
version: '3.8'
services:
mssql:
build:
context: .
dockerfile: docker/mssql/Dockerfile
ports:
- '1434:1433'
environment:
ACCEPT_EULA: "Y"
SA_PASSWORD: "YourStrong!Passw0rd"
volumes:
- mssql-data:/var/opt/mssql
As you can see from my dockerfile path, that's in a sub-path and looks like this:
FROM mcr.microsoft.com/mssql/server:2019-latest
COPY ./docker/mssql/TESTDB.bak /var/opt/mssql/backup/TESTDB.bak
RUN ( /opt/mssql/bin/sqlservr --accept-eula & ) | grep -q "Service Broker manager has started" && /opt/mssql-tools/bin/sqlcmd -S localhost,1433 -U SA -P "YourStrong!Passw0rd" -Q 'RESTORE DATABASE TESTDB FROM DISK = "/var/opt/mssql/backup/TESTDB.bak" WITH MOVE "TESTDB_Data" to "/var/opt/mssql/data/TESTDB.mdf", MOVE "TESTDB_Log" to "/var/opt/mssql/data/TESTDB_log.ldf"'
(Yes, I realize that the password in the RUN command is redundant, I had tried to use a variable there earlier and since it wasn't working I hard coded it.)
When I run docker-compose up -d, I always get this error: Login failed for user 'SA'
I wasted way too much time thinking there was actually something wrong with the password until I realized that if I add the environment variables directly in the Dockerfile, it works. So in my Dockerfile, above the RUN command, I can just do this:
ENV ACCEPT_EULA=Y
ENV SA_PASSWORD=YourStrong!Passw0rd
So I concluded that my environment variables simply aren't being read. I tried with quotes, without quotes, using env_file instead, nothing seems to work. I also tried the following format, no luck:
environment
- ACCEPT_EULA=Y
- SA_PASSWORD=YourStrong!Passw0rd
I also tried using MSSQL_SA_PASSWORD instead of SA_PASSWORD, as well as having both in there. I assumed that was unlikely to be the problem though given SA_PASSWORD works fine. Lastly, I tried using a 2017 image in case it was image specific, that didn't work either.
I'm assuming it must be something silly I'm missing. I saw a lot of talk of .env in the root being different, but if I understood correctly people go wrong with that when they try to use environment values in their docker-compose.yml file, which is not what I'm doing here. So I'm about ready to lose my mind on this as it seems like such a simple, basic thing.
I think you're confusing the ENV statement in Dockerfile with the environment variables set when running an image. The key is still in the details of the docs. It notes that they are the same as saying docker run -e, not docker build.
What's causing more confusion, when you use ENV, you are setting defaults for when the image runs later:
https://docs.docker.com/engine/reference/builder/#env
If you haven't yet, I very much recommend getting familiar with building and running your image with docker run and docker build before moving on to compose, it's much less confusing that way.
The issue with your build here stems from a confusion between the build-time and run-time environment variables: with the environment or env_file properties you specify the environment variables to be set for the service container.
But the RUN command in your Dockerfile is executed at the build-time of the image! To pass variables when building a new image you should use build args instead, as you already mentioned in your comment:
services:
mssql:
build:
context: .
dockerfile: docker/mssql/Dockerfile
args:
SA_PASSWORD: "YourStrong!Passw0rd"
# ...
With this you can use the SA_PASSWORD as a build ARG:
FROM mcr.microsoft.com/mssql/server:2019-latest
COPY ./docker/mssql/TESTDB.bak /var/opt/mssql/backup/TESTDB.bak
ARG SA_PASSWORD
RUN ( /opt/mssql/bin/sqlservr --accept-eula & ) | grep -q "Service Broker manager has started" && /opt/mssql-tools/bin/sqlcmd -S localhost,1433 -U SA -P "$SA_PASSWORD" -Q 'RESTORE DATABASE TESTDB FROM DISK = "/var/opt/mssql/backup/TESTDB.bak" WITH MOVE "TESTDB_Data" to "/var/opt/mssql/data/TESTDB.mdf", MOVE "TESTDB_Log" to "/var/opt/mssql/data/TESTDB_log.ldf"'
If you want to move the actual password to a .env file you can use variable substitution in the compose.yml:
services:
mssql:
build:
# ...
args:
SA_PASSWORD: "$SA_PASSWORD"
# ...
In your docker-compose.yml, have you tried:
- ACCEPT_EULA=Y
- SA_PASSWORD=YourStrong!Passw0rd
Both responses above are fine, just a few more things:
SA_PASSWORD is deprecated instead use MSSQL_SA_PASSWORD
It is always nice to define .env files with the variables for instance:
sapassword.env
MSSQL_SA_PASSWORD=YourStrong!Passw0rd
sqlserver.env
ACCEPT_EULA=Y
MSSQL_DATA_DIR=/var/opt/sqlserver/data
MSSQL_LOG_DIR=/var/opt/sqlserver/log
MSSQL_BACKUP_DIR=/var/opt/sqlserver/backup
And in docker-compose.yml instance the env files the following way:
environment:
- sqlserver.env
- sapassword.env
We have used the technique detailed here to expose host environment variables to Docker build in a secured fashion.
# syntax=docker/dockerfile:1.2
FROM golang:1.18 AS builder
# move secrets out of the build process (and docker history)
RUN --mount=type=secret,id=github_token,dst=/app/secret_github_token,required=true,uid=10001 \
export GITHUB_TOKEN=$(cat /app/secret_github_token) && \
<nice command that uses $GITHUB_TOKEN>
And this command to build the image:
export DOCKER_BUILDKIT=1
docker build --secret id=github_token,env=GITHUB_TOKEN -t cool-image-bro .
The above works perfectly.
Now we also have a docker-compose file running in CI that needs to be modified. However, even if I confirmed that the ENV vars are present in that job, I do not know how to assign the environment variable to the github_token named secret ID.
In other words, what is the equivalent docker-compose command (up --build, or build) that can accept a mapping of an environment variable with a secret ID?
Turns out I was a bit ahead of the times. docker compose v.2.5.0 brings support for secrets.
After having modified the Dockerfile as explained above, we must then update the docker-compose to defined secrets.
docker-compose.yml
services:
my-cool-app:
build:
context: .
secrets:
- github_user
- github_token
...
secrets:
github_user:
file: secrets_github_user
github_token:
file: secrets_github_token
But where are those files secrets_github_user and secrets_github_token coming from? In your CI you also need to export the environment variable and save it to the default secrets file location. In our project we are using Tasks so we added these too lines.
Note that we are running this task from our CI, so you could do it differently without Tasks for example.
- printenv GITHUB_USER > /root/project/secrets_github_user
- printenv GITHUB_TOKEN > /root/project/secrets_github_token
We then update the CircleCI config and add two environment variable to our job:
.config.yml
name-of-our-job:
environment:
DOCKER_BUILDKIT: 1
COMPOSE_DOCKER_CLI_BUILD: 1
You might also need a more recent Docker version, I think they introduced it in a late 19 release or early 20. I have used this and it works:
steps:
- setup_remote_docker:
version: 20.10.11
Now when running your docker-compose based commands, the secrets should be successfully mounted through docker-compose and available to correctly build or run your Dockerfile instructions!
I'm building a docker image from a project where I have a file with default credentials for the database. At the docker container run time, I want to pass the real credentials and replace the variables defined on that file. What is the best way to do it? I tried to use environment variables, but it's not working.
db_config.yml:
host: ${HOST}
user: ${USER}
pass: ${PASS}
port: ${PORT}
db: ${DB_NAME}
docker-compose.yml:
version: '2.3'
services:
test_ctr:
container_name: test
image: container:latest
network_mode: "host"
environment:
- HOST=${HOST}
- USER=${USER}
- PASS=${PASS}
- PORT=${PORT}
- DB_NAME=${DB_NAME}
db_config.yml is in builded image and language is Python. Basically when I run container, db_config.yml is red by a script and use file's credentials. When I create the image, this db_config.yml have default credentials. but when I run the container, I want to replace this file
To debug this try running:
docker exec -it <name-of-the-container> <command>
In your case this translates to:
docker exec -it test sh
This should open a shell inside the container.
Then type:
printenv
This will print all Environment variables and their values (that way You will see if the values You have passed are present)
There will be a problem if the container is crashing at startup (in this case it's not possible to use docker exec).
TIP:
Use .env file located in the same directory as docker-compose.yml (or whatever your docker-compose file is) to pass variables.
.env:
KEY1=value1
KEY2=value2
In your case this might look something like:
HOST=1.2.3.4
USER=sa
PASSWORD=42
PORT=4242
DB_NAME=mydb
When your running:
docker-compose up
docker-compose will look for this .env file and will inject the values from this file
Good luck
I have a service that I am bringing up through Rancher via docker-compose. The issue I am running into is that I need to set a password after the container has been deployed.
The way rancher secrets work, is that I set my secret in and rancher will mount a volume on my container with a file containing my secret. I was hoping to be able to execute a script to grab that secret, and set it as a password on my config file.
I don't believe I have a way to get that secret in through the Dockerfile as I don't want the secret to be in git, so I'm left looking at doing it via docker-compose.
Does anyone know if this is possible?
This is the way I use for calling a script after a container is started without overriding the entrypoint.
In my example, I used it for initializing the replicaset of my local MongoDB
services:
mongo:
image: mongo:4.2.8
hostname: mongo
container_name: mongodb
entrypoint: ["/usr/bin/mongod","--bind_ip_all","--replSet","rs0"]
ports:
- 27017:27017
mongosetup:
image: mongo:4.2.8
depends_on:
- mongo
restart: "no"
entrypoint: [ "bash", "-c", "sleep 10 && mongo --host mongo:27017 --eval 'rs.initiate()'"]
In the first part, I simply launch my service (mongo)
The second service use a "bash" entry point AND a restart: no <= important
I also use a depends_on between service and setup service for manage the launch order.
The trick is to overwrite the compose COMMAND to perform whatever init action you need before calling the original command.
Add a script in your image that will perform the init work that you want like set password, change internal config files, etc. Let's call it init.sh. You add it to your image.
Dockerfile:
FROM: sourceimage:tag
COPY init.sh /usr/local/bin/
ENTRYPOINT []
The above overrides whatever ENTRYPOINT is defined in the sourceimage. That's to make this example simpler. Make sure you understand what the ENTRYPOINT is doing in the Dockerfile from the sourceimage and call it in the command: of the docker-compose.yml file.
docker-compose.yml:
services:
myservice:
image: something:tag
...
command: sh -c "/usr/local/bin/init.sh && exec myexecutable"
It's important to use exec before calling the main command. That will install the command as the first process (PID1) which will make it receive signals like STOP, KILL (Ctrl-C on keyboard) or HUP.
You can also use volumes to do this:
services:
example:
image: <whatever>
volume: ./init.sh:/init.sh
entrypoint: sh -c "/init.sh"
Note that this will mount init.sh to the container, not copy it (if that matters, usually it doesn't). Basically processes within the container can modify init.sh and it would modify the file as it exists in your actual computer.
docker-compose specify how to launch containers, not how to modify an existing running container.
The Rancher documentation mentions that, for default usage of secrets, you can reference the secret by name in the secrets array in the docker-compose.yml.
The target filename will be the same name as the name of the secret.
By default, the target filename will be created as User ID and Group ID 0, and File Mode of 0444.
Setting external to true in the secrets part will make sure it knows the secret has already been created.
Example of a basic docker-compose.yml:
version: '2'
services:
web:
image: sdelements/lets-chat
stdin_open: true
secrets:
- name-of-secret
labels:
io.rancher.container.pull_image: always
secrets:
name-of-secret:
external: true
As illustrated in "How to Update a Single Running docker-compose Container", updating a container would involve a "build, kill, and up" sequence.
docker-compose up -d --no-deps --build <service_name>
What I'm trying to do
I want to run a yesod web application in one docker container, linked to a postgres database in another docker container.
What I've tried
I have the following file hierarchy:
/
api/
Dockerfile
database/
Dockerfile
docker-compose.yml
The docker-compose.yml looks like this:
database:
build: database
api:
build: api
command: .cabal/bin/yesod devel # dev setting
environment:
- HOST=0.0.0.0
- PGHOST=database
- PGPORT=5432
- PGUSER=postgres
- PGPASS
- PGDATABASE=postgres
links:
- database
volumes:
- api:/home/haskell/
ports:
- "3000:3000"
Running sudo docker-compose up fails either to start the api container at all or, just as often, with the following error:
api_1 | Yesod devel server. Press ENTER to quit
api_1 | yesod: <stdin>: hGetLine: end of file
personal_api_1 exited with code 1
If, however, I run sudo docker-compose database up & then start up the api container without using compose but instead using
sudo docker run -p 3000:3000 -itv /home/me/projects/personal/api/:/home/haskell --link personal_database_1:database personal_api /bin/bash
I can export the environment variables being set up in the docker-compose.yml file then manually run yesod devel and visit my site successfully on localhost.
Finally, I obtain a third different behaviour if I run sudo docker-compose run api on its own. This seems to start successfully but I can't access the page in my browser. By running sudo docker-compose run api /bin/bash I've been able to explore this container and I can confirm the environment variables being set in docker-compose.yml are all set correctly.
Desired behaviour
I would like to get the result I achieve from running the database in the background then manually setting the environment in the api container's shell simply by running sudo docker-compose up.
Question
Clearly the three different approaches I'm trying do slightly different things. But from my understanding of docker and docker-compose I would expect them to be essentially equivalent. Please could someone explain how and why they differ and, if possible, how I might achieve my desired result?
The error-message suggests the API container is expecting input from the command-line, which expects a TTY to be present in your container.
In your "manual" start, you tell docker to create a TTY in the container via the -t flag (-itv is shorthand for -i -t -v), so the API container runs successfully.
To achieve the same in docker-compose, you'll have to add a tty key to the API service in your docker-compose.yml and set it to true;
database:
build: database
api:
build: api
tty: true # <--- enable TTY for this service
command: .cabal/bin/yesod devel # dev setting