I'm looking to be able to run swarm from same docker-compose file which uses env variables. Currently I only achieved that all nodes are replicating Leaders env. Is it possible to let each node start from its own local env var?
My docker-compose
version: '3.1'
networks:
base:
services:
test:
container_name: ${Name}
restart: always
image: ubuntu:latest
environment:
- Name=${Name}
command: sh -c "echo $Name && sleep 30"
networks:
- base
use env_file option
https://docs.docker.com/compose/environment-variables/
# .env, file
Name=<your_name>
# <your_name>.env, file
TEST_ENV=stackoverflow
# docker-compose.yaml, file
version: '3.1'
services:
test:
container_name: ${Name}
restart: always
image: ubuntu:latest
env_file:
- ${Name}.env
command: sh -c "set | grep TEST_ENV && sleep 30"
docker logs <your_name>
# TEST_ENV='stackoverflow'
You can set env_files with different names in different containers.
for example
# docker-compose.yaml, file
version: '3.1'
services:
test1:
container_name: test1
restart: always
image: ubuntu:latest
env_file:
- first.env
command: sh -c "set | grep TEST_FIRST_ENV && sleep 30"
test2:
container_name: test2
restart: always
image: ubuntu:latest
env_file:
- second.env
command: sh -c "set | grep TEST_SECOND_ENV && sleep 30"
Environment variables referenced in the docker-compose.yml file are not resolved on the leader even, they are resolved on whatever jump box you are deploying too the swarm from.
If you want to reference the env vars from the host system, from the command, or entrypoint, iirc you can escape the reference to "$$Name", but this will only make the env variable available to the entrypoint or command script which are evaluated on the host, not to values like the container_name.
Given your specific use case, perhaps service creation templates are what you are looking for: They let you inject per service instance values into hostname, mount and env.
version: '3.8'
services:
test:
env:
MY_HOSTNAME: "{{.Node.Hostname}}"
...
See Create Service Using Templates for the full list of supported values.
Related
I have a docker-compose file that will startup 6 different microservices. The way our docker-repository is setup prevents the use of the 'latest' tag so I am looking for a way to run a script before docker-compose pulls the microservice images, which will set environment variables in the scope of the docker-compose.yml file.
version: '3'
services:
#Service 1
svc1:
image: some-snapshot.docker.privaterepo.com/some-service:${LATEST_SVC_TAG}
container_name: service1
ports:
- "8080:8080"
#Service 2
svc2:
image: some-snapshot.docker.privaterepo.com/some-service2:${LATEST_SVC2_TAG}
container_name: service2
ports:
- "8081:8081"
I'm not sure you really need "a script". You can just run something like:
LATEST_SVC_TAG=1.1 LATEST_SVC2_TAG=2.5 docker-compose up -d
Alternately, you could place those into a .env file locally:
cat > .env <<EOF
LATEST_SVC_TAG=1.1
LATEST_SVC2_TAG=2.3
EOF
docker-compose up -d
On my image I want to set some environment variables eg: MY_VAR where it will have a static value eg: MY_VAR=12 but I do NOT want to be able to set it via docker's -e param or via docker-compose.yml's environment section.
Furthermore I do not want to be as build argument when i do either docker build or docker-compose build
How can I do that?
You can do that from an entrypoint script.
In your Dockerfile:
ENTRYPOINT ["/entrypoint.sh"]
Example entrypoint.sh:
#!/bin/sh
export VAR=foobar
exec /usr/bin/python "$#"
To be more flexible and allow setting it with the -e option:
export VAR=${VAR:-"foobar"}
...
The best solution for your question is to include an env_file on your docker-compose build
version: '3.2'
services:
db:
restart: always
image: postgres:alpine
volumes:
- backup-data:/var/lib/postgresql/data
env_file:
- ./env/.dev
Then in your env_file:
POSTGRES_USER=my_user
POSTGRES_PASSWORD=my_password
POSTGRES_DB=my_db
I'm new to Docker. I am writing a docker-compose file which creates 2 containers, foo and bar, sharing a volume data:
services:
foo:
container_name: foo
build: ./foo
volumes:
- data:/var/lib/
bar:
container_name: bar
build: ./bar
volumes:
- data:/var/lib/
depends_on:
- foo
volumes:
data:
Now, I want to use the environment variable TAG, to tag containers and volumes, in order to specify if it's for test or production. I expects something like this:
services:
foo:
container_name: foo_${TAG}
build: ./foo
volumes:
- data_${TAG}:/var/lib/
bar:
container_name: bar_${TAG}
build: ./bar
volumes:
- data_${TAG}:/var/lib/
depends_on:
- foo
volumes:
data_${TAG}:
Obviously, docker-compose is unhappy because of the last line containing data_${TAG}:.
How can I name my volume with TAG env variable?
If you create your volumes in advance, you can use the variable on external volume names like this (note that the reference inside of compose is a fixed name but it points to a variable external volume name):
$ cat docker-compose.volvar.yml
version: '2'
volumes:
data:
external:
name: test-data-${TAG}
services:
source:
image: busybox
command: /bin/sh -c 'echo From ${TAG} >>/data/common.log && sleep 10m'
environment:
- TAG
volumes:
- data:/data
target:
image: busybox
command: tail -f /data/common.log
depends_on:
- source
environment:
- TAG
volumes:
- data:/data
Create your volumes in advance with a docker volume create command:
$ docker volume create test-data-dev
test-data-dev
$ docker volume create test-data-uat
test-data-uat
$ docker volume create test-data-stage
test-data-stage
And here's an example of running it (I didn't use different directories or change the project name, so compose just replaced my containers each time, but I could have easily changed the project to run them all concurrently with the same results):
$ TAG=dev docker-compose -f docker-compose.volvar.yml up -d
Creating test_source_1
Creating test_target_1
$ docker logs test_target_1
From dev
$ TAG=uat docker-compose -f docker-compose.volvar.yml up -d
Recreating test_source_1
Recreating test_target_1
$ docker logs test_target_1
From uat
$ TAG=stage docker-compose -f docker-compose.volvar.yml up -d
Recreating test_source_1
Recreating test_target_1
$ docker logs test_target_1
From stage
$ # just to show that the volumes are saved and unique,
$ # rerunning uat generates a second line
$ TAG=uat docker-compose -f docker-compose.volvar.yml up -d
Recreating test_source_1
Recreating test_target_1
$ docker logs test_target_1
From uat
From uat
I don't know if this is possible like that but here is what I do:
I have a docker-compose.yml file like that
services:
foo:
container_name: foo_${TAG}
build: ./foo
volumes:
- /var/lib/
bar:
container_name: bar_${TAG}
build: ./bar
volumes:
- /var/lib/
depends_on:
- foo
And then I create a file docker-compose.override.yml that contains
services:
foo:
volumes:
- data_dev:/var/lib/
bar:
volumes:
- data_dev:/var/lib/
This way when you launch docker-compose, it will use the main file and use the other file to override its values.
You should then have 3 files:
docker-compose.yml
docker-compose.override-prod.yml
docker-compose.override-dev.yml
And then when you build you have the choice between those 2:
(What I do) I copy docker-compose.override-.yml to docker-compose.override.yml and Docker Compose with automatically takes those 2 files
You can provide the 2 files to use to the docker compose file (I forgot what the paramter is... I guess it's "-f")
I hope it helps
docker-compose.yml
version: '2'
services:
app:
build:
context: .
command: python src/app.py
restart: on-failure
depends_on:
- db
environment:
- TJBOT_DB_HOST=db
- TJBOT_API_KEY
- TJBOT_AUTO_QUESTION_TIME
env_file:
- .env
db:
image: mongo:3.0.14
volumes:
- mongodbdata:/data/db
volumes:
mongodbdata:
If I change the .env file, how could I reload the container to use the new environment variables with minimum downtime?
If you are running the yml with docker-compose, you can just run docker-compose up -d and it will recreate any containers that have changes and leave all unchanged services untouched.
$ cat docker-compose.env2.yml
version: '2'
services:
test:
image: busybox
# command: env
command: tail -f /dev/null
environment:
- MY_VAR=hello
- MY_VAR2=world
test2:
image: busybox
command: tail -f /dev/null
environment:
- MY_VAR=same ole same ole
$ docker-compose -f docker-compose.env2.yml up -d
Creating network "test_default" with the default driver
Creating test_test_1
Creating test_test2_1
$ vi docker-compose.env2.yml # edit the file to change MY_VAR
$ docker-compose -f docker-compose.env2.yml up -d
Recreating test_test_1
test_test2_1 is up-to-date
If you run the containers as a docker stack deploy -c docker-compose.yml with a version 3 file format, you can do a rolling update of the service which will prevent any downtime if you have multiple instances of your service running. This functionality is still very new, you'll want 1.13.1 to fix some of the issues with updates, and as with anything this new, bugs are still being worked out.
When I run docker-compose build && docker-compose up redis, with environment specified in docker-compose.yaml and RUN env in the Dockerfile, the environment variables I set don't get printed.
Why does this not work?
I'm using docker-compose version 1.4.2.
Here are the relevant files:
docker-compose.yaml with environment as a list of KEY=value pairs:
redis:
build: ../storage/redis
ports:
- "6379:6379"
environment:
- FOO='bar'
docker-compose.yaml with environment as a dictionary:
redis:
build: ../storage/redis
ports:
- "6379:6379"
environment:
- FOO: 'bar'
Dockerfile:
FROM redis:2.6
MAINTAINER me#email.com
RUN mkdir -p /var/redis && chown -R redis:redis /var/redis
RUN echo '-------------- env ---------------'
RUN env
COPY redis.conf /usr/local/etc/redis/redis.conf
EXPOSE 6379
ENTRYPOINT ["redis-server", "/usr/local/etc/redis/redis.conf"]
That's normal
docker-compose only sets the environment variables specified in the environment directive in the docker-compose.yaml file during the run phase of the container, and not during the build phase.
So if you do docker-compose run --entrypoint "/bin/bash" redis -c env you will be able to see your env variables.
If you want to set variables inside your Dockerfile (to be able to see them during the build phase) you can add inside your dockerfile before your RUN env:
ENV FOO bar
Well
I have tested and found following solutions for docker compose with env file or without env file. I will show you two different approach
Lets say you have following docker compose yml file
version: '3.8'
services:
db:
image: postgres:13
volumes:
- "./volumes/postgres:/var/lib/postgresql/data"
ports:
- "5432:5432"
env_file: docker.env
Now you need to setup the postgres variable in a file called docker.env. Remember you need to keep the docker_compose.yml file and docker.env file in same folder.
Next, In the docker.env file you need to have the database variable and value like this:
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=myapp_db
Now hit docker-compose up . It should work.
Lets say now you dont like to specify the env file name in the docker-compose.yml file. So you have to write docker-compose.yml file like this:
version: '3.8'
services:
db:
image: postgres:13
volumes:
- "./volumes/postgres:/var/lib/postgresql/data"
ports:
- "5432:5432"
environments:
- POSTGRES_USER=${PGU}
-POSTGRES_PASSWORD=${PGP}
-POSTGRES_DB=${PGD}
Now your docker.env file should look like this:
PGU=postgres
PGP=postgres
PGD=myapp_db
now hit docker-compose --env-file docker.env up
you are good to go.
This is because you were using environment when (I guess) you wanted to use args inside the build block:
redis:
build:
context: ../storage/redis
args:
- FOO: 'bar'
ports:
- "6379:6379"
Your Dockerfile would define FUN in the (image) environment:
FROM redis:2.6
RUN mkdir -p /var/redis && chown -R redis:redis /var/redis
# Read FUN from (build) arguments
# (may define a default: ARG FUN='wow')
ARG FUN
# Define env variable FUN with value from ARG
ENV FUN=$FUN
RUN echo '-------------- env ---------------'
RUN env
COPY redis.conf /usr/local/etc/redis/redis.conf
EXPOSE 6379
ENTRYPOINT ["redis-server", "/usr/local/etc/redis/redis.conf"]
The environment block is used to define variables for the running container (when docker-compose up, NOT when docker-compose build).