How to specify environment variables with docker image in docker compose - docker

I've got a workflow where I build a specific image and then (after pushing to an ECR repo and then pulling it onto an AWS server) essentially run it with a docker-compose file. My docker compose file looks as follows:
version: "3.8"
services:
web:
image: <my-aws-server>/my-repo:latest
command: gunicorn vms.wsgi:application --bind 0.0.0.0:8000
expose:
- 8000
nginx:
build: ../nginx
ports:
- 1337:80
depends_on:
- web
and my dockerfile is something like this:
FROM python:3
ENV PYTHONUNBUFFERED=1
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
EXPOSE 8000
CMD [ "python", "manage.py", "runserver", "0.0.0.0:8000"]
I'd like to be able to do something like this in my docker-compose:
version: "3.8"
services:
web:
image: <my-aws-server>/my-repo:latest
env: SECRET_PASSWORD #note change here
command: gunicorn vms.wsgi:application --bind 0.0.0.0:8000
expose:
- 8000
nginx:
build: ../nginx
ports:
- 1337:80
depends_on:
- web
where I specify the environment variables, which are stored in a file on the server. Is there any way I can do this? Perhaps it's impossible if the image file is just a binary.
Or do I have to actually pass in the environment variables from the get-go, when I build the image in my GitHub action, here:
steps:
- uses: actions/checkout#v2
name: Check out code
- uses: mr-smithers-excellent/docker-build-push#v5
name: Build & Push Docker image
with:
image: my-image
registry: ${{ secrets.AWS_ECR_REGISTRY }}
tags: latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
Edit: in my GitHub actions, I tried something like this:
- name: Start new container
run: ssh staging 'cd my_dir; sudo docker-compose --env-file ~/code/secrets/.env -f docker-compose.prod.yml up -d'
but that didn't seem to work. Is there something I'm doing wrong there? Or should that have worked as expected where whatever environment variables are in that file will be used in the pre-built image? (I'm not building it again, just starting the image, as is evident from the docker compose file).

There is the env_file directive. That will pass variables from the specified file to the container at runtime.
Reference:
https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option

Related

docker-compose build env not populated in docker image

Use case is to build and image and deploy to Rancher 2.5.5 with gitlab-ci.yml. Since envs couldn't be passed directly in my situation I'm trying to build-in envs to docker image with docker-compose build (dev/stage things is the next thing, just let's leave it for now). docker-compose run --env-file works, but docker-compose build ignores envs.
Any advice will be appreciated
P.S. if you know the way to pass envs to rancher2 container somehow from gitlab-ci it also resolves the problem
I've tried the following:
set it in docker-compose
version: '3'
services:
testproject:
build:
context: .
env_file: .env-dev
image: example.registry/testimage:latest
set it in gitlab-ci
variables:
IMAGE: "$CI_REGISTRY_IMAGE:latest"
build-image:
stage: build
allow_failure: false
tags:
- docker
services:
- docker:dind
script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
- docker-compose --env-file .env-dev build
- docker-compose push
deploy:
stage: deploy
image: kpolszewski/rancher2-gitlab-deploy
tags:
- docker
script:
- upgrade --cluster $CLUSTER --environment $ENV --stack $NAMESPACE --service $SERVICE --new-image $IMAGE
source it in Dockerfile entrypoint
set it in .env file
nothing works
I can see new image in the registry and local (when I test it locally ) but no env inside when I run container
If you want to set env values on the build stage, you can use build-args as follows:
FROM ...
ARG SOME_ARG
ENV SOME_ENV=$SOME_ARG
Then, in your docker-compose.yml:
version: '3'
services:
testproject:
build:
context: .
args:
SOME_ARG: "SOME_VALUE"
env_file: .env-dev
image: example.registry/testimage:latest
But think twice, are you sure you want your ENV variables be dynamically set on the build stage?

Unable to start Tomcat from docker-compose

I'm unable to start Tomcat server from docker compose.
When I log into container using docker exec -it <container id> bash and see ps -eaf | grep "tomcat" it is showing empty. Tomcat server is not running.
docker-compose.yml file:
version: "3"
services:
meghcore:
build: ./Core
container_name: 'meghcore'
expose:
- '8080'
ports:
- '8080:8080'
volumes:
- meghcore:/opt/Tomcat1/webapps/
command: /bin/bash
tty: true
stdin_open: true
networks:
- meghnet
volumes:
meghcore:
networks:
meghnet:
driver: bridge
Dockerfile file:
FROM tomcat:8.5.35
WORKDIR /app
COPY . /app
RUN mv /app/*.war /opt/Tomcat1/webapps/
ENV PATH $PATH:/opt/Tomcat1/bin
WORKDIR /opt/Tomcat1/bin
EXPOSE 8080
CMD ["catalina.sh", "run"]
Since you specify an alternate command: in your docker-compose.yml file, that overrides the CMD in the Dockerfile. You don't need most of the options you specify there at all, and several of them (the alternate command:, the volumes: overwriting the actual application) interfere with the normal container operation.
A complete, functional docker-compose.yml would be
version: "3"
services:
meghcore:
build: ./Core
ports:
- '8080:8080'
None of the other options you list out are necessary. If there were other containers listed in the file, they could still communicate using their Docker Compose service names, without any special setup (another container in this same file could successfully call http://meghcore:8080).
What is happening is command specify in docker-compose.yml is overwriting the CMD provided in dockerfile.
kindly update command with the command available in dockerfile or remove command from docker-compose.yml
Problem is resolved by adding below commands in dockerfile and removed command from docker compose file.
ENV PATH $PATH:$JAVA_HOME/bin
ENV PATH $PATH:/opt/Tomcat1/bin
WORKDIR /opt/Tomcat1/bin
EXPOSE 8080
CMD ["catalina.sh", "run"]

docker-compose and using local image with mounted volume

I have an image I create with Dockerfile
FROM mhart/alpine-node:latest
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY src /app
Now in docker-compose.yml I build this image
version: '3.7'
services:
enginetonic:
build:
context: .
image: enginetonic:compose
mongodb:
image: mongo:latest
container_name: 'mongodb'
ports:
- 27017:27017
restart: always
monitor-service:
image: enginetonic:compose
container_name: monitorService
command: nodemon monitor/monitor.js
restart: on-failure
#common services
access-token-service:
image: enginetonic:compose
container_name: accessTokenService
command: nodemon service/access-token-service/access-token-service.js
restart: on-failure
depends_on:
- mongodb
In all documentation to bind:mount or use volumes I found, it is used with other docker commands
example
$ docker service create \
--mount 'type=volume,src=<VOLUME-NAME>,dst=<CONTAINER-PATH>,volume-driver=local,volume-opt=type=nfs,volume-opt=device=<nfs-server>:<nfs-path>,"volume-opt=o=addr=<nfs-address>,vers=4,soft,timeo=180,bg,tcp,rw"'
--name myservice \
<IMAGE>
How to use volumes, so that every service that covers the whole /src/ directory, so that every service I start with nodemon reflects the files changed in the whole source code?
I would do a volume map in docker-compose.yml like this:
volumes:
- ./app/monitor:/path/to/your/workdir/monitor
And adjust the command to use file monitor, like nodemon, to restart service when there is any file changes:
command: ["nodemon", "/path/to/your/workdir/monitor/monitor.js"]
You may need to adjust the nodemon arguments or configs based on what you need.
PS. you do not need to tag/push your image. Simply build it directly in docker-compose#build

How to build drone image for ARMv7?

First, I downloaded the drone image:
go get github.com/drone/drone/cmd/...
Second, I built the image for arm:
GOARM=7 go build -o release/drone-server github.com/drone/drone/cmd/drone-server
After that, I built the image for docker:
docker -f ./go-workspace/src/github.com/drone/drone/Dockerfile build -t drone/drone .
The docker file looks like so:
# docker build --rm -t drone/drone .
FROM drone/ca-certs
EXPOSE 8000 9000 80 443
ENV DATABASE_DRIVER=sqlite3
ENV DATABASE_CONFIG=/var/lib/drone/drone.sqlite
ENV GODEBUG=netdns=go
ENV XDG_CACHE_HOME /var/lib/drone
ADD release/drone-server /bin/
ENTRYPOINT ["/bin/drone-server"]
That's my docker-compose.yml:
version: '2'
services:
drone-server:
image: drone/drone:latest
ports:
- 8000:8000
- 9000:9000
volumes:
- /var/lib/drone:/var/lib/drone
restart: always
env_file:
- /etc/drone/server.env
drone-agent:
image: drone/agent:linux-arm
command: agent
depends_on:
- drone-server
volumes:
- /var/run/docker.sock:/var/run/docker.sock
restart: always
env_file:
- /etc/drone/agent.env
The agent.env file:
DRONE_SECRET=xxx
DRONE_SERVER=[server-hostname]:9000
DOCKER_ARCH=arm
The server.env file:
# Service settings
DRONE_SECRET=xxx
DRONE_HOST=https://[server-hostname]/drone
DRONE_OPEN=false
DRONE_GOGS=true
DRONE_GOGS_URL=https://[server-hostname]/git
DRONE_GOGS_PRIVATE_MODE=true
However, when running docker-compose -f /etc/drone/docker-compose.yml up, I get the following error:
drone-server_1 | standard_init_linux.go:190: exec user process caused "no such file or directory"
And the drone-server exits with code 1.
I configured Apache to reach drone trough a proxy as described here: http://readme.drone.io/0.5/install/setup/apache/
Any help is appreciated.

docker-compose not setting environment variables

When I run docker-compose build && docker-compose up redis, with environment specified in docker-compose.yaml and RUN env in the Dockerfile, the environment variables I set don't get printed.
Why does this not work?
I'm using docker-compose version 1.4.2.
Here are the relevant files:
docker-compose.yaml with environment as a list of KEY=value pairs:
redis:
build: ../storage/redis
ports:
- "6379:6379"
environment:
- FOO='bar'
docker-compose.yaml with environment as a dictionary:
redis:
build: ../storage/redis
ports:
- "6379:6379"
environment:
- FOO: 'bar'
Dockerfile:
FROM redis:2.6
MAINTAINER me#email.com
RUN mkdir -p /var/redis && chown -R redis:redis /var/redis
RUN echo '-------------- env ---------------'
RUN env
COPY redis.conf /usr/local/etc/redis/redis.conf
EXPOSE 6379
ENTRYPOINT ["redis-server", "/usr/local/etc/redis/redis.conf"]
That's normal
docker-compose only sets the environment variables specified in the environment directive in the docker-compose.yaml file during the run phase of the container, and not during the build phase.
So if you do docker-compose run --entrypoint "/bin/bash" redis -c env you will be able to see your env variables.
If you want to set variables inside your Dockerfile (to be able to see them during the build phase) you can add inside your dockerfile before your RUN env:
ENV FOO bar
Well
I have tested and found following solutions for docker compose with env file or without env file. I will show you two different approach
Lets say you have following docker compose yml file
version: '3.8'
services:
db:
image: postgres:13
volumes:
- "./volumes/postgres:/var/lib/postgresql/data"
ports:
- "5432:5432"
env_file: docker.env
Now you need to setup the postgres variable in a file called docker.env. Remember you need to keep the docker_compose.yml file and docker.env file in same folder.
Next, In the docker.env file you need to have the database variable and value like this:
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=myapp_db
Now hit docker-compose up . It should work.
Lets say now you dont like to specify the env file name in the docker-compose.yml file. So you have to write docker-compose.yml file like this:
version: '3.8'
services:
db:
image: postgres:13
volumes:
- "./volumes/postgres:/var/lib/postgresql/data"
ports:
- "5432:5432"
environments:
- POSTGRES_USER=${PGU}
-POSTGRES_PASSWORD=${PGP}
-POSTGRES_DB=${PGD}
Now your docker.env file should look like this:
PGU=postgres
PGP=postgres
PGD=myapp_db
now hit docker-compose --env-file docker.env up
you are good to go.
This is because you were using environment when (I guess) you wanted to use args inside the build block:
redis:
build:
context: ../storage/redis
args:
- FOO: 'bar'
ports:
- "6379:6379"
Your Dockerfile would define FUN in the (image) environment:
FROM redis:2.6
RUN mkdir -p /var/redis && chown -R redis:redis /var/redis
# Read FUN from (build) arguments
# (may define a default: ARG FUN='wow')
ARG FUN
# Define env variable FUN with value from ARG
ENV FUN=$FUN
RUN echo '-------------- env ---------------'
RUN env
COPY redis.conf /usr/local/etc/redis/redis.conf
EXPOSE 6379
ENTRYPOINT ["redis-server", "/usr/local/etc/redis/redis.conf"]
The environment block is used to define variables for the running container (when docker-compose up, NOT when docker-compose build).

Resources