docker stack deploy isnt respecting the extra_hosts parameter in my compose file. when i do a simple docker-compose up the entry is created in the /etc/hosts however when i do docker deploy –compose-file docker-compose.yml myapp it ignores extra_hosts, any insights?
Below is the docker-compose.xml:
version: '3'
services:
web:
image: user-service
deploy:
labels:
- the label
build:
context: ./
environment:
DATABASE_URL: jdbc:postgresql://dbhost:5432/postgres
ports:
- 9002:9002
extra_hosts:
- "dbhost: ${DB_HOST}"
networks:
- wellness_swarm
env_file:
- .env
networks:
wellness_swarm:
external:
name: wellness_swarm
the docker-compose config also displays the compose file properly.
This may not be a direct answer to the question as it doesn't use env variables but what I found was that the extra_hosts block in the compose file was ignored in swarm mode if entered in the format above.
i.e. this works for me and puts entries in /etc/hosts in the container:
extra_hosts:
retisdev: 10.48.161.44
retistesting: 10.48.161.44
whereas when entered in the other format it gets ignored when deploying as a service
extra_hosts:
- "retisdev=10.48.161.44"
- "retistesting=10.48.161.44"
I think it's an ordering issue. The ${} variable you've got in the compose file runs during the YAML processing before the service definition is created. Then stack deploy processes the .env file for running in the container as envvars, but the YAML variable is needed first...
To fix that, you should use the docker-compose config command first, to process the YAML, and then use the output of that to send to the stack deploy.
docker-compose config will show you the output you're likely wanting.
Then do a pipe to get a one-liner.
docker-compose config | docker stack deploy -c - myapp
Note: Ideally you wouldn't use the extra_hosts, but rather put the envvar directly in the connection string. Your way seems like unnecessary complexity and isn't the usual way I see a connection string created.
e.g.
version: '3'
services:
web:
image: user-service
deploy:
labels:
- the label
build:
context: ./
environment:
DATABASE_URL: jdbc:postgresql://${DB_HOST}:5432/postgres
ports:
- 9002:9002
networks:
- wellness_swarm
env_file:
- .env
networks:
wellness_swarm:
external:
name: wellness_swarm
As i see https://github.com/moby/moby/issues/29133 seems like it is by design where in the compose command takes into consideration the environment variables mentioned in .env file however the deploy command ignores that :( why is that so, pretty lame reasons!
Related
I have a portainer stack running one container. Lets use microbin as an example.
The docker-compose yaml looks like this:
---
version: "3"
services:
paste:
image: danielszabo99/microbin:latest
container_name: microbin
restart: always
ports:
- "8525:8080"
volumes:
- /mnt/docker_volumes/microbin-data:/app/pasta_data
This particular container is hosted on docker hub, and the maintainer provides examples of command line arguments that can be appended to the dockerfile to activate various features easily. One example would be:
--no-listing
Disables the /pastalist endpoint, essentially making all pastas private.
So this brings me to my issue. I don't want to maintain my own custom dockerfile, and in the past I have always inserted environment variables into the docker-compose yaml to call features like this.
An example would be like this - I have a stack running for Authentik (a sso/saml/idp gateway with a pretty web interface). You can see the "environment:" section and the variables I am calling.
server:
image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2022.5.3}
restart: unless-stopped
command: server
environment:
AUTHENTIK_REDIS__HOST: redis
AUTHENTIK_POSTGRESQL__HOST: postgresql
AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
AUTHENTIK_ERROR_REPORTING__ENABLED: "true"
# WORKERS: 2
volumes:
- ./media:/media
- ./custom-templates:/templates
- geoip:/geoip
env_file:
- stack.env
So - not knowing how the development side of making these containers and hosting them on docker-hub goes... is there a way for me to use these command line arguments for microbin as environment variables in my docker-compose yaml / stack configuration file, or am I going to have to wait on the maintainer to implement this as a feature?
Thanks for your help in advance.
You can pass command line arguments in your docker-compose.yml file using the command attribute. That assumes of course the process started within the Docker image can deal with those, but that seems to be the case for your image and should generally be the case.
version: "3"
services:
paste:
image: danielszabo99/microbin:latest
container_name: microbin
restart: always
ports:
- "8525:8080"
volumes:
- /mnt/docker_volumes/microbin-data:/app/pasta_data
command: my command line --args here
See Docker Compose Reference - command for details.
Hi guys and excuse me for my English. I'm using docker swarm, when I attempt to deploy docker application with this command
docker stack deploy -c docker-compose.yml -c docker-compose.prod.yml chatappapi
it shows the next error : services.chat-app-api Additional property pull_policy is not allowed
why this happens?
how do I solve this?
docker-compose.yml
version: "3.9"
services:
nginx:
image: nginx:stable-alpine
ports:
- "5000:80"
volumes:
- ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
chat-app-api:
build: .
image: username/myapp
pull_policy: always
volumes:
- ./:/app
- /app/node_modules
environment:
- PORT= 5000
- MAIL_USERNAME=${MAIL_USERNAME}
- MAIL_PASSWORD=${MAIL_PASSWORD}
- CLIENT_ID=${CLIENT_ID}
- CLIENT_SECRET=${CLIENT_SECRET}
- REDIRECT_URI=${REDIRECT_URI}
- REFRESH_TOKEN=${REFRESH_TOKEN}
depends_on:
- mongo-db
mongo-db:
image: mongo
environment:
MONGO_INITDB_ROOT_USERNAME: 'username'
MONGO_INITDB_ROOT_PASSWORD: 'password'
ports:
- "27017:27017"
volumes:
- mongo-db:/data/db
volumes:
mongo-db:
docker-compose.prod.yml
version: "3.9"
services:
nginx:
ports:
- "80:80"
chat-app-api:
deploy:
mode: replicated
replicas: 8
restart_policy:
condition: any
update_config:
parallelism: 2
delay: 15s
build:
context: .
args:
NODE_ENV: production
environment:
- NODE_ENV=production
- MONGO_USER=${MONGO_USER}
- MONGO_PASSWORD=${MONGO_PASSWORD}
- MONGO_IP=${MONGO_IP}
command: node index.js
mongo-db:
environment:
MONGO_INITDB_ROOT_USERNAME: ${MONGO_USER}
MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD}
Information
docker-compose version 1.29.2
Docker version 20.10.8
Ubuntu 20.04.2 LTS
Thanks in advance.
Your problem line is in docker-compose.yml
chat-app-api:
build: .
image: username/myapp
pull_policy: always # <== this is the bad line, delete it
The docker compose file reference doesn't have any pull_policy in the api because
If the image does not exist, Compose attempts to pull it, unless you have also specified build, in which case it builds it using the specified options and tags it with the specified tag.
I think pull_policy used to be a thing for compose? Maybe keep the latest api documentation open to refer to/search through whilst you're developing (things can and do change fairly frequently with compose).
If you want to ensure that the most recent version of an image is pulled onto all servers in a swarm then run docker compose -f ./docker-compose.yml pull on each server in turn (docker stack doesn't have functionality to run this over an entire swarm yet).
As an aside: I wouldn't combine two .yml files with a single docker stack command without a very good reason to do so.
You are mixing docker-compose and docker swarm ideas up in the same files:
It is probably worth breaking your project up into 3 files:
docker-compose.yml
This would contain just the basic service definitions common to both compose and swarm.
docker-compose.override.yml
Conveniently, docker-compose and docker compose both should read this file automatically. This file should contain any "port:", "depends_on:", "build:" directives, and any convenience volumes use for development.
stack.production.yml
The override file to be used in stack deployments should contain everything understood by swarm and not compose, and b. everything required for production.
Here you would use configs: or even secrets: rather than volume mappings to local folders to inject content into containers. Rather than relying on ports: directives, you would install an ingress router on the swarm such as traefik. and so on.
With this arrangement, docker compose can be used to develop and build your compose stack locally, and docker stack deploy won't have to be exposed to compose syntax it doesn't understand.
pull_policy is in the latest version of docker-compose.
To upgrade your docker-compose refer to:
https://docs.docker.com/compose/install/
The spec for more info:
https://github.com/compose-spec/compose-spec/blob/master/spec.md#pull_policy
I am using docker-compose to run a traefik container. The Domain of this Container should be set by an environment file but everytime i start this service it says:
WARNING: The DOMAIN variable is not set. Defaulting to a blank string
My compose-file setup:
version: '3.5'
networks:
frontend:
name: frontend
backend:
name: backend
services:
Traefik:
image: traefik:latest
command: --api --docker --acme.email="test#test.de"
restart: always
container_name: Traefik
networks:
- backend
- frontend
env_file: ./env.env
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./traefik/traefik.toml:/traefik.toml
- ./traefik/acme.json:/acme.json
ports:
- "80:80"
- "443:443"
labels:
- "traefik.docker.network=frontend"
- "traefik.enable=true"
- "traefik.frontend.rule=Host:traefik.${DOMAIN}"
- "traefik.port=8080"
- "traefik.protocol=http"
My env.env file setup:
DOMAIN=fiture.de
Thanks for your Help!
env_file: ./env.env
The file env.env isn't loaded to parse the compose file, it is loaded to add environment variables within the container being run. At the point docker processes the above instruction, the yaml file has already been loaded and variables have been expanded.
If you are using docker-compose to deploy containers on a single node, you can rename the file .env and docker-compose will load variables from that file before parsing the compose file.
If you are deploying with docker stack deploy, then you need to import the environment variables into your shell yourself. An example of doing that in bash looks like:
set -a && . ./env.env && set +a && docker stack deploy ...
I want to have two docker-compose files, where one overrides another.
(The motivation comes from Docker Compose Docs)
The use case comes from the buildbot environment. The first docker-compose file should define a simple service. This is a service that is going to be tested. Let's take
version: '2'
services:
service-node:
build:
context: ./res
dockerfile: Dockerfile
image: my/server
env_file: .env
The second docker-compose file (let's name it docker-compose.test.yml) overrides the service-node to add a buildbot worker feature, and creates the second container, i.e. buildbot master node, that is going to control testing machinery. Let's take
version: '2'
services:
service-node:
build:
context: ./res
dockerfile: buildbot.worker.Dockerfile
image: my/buildbot-worker
container_name: bb-worker
env_file: ./res/buildbot.worker.env
environment:
- BB_RES_DIR=/var/lib/buildbot
networks:
testlab:
aliases:
- bb-worker
volumes:
- ./vol/bldbot/worker:/home/bldbotworker
depends_on:
- bb-master
bb-master:
build:
context: ./res
dockerfile: buildbot.master.Dockerfile
image: my/buildbot-master
container_name: bb-master
env_file: ./res/buildbot.master.env
environment:
- BB_RES_DIR=/var/lib/buildbot
networks:
- testlab
expose:
- "9989"
volumes:
- ./vol/bldbot/master:/var/lib/buildbot
networks:
testlab:
driver: bridge
Generally this configuration works, i.e. the command
docker-compose -f docker-compose.yml -f docker-compose.test.yml up -d
builds both images and runs both containers, but there is one shortcoming, i.e. the command
docker-compose ps
shows only one service, bb-worker. At the same time
docker ps
shows both.
Furthermore, the command
docker-compose down
stops only one service, and outputs the message/warning Found orphan containers. Of course, the message refers to bb-master.
How can I override the basic docker-compose.yml file to be able to add additional non-orphan service?
You need to run all docker-compose commands with the flags, e.g.:
docker-compose -f docker-compose.yml -f docker-compose.test.yml down
Alternatively, you can make this the default by writing the following to a .env file in the same folder:
COMPOSE_FILE=docker-compose.yml:docker-compose.test.yml
NOTE:
In windows you need tu use ";" as the separator (#louisvno)
I'm using docker compose to run my application. And for do that I need to set the hosts inside container (it's depends on the environment i'm running).
My approach was:
Create an environment file and set the variable:
#application.env
SERVER_IP=10.10.9.134
My docker compose file looks like:
version: '2'
services:
api:
container_name: myApplication
env_file:
- application.env
build: ./myApplication/
entrypoint: ./docker/api-startup.sh
ports:
- "8080:8080"
depends_on:
- redis
extra_hosts: &extra_hosts
myip: $SERVER_IP
But my problem is that the variable SERVER_IP is never replaced.
When I run docker-compose config I see:
services:
api:
build:
context: /...../myApplication
container_name: myApplication
depends_on:
- redis
entrypoint: ./docker/api-startup.sh
environment:
SERVER_IP: 10.10.9.134
extra_hosts:
myip: ''
ports:
- 8080:8080
I've tried to replace the variable reference using $SERVER_IP or ${SERVER_IP} but it didn't work.
I created a file .env, added single line HOST=test.example.com, then did this in docker-compose:
extra_hosts:
- myip:${HOST}
docker-compose config then shows
extra_hosts:
myip: test.example.com
To do this I followed the documentation from Docker-compose environment variables the section about .env file
UPDATE
According to the Docker documentation,
Note: If your service specifies a build option, variables defined in
environment files will not be automatically visible during the build.
Use the args sub-option of build to define build-time environment
variables.
It basically means if you place your variables in .env file, you can use them for substitution in docker-compose.yml, but if you use env_file option for the particular container, you can only see the variables inside the Docker container, not during the build. It is also logical, env_file replaces docker run --env-file=FILE ... and nothing else.
So, you can only place your values into .env. Alternatively, as William described, you can use host's environment variables.
EDIT
Try the following:
version: '2'
services:
api:
container_name: myApplication
env_file:
- application.env
build: ./myApplication/
entrypoint: ./docker/api-startup.sh
ports:
- "8080:8080"
depends_on:
- redis
extra_hosts:
- "myip:${SERVER_IP}"
Ensure curly bracers and that the environment variable exists on the host os.