Expose docker port based on environment variable in compose - docker

I am using docker compose to set up application environments. There are two distinct environments, test and production.
In a test environment, I need to expose additional ports (for debugging). These ports should remain closed in a production environment.
I would also like to use the same image and docker-compose.yml file. Using the same image is no problem but I am struggeling with the compose file. In it, I would like to open or close a port based on an environment variable.
The current setup is pretty much the standard, like this:
# ...
ports:
- "8080:8080" # HTTP Server port
- "9301:9301" # debug port
# ...
In this example, both ports are always exposed. Is it possible to expose the port 9301 only if a certain environment variable, say EXPOSE_DEBUG, is set?

You can use profiles or a second compose file.
services:
app-prod:
&app
image: busybox
profiles:
- production
ports:
- 8080:8080
app-dev:
<<: *app
profiles:
- development
ports:
- 8080:8080
- 9090:9090
Then you can use the below command or an environment variable to set the profile, COMPOSE_PROFILES.
docker compose --profile <profile-name> up
Alternatively, you can use a second compose file and override the ports.
# compose.yaml
services:
app:
image: busybox
ports:
- 8080:8080
# compose.dev.yaml
services:
app:
ports:
- 8080:8080
- 9090:9090
Then you can use the file after the main file to patch it:
docker compose -f compose.yaml -f compose.dev.yaml up
The file(s) to use can also be controls with an environment variable, COMPOSE_FILE.
If you name the file compose.override.yaml, docker will automatically use it, so you don't have to point to it with the -f flag. Be careful that you don't add this file to your production system, if you choose to do this.
You could also bind the debug port to the loopback interface so that you can only access it locally.
ports:
- 8080:8080
- 127:0.0.1:9090:9090

The solution I usually use in my projects is to make a bash script that writes the docker-compose.yml based on the value of the environment variable. But you could write it with any other programming language as well.

Conditional statements (if else) are not supported in docker compose.
Use additional software like jinja-compose adding Jinja2 logic to docker-compose
Use just two different files (dc-dev.yml and dc-prod.yml) and give them as arg (docker compose -f)
Generate docker-compose.yml programmatically by yourself
Use profiles (Was to slow, see answer of the fool)
To just maintain dev/prod environments in my opinion solution 2 is the most efficient in terms of effort.
To follow your approach:
You can set port mapping by envs like:
.env-File or add them in docker compose up -e command
PORT1="8080:8080"
PORT2="9301:9301"
docker-compse.yml
services:
container1:
ports:
- ${PORT1}
- ${PORT2}
But afaik there is no way to omit one of them

Related

Can docker-compose profiles be used together with docker-compose override to have a common code with configurable environments?

I have the requirement of having a unique docker-compose.yaml / infrastrucutre code which will be versioned across the different deployment stages.
I would like to have some ports exposed in development and not in production. As I learned from other questions this seems to not be possible (using the same .env file that is used to configure other environment variables for the containers).
My idea would be to have my docker-compose.yaml, for example:
version: "3.9"
services:
myservice:
image: myimage
# **
# configuration
# **
ports:
- 80:80
- 19980:19980
Then in production overrideit with a profile docker-compose.production.yaml
version: "3.9"
services:
myservice:
profiles:
- production
ports:
- 80:80
This would allow to have always the same configuration (both .yamlf iles) and switch between them by just calling the docker-compose up command with the production profile (--profile).
My question is, does this work as expected or is the service always overwritten also when the profile flag is not provided?
Compose profiles only affect which services start; they do not have any effect on the options those services use. If you have multiple Compose files then the options in those files are merged according to a set of rules. My expectation is that this would take effect before the profile selection took place.
What you're describing seems like a fairly routine setup for multiple Compose files, without using the profile feature. The most common case I've seen is that a "development" setup strictly adds options to a "production" setup. In your example, both "production" and "development" publish port 80, but only development also publishes the debugger port. There also might be additional environment variables or bind mounts that only make sense in development, but you (usually) are not trying to remove values.
So in this setup, your base docker-compose.yml file would contain the production setup, with the minimum values that are used in all environments.
# docker-compose.yml
version: '3.8'
services:
myservice:
image: myimage
ports:
- '80:80'
Then you'd have a second file that only has the options that are added for the development setup:
# docker-compose.dev.yml / docker-compose.override.yml (see below)
version: '3.8'
services:
myservice:
# (do not need to repeat `image:`; could add `build:`)
ports:
- '19980:19980'
If you name the file docker-compose.override.yml, Compose will use both files automatically, and you need to make sure to push the base file but not the override file to the production environment.
# uses both files, if docker-compose.override.yml is present
docker-compose up -d
If you name it something else, you need to explicitly name all of the files with docker-compose -f options, on every Compose invocation.
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d
docker-compose -f docker-compose.yml -f docker-compose.dev.yml ps
(Or you can set the $COMPOSE_FILE environment variable, but you have to remember to set it in every shell session in every environment.)

Docker-compose redis: start with fixture?

I am using docker-compose to create a redis container. However, I need it to start with some default key values. Is this possible?
You need to modify your DockerCompose file, You can also add from some file which contains key value but here is the simplest example that adds and get key in DockerCompose file.
version: '2'
services:
redis:
image: 'bitnami/redis:latest'
environment:
- ALLOW_EMPTY_PASSWORD=yes
ports:
- '6379:6379'
command:
- /bin/sh
- -c
- |
nohup redis-server &
sleep 5
echo "adding some default key value"
redis-cli SET docker awesome
echo "Get docker key value"
redis-cli GET docker
# this will keep container running
tail -f /dev/null
There are several approaches but be aware that, by default, services start in an arbitrary order using Docker Compose and, even if you use depends_on this only checks that containers are running (e.g. redis) and not that they've completed some initialization process.
1. Easiest: Pre-create
See the option to run the redis image with persistent storage:
https://hub.docker.com/_/redis/
Using this approach, you'd either mount a local directory into the container's /data directory or create a (data) volume and use that. Then, you'd pre-populate the redis server by running the redis-cli against it.
One hack to doing this is to your planned docker-compose.yml file but docker-compose --file=/path/to/docker-compost.yaml up redis where redis is the name of the redis service too. You'll need to ensure the redis service is accessible from the host --ports: 6379:6379 perhaps so that the external redis-cli can access it.
This approach works well for local-only use but does not facilitate deploying the solution elsewhere.
2. Resilient: Test for keys
Docker Compose -- to my knowledge -- doesn't offer an elegant equivalent to Kubernetes' init containers which are run before the dependent container.
With Docker Compose, you could include an initialization (run once) redis-cli to populate the server but you must then augment any clients to check that this has completed or for the existence of this data before starting (successfully).
The simplest solution for this is for the redis clients to fail and restart: always if the redis keys aren't present.
A more advanced solution would be to define a healthcheck for the existence of the redis keys and then depends_upon: ... condition: service_healthy (see link)
See also startup order in Docker Compose described here

How to override docker-compose values in multiple combined files?

Letzt imagine i have 3 compose files (only focus on the mysql service)
docker-compose.yml
docker-compose.staging.yml
docker-compose.prod.yml
In my docker compose.yml i have my basic mysql stuff with dev als build target
version: "3.4"
services:
mysql:
build:
target: dev
...
And start it with
docker-compose up -d
In my staging environment i would like to expose port 3306, but also want another build target so i would create the docker-compose.staging.yml with the following content.
version: "3.4"
services:
mysql:
build
target: prod
ports:
- 3306:3306
And combine it with
docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d
So the build target is overwritten and the port 3306 is now exposed to the outside.
Now i want the same in the docker-compose.prod.yml, just without having the port 3306 exposed to the outside ... How can i override the ports directive to not having ports exposed?
I tried to put an empty array in the prod.yml without success (port is still exposed):
version: "3.4"
services:
mysql:
ports: []
In the end i would like to stack the up command like this:
docker-compose -f docker-compose.yml -f docker-compose.staging.yml -f docker-compose.prod.yml up -d
I also know the docs says
For the multi-value options ports, expose, external_links, dns, dns_search, and tmpfs, Compose concatenates both sets of values
But how can i reach my goal anyway without duplicating configuration?
Yes for sure, i could omit the docker-compose.staging.yml but in the staging.yml are build steps defined, which should also be used for the prod stage to not have any differences between the built container.
So duplicating things isn't really an option.
Thanks
I would actually strongly suggest just not using the "target" command in your compose files. I find it to be extremely beneficial to build a single image for local/staging/production - build once, test it, and deploy it in each environment. In this case, you change things using environment variables or mounted secrets/config files.
Further, using compose to build the images is... fragile. I would recommend building the images in a CI system, pushing them to a registry, and then using the image version tags in your compose file- it is a much more reproducible system.
You might consider using extends key in your compose files like this:
mysql:
extends:
file: docker-compose.yml
service: mysql
ports:
- 3306:3306
# other definitions
Although you'd have to change your compose version from 3.4 to < 3 ( like 2.3 ) because v3 doesn't support this feature ref as there is a open feature request hanging for a long time now.
Important note here is that you shouldn't expose any ports in your base docker-compose.yml file, only on the specific composes.
Oficial docs ref for extends
edit
target clause is not supported in v2.0 so I've adjusted the answer to match the extends and target requirement. That's compose v2.3.
edit from comments
As there is a deploy keyword requirement, then there is compose v3 requirement. And as for now, there is no possibility to extend composes. I've read in some official doc (can't find it now for ref) that they encourage us to use flat composes specific for environment so that it's always clear. Also Docker states that's hard to implement in v3 (ref in the above issue) and it's not going to be implemented anywhere soon. You have to use separate compose files per environment.

docker-compose scale with nginx and without environment variable

I use docker-compose to describe the deployment of one of my application. The application is composed of a
mongodb database,
a nodejs application
a nginx front end the static file of nodejs.
If i scale the nodejs application, i would like nginx autoscale to the three application.
Recently i use the following code snippet :
https://gist.github.com/cmoore4/4659db35ec9432a70bca
This is based on the fact that some environment variable are created on link, and change when new server are present.
But now with the version 2 of the docker-compse file and the new link system of docker, the environment variable doesn't exist anymore.
How my nginx can now detect the scaling of my application ?
version: '2'
services:
nodejs:
build:
context: ./
dockerfile: Dockerfile.nodejs
image: docker.shadoware.org/passprotect-server:1.0.0
expose:
- 3000
links:
- mongodb
environment:
- MONGODB_HOST=mongodb://mongodb:27017/passprotect
- NODE_ENV=production
- DEBUG=App:*
nginx:
image: docker.shadoware.org/nginx:1.2
links:
- nodejs
environment:
- APPLICATION_HOST=nodejs
- APPLICATION_PORT=3000
mongodb:
image: docker.shadoware.org/database/mongodb:3.2.7
Documentation states here that:
Containers for the linked service will be reachable at a hostname identical to the alias, or the service name if no alias was specified.
So I believe that you could just set your services names in that nginx conf file like:
upstream myservice {
yourservice1;
yourservice2;
}
as they would be exported as host entries in /etc/hosts for each container.
But if you really want to have that host:port information as environment variables you could write a script to parse that docker-compose.yml and define an .env file, or doing it manually.
UPDATE:
You can get that port information from outside the container, this will return you the ports
docker inspect --format='{{range $p, $conf := .NetworkSettings.Ports}} {{$p}} -> {{(index $conf 0).HostPort}} {{end}}' your_container_id
But if you want to do it from the inside of a containers then what you want is a service discovery system like zookeeper
There's a long feature request thread in docker's repo, about that.
One workaround solution caught my attention. You could try building your own nginx image based on that.

Docker compose: Multiple isolated environments on a single host with overridable volumes and ports

We have a requirement of creating multiple isolated envs of our app in a single host, using a single compose file.
I realized that by specifying the project name using -p option we can create multiple isolated envs using docker compose in a single host.
However is it possible to override the ports: and volumes: in the compose file for different environment, without having 2 seperate docker compose files?
For instance, I would like to override the following properties, preferably through command-line args.
For dev environment
ports:
8081:8080
volumes:
/etc/myapp/dev/properties/:/etc/myapp/properties
For QA environment
ports:
8082:8080
volumes:
/etc/myapp/qa/properties/:/etc/myapp/properties
You can use a template.yml and pass the variables you want to generate docker-compose.yml
First, create a template.yml with the following content:
version: "2"
...
ports:
"$HOST_PORT":8080
volumes:
"$HOST_VOLUME":/etc/myapp/properties
Now, you can create a script with the variables you want by environment. For dev environment it would look like this:
#!/bin/bash
# Variables to use in template.yml
export HOST_PORT="8081"
export HOST_VOLUME="/etc/myapp/dev/properties/"
# build docker-compose.yml from the template
source env.sh; rm -rf docker-compose.yml; envsubst < "template.yml" > "docker-compose.yml";
This will generate a docker-compose.yml with the concrete values.
Here's an usage example: https://github.com/bsferreira/mysql-fabric
Needed this as well and stumbled upon this question. There is built in support for isolating environments:
Multiple isolated environments on a single host
Compose uses a project name to isolate environments from each other. You can make use of this project name in several different contexts:
* on a dev host, to create multiple copies of a single environment, such as when you want to run a stable copy for each feature branch of a project
* on a CI server, to keep builds from interfering with each other, you can set the project name to a unique build number
* on a shared host or dev host, to prevent different projects, which may use the same service names, from interfering with each other
The default project name is the basename of the project directory. You can set a custom project name by using the -p command line option or the COMPOSE_PROJECT_NAME environment variable.
The default project directory is the base directory of the Compose file. A custom value for it can be defined with the --project-directory command line option.
https://docs.docker.com/compose/#multiple-isolated-environments-on-a-single-host
You can use Variable substitution with Declare default environment variables in file.
For example, my compose project structure is
C:\docker-compose
└───multiple-envs
│ .env
│ .env-dev
│ docker-compose.yml
│
├───dev-files
└───files
docker-compose.yml, file content below, you can set ports and other values using ${ENV_VAR} syntax. It will automatically substitute by docker compose cli.
You can instruct docker compose to show error message when the ${ENV_VAR} not set or empty using syntax ${HOST_MYSQL_PORT:?HOST_MYSQL_PORT is not set}, after :? is the error message.
services:
mysqldb:
image: mysql
restart: always
ports:
- ${HOST_MYSQL_PORT:?HOST_MYSQL_PORT is not set}:3306
volumes:
- type: bind
source: ${VOLUMES_SOURCE:?VOLUMES_SOURCE is not set}
target: /mnt
read_only: true
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:?MYSQL_ROOT_PASSWORD is not set}
MYSQL_USER: ${MYSQL_USER:?MYSQL_USER is not set}
MYSQL_PASSWORD: ${MYSQL_PASSWORD:?MYSQL_PASSWORD is not set}
.env file, set ENV_VAR=VAL in file called .env, let's say for production.
HOST_MYSQL_PORT=13306
VOLUMES_SOURCE=./files
MYSQL_ROOT_PASSWORD=p419460507
MYSQL_USER=u130085860
MYSQL_PASSWORD=p689273542
.env-dev file, set ENV_VAR=VAL in file called .env-dev, let's say for development.
HOST_MYSQL_PORT=23306
VOLUMES_SOURCE=./dev-files
MYSQL_ROOT_PASSWORD=dev419460507
MYSQL_USER=dev130085860
MYSQL_PASSWORD=dev689273542
To compose in different config, specify environment file when invoke docker compose cli, --env-file multiple-envs\.env for production or --env-file multiple-envs\.env-dev for development.
C:\docker-compose> docker-compose --project-directory multiple-envs --env-file multiple-envs\.env up --detach
[+] Running 2/2
- Network multiple-envs_default Created 0.6s
- Container multiple-envs-mysqldb-1 Started 1.8s
C:\docker-compose> docker-compose --project-directory multiple-envs --env-file multiple-envs\.env-dev up --detach
[+] Running 1/1
- Container multiple-envs-mysqldb-1 Started 11.7s

Resources