I am trying to setup a cluster of 3 RavenDB instances using docker-compose and I am having problems with the RavenDB server not picking up the values in the RAVEN_ environment variables.
At first, I was running a single instance, using this docker-compose file:
version: '3'
services:
ravendb:
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:8080"
- "38888:38888"
volumes:
- ../data:/opt/RavenDB/Server/RavenData
With a simple Dockerfile that used the latest raven image and simply copied a settings.json file into the container.
FROM ravendb/ravendb
COPY settings.json /opt/RavenDB/Server/settings.json
{
"License.Eula.Accepted": true,
"License": {/*License here*/},
"Setup.Mode": "Unsecured",
"Security.UnsecuredAccessAllowed": "PublicNetwork",
"ServerUrl": "http://0.0.0.0:8080",
"ServerUrl.Tcp": "tcp://0.0.0.0:38888"
}
Now that I am trying to setup 3 instances, I wanted to avoid this way of creating the containers, since I would have to have different Dockerfiles and a settings.json file for each one.
Therefore, I thought of using a single docker-compose file that creates three containers and configures each one with enviroment variables.
I started with a single instance, to see if any problems would arise:
version: '3'
services:
raven1:
container_name: raven1
image: ravendb/ravendb
ports:
- "8080:8080"
- "38888:38888"
environment:
- RAVEN_Security_UnsecuredAccessAllowed=PublicNetwork
- RAVEN_Setup_Mode=Unsecured
- RAVEN_License_Eula_Accepted=true
- "RAVEN_ServerUrl=http://0.0.0.0:8080"
- "RAVEN_ServerUrl_Tcp=tcp://0.0.0.0:38888"
volumes:
- ../data:/opt/RavenDB/Server/RavenData
And arise they did! Despite the environment variables being set correctly, they are not picked up by the server, and the settings.json file is the default one.
root#8ad95cc439d4:/opt/RavenDB/Server# env
RAVEN_ARGS=
RAVEN_Security_UnsecuredAccessAllowed=PublicNetwork
RAVEN_AUTO_INSTALL_CA=true
RAVEN_ServerUrl=http://0.0.0.0:8080
RAVEN_SETTINGS=
RAVEN_ServerUrl_Tcp=tcp://0.0.0.0:38888
RAVEN_IN_DOCKER=true
RAVEN_Setup_Mode=Unsecured
RAVEN_License_Eula_Accepted=true
RAVEN_DataDir=RavenData
root#8ad95cc439d4:/opt/RavenDB/Server# cat settings.json
{
"Security.UnsecuredAccessAllowed": "PrivateNetwork"
}
Any idea why this might be happening? I can't seem to find any mention of issues regarding this.
Do I understand correctly you expected the configuration from environment variables to be included in the settings.json file in the container?
If that's the case I would like to clarify that passing environment variables does not modify RavenDB's settings.json file. Instead RavenDB loads them up directly from the environment.
Configuration options are loaded in the following order of precedence:
command line arguments
settings.json configuration file
RAVEN_ prefixed environment variables
So if you wanted to override the configuration option Security.UnsecuredAccessAllowed found in settings.json file you would need to either change the file on the container or pass it as a CLI argument --Security.UnsecuredAccessAllowed PublicNetwork.
Both cases are supported by RavenDB docker images:
to clear the default settings.json you can pass RAVEN_SETTINGS={} environment variable to the container.
to pass command line arguments to the RavenDB server binary you can use RAVEN_ARGS environment variable. E.g. RAVEN_ARGS=--Security.UnsecuredAccessAllowed PublicNetwork
Related
I need some help with the following template:
services:
nginx:
image: nginx
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.nginx-${COMPOSE_PROJECT_NAME}.rule=Host(`fuu.bar`)"
networks:
- treafik
My goal is to create a template which I can use e. g. in portainer with almost zero configuration.
I thought that the following variables are available in docker-compose config but the expression ${COMPOSE_PROJECT_NAME} results in an empty string: docker-compose config
services:
nginx:
image: nginx
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.nginx-.rule=Host(`fuu.bar`)"
networks:
- treafik
Are there any default environment variables provided by docker-compose which I can use for environment interpolation?
---- Update
I use traefik (v2) as a reverse proxy. To make the containers available through treafik, you need to define routers on every service. The router name has to be unique. Lets imagine you deploy 2 or more stacks of the above template. The router name has to be unique for all services across all stacks. Because Im a lazy guy, I tried to simply integrate the environment variable COMPOSE_PROJECT_NAME (which I know is already unique in my setup because every stack must have a unique name). But the variable is not available when deploying the stack.
Of course, I could simply define the variable COMPOSE_PROJECT_NAME by myself in a .env-file, but i hoped that there are any default environment variables provided by docker.
You can use environment variables to passing strings to your docker file.
There are many ways through docker documentation. For example:
You can set default values for any environment variables referenced in the Compose file, or used to configure Compose, in an environment file named .env. The .env file path is as follows:
Starting with +v1.28, .env file is placed at the base of the project
directory
Project directory can be explicitly defined with the --file option or
COMPOSE_FILE environment variable. Otherwise, it is the current
working directory where the docker compose command is executed
(+1.28).
For previous versions, it might have trouble resolving .env file with
--file or COMPOSE_FILE. To work around it, it is recommended to use --project-directory, which overrides the path for the .env file. This inconsistency is addressed in +v1.28 by limiting the filepath to the
project directory.
Is there a way to pass environment variables from one service to the other inside docker-compose.yml ?
services:
testService:
environment:
TEST_KEY: 1234
testServiceTests:
environment:
TEST_KEY: I want to pull in the value 1234 here from service1
No.
However, there's an alternative. You may provide environment variables to all the services within the Docker Compose file by exposing them either from your shell, when you run the Compose or by using a special .env file, See documentation.
Using this approach, you would have a global (for the Compose) environment variable, say GLOBAL_TEST_KEY (it needn't have a different name) and you would be able to share this across multiple services:
services:
testService:
environment:
TEST_KEY: ${GLOBAL_TEST_KEY}
testServiceTests:
environment:
TEST_KEY: ${GLOBAL_TEST_KEY}
And then: docker-compose run -e GLOBAL_TEST_KEY="Some value" ....
Or, create a file called .env alongside docker-compose.yaml and, in .env:
GLOBAL_TEST_KEY="Some value"
And then: docker-compose run ...
NOTE No need to reference .env as it's included by default
I have the following compose file:
services:
myproject:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=https://+:443;http://+:80
- ASPNETCORE_HTTPS_PORT=44308
- PROJECT_NAME=MyProject
volumes:
- ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
- ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro
- ${APPDATA}/Turma/${PROJECT_NAME}/Logs:/var/logs/${PROJECT_NAME}
On the line:
- ${APPDATA}/Turma/${PROJECT_NAME}/Logs:/var/logs/${PROJECT_NAME}
It recognises ${APPDATA} but for ${PROJECT_NAME} it uses the literal string not the environment variable value.
Is there a way to make this work so the actual project name is used in path?
As far as I am aware, you cannot reference your env variables defined within the compose file later in the same compose file and have them interpreted. Your definition of $APPDATA works since that's set in the host's environment, not the compose file.
I tested both using the env variable and a .env file with compose 2.3 and 3, and neither worked.
I recommend wrapping your compose file in a run script where you can set the variables needed in your host shell, so you can have those interpreted properly. If you're deploying with a standard tool such as ansible, jenkins, etc. those can all set variables for you. This can look like the following:
#!/bin/bash
export PROJECT_NAME=foo
docker-compose up -d
unset PROJECT_NAME
Although it may not work for creating volumes, if you just need the variable to do something during the container's runtime (such as setting another environment variable), that can be put into an entrypoint script as well.
I want to set credentials to use Google Translate Api Client so I have to set environment variable GOOGLE_APPLICATION_CREDENTIALS that value is path to credential file (from Google Cloud).
When I have been used docker build and docker run it was pretty easy.
I have been used docker run
--env GOOGLE_APPLICATION_CREDENTIALS=/usr/src/app/CryptoTraderBot-901d31d199ce.json and environment variable has been set.
More difficult things come when I tried to set it in docker-compose. I have to use docker-compose because I need few containers so it is only way to achieve this.
Based on Docker compose environment variables documentation I created my docker-compose.yml file that looks like this:
version: "3"
services:
redis:
image: redis:4-alpine
crypto-bot:
build: .
depends_on:
- redis
environment:
- GOOGLE_APPLICATION_CREDENTIALS = /usr/src/app/CryptoTraderBot-901d31d199ce.json
I also have been tried multiple combination of path to .json file but none of this has been worked properly.
Have you got any idea how can I set it properly ?
While creating this question I have been resolve this problem in a funny and easy way but I have been thought that I post answer to help someone in the future with similiar problem.
All you have to do is remove " " (space) next = sign so two last lines of docker-compose.yml should looks like this:
environment:
- GOOGLE_APPLICATION_CREDENTIALS=/usr/src/app/CryptoTraderBot-901d31d199ce.json
Docker Compose has a newer feature called secrets. You can bind the credentials like this:
services:
secret-service:
build:
context: secret-service
environment:
- GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gcp-credentials
secrets:
- gcp-credentials
secrets:
gcp-credentials:
file: ./gcp-credentials.json
Reference: https://docs.docker.com/compose/compose-file/#secrets
I'm trying to get a docker-compose file working with multiple .env files, and I'm not having any luck. I'm trying to setup three .env files:
global settings that are the same across all container instances
environment-specific settings (stuff just for test or dev)
local settings - overridable things that a developer might need to change in case they have conflicts with, say, a port number
My docker-compose.yml file looks like this:
version: '2'
services:
db:
env_file:
- ./.env
- ./.env.${ENV}
- ./.env.local
image: postgres
ports:
- ${POSTGRES_PORT}:5432
.env looks like this:
POSTGRES_USER=myapp
and the .env.development looks like this:
POSTGRES_PASSWORD=supersecretpassword
POSTGRES_HOST=localhost
POSTGRES_PORT=25432
POSTGRES_DB=myapp_development
.env.local doesn't exist in this case.
After running ENV=development docker-compose up, I receive the following output:
$ ENV=development docker-compose up
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.db.ports is invalid: Invalid port ":5432", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
From that error message, it looks like none of my environment variables are being used. I just upgraded to the newest available docker-compose as well - same errors:
$ docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6
Any ideas here? Would be nice to have a single docker-compose.yml that would work across multiple environments.
In order to apply different/multiple env_files depending on the running environment, such as development/staging/production, I think a better way for docker-compose is to use multiple docker-compose yml files.
For example:
1. Start with a base file that defines the canonical configuration for the services.
docker-compose.yml
web:
image: example/my_web_app:latest
env_file:
- .env
2. Add the override file for development, as its name implies, can contain configuration overrides for existing services or entirely new services.
docker-compose.override.yml
web:
build: .
volumes:
- '.:/code'
ports:
- 8883:80
env_file:
- .env.dev
When you run docker-compose up it reads the overrides automatically.
3. Create another override file for the production environment.
docker-compose.prod.yml
web:
ports:
- 80:80
env_file:
- .env.prod
To deploy with this production Compose file you can run
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up
Note
My Docker version:
$ docker -v
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose -v
docker-compose version 1.22.0, build f46880fe
Reference: https://docs.docker.com/compose/extends/
Keep in mind that there are 2 different environments where you are defining variables. The host machine where you are executing the docker-compose command, and the container itself (running the db service in your case).
Your docker-compose.yml file has access to your host's environment variables. Hence ENV is reachable from the docker-compose command, but not these in your .env files.
On the contrary, the value for ENV is not reachable inside the container, but all variables defined in your .env files will.
I don't know if you really need your db container to access the variables defined on your .env.development. But at least seem that your host machine needs to have the content of that file, so when the docker-compose command is called, the POSTGRES_PORT variable is defined.
To fix your specific problem you would need to define the environment variables on your host machine too, not only for the container. You could do something like this:
#Set for host
ENV=development
#Also sets the variables on the host
source ./.env.$ENV
#POSTGRES_PORT defined in .env.development is used here
docker-compose up
#since env_file also contains .env.development, the variables will be reachable from the container.
Hope that helps.
There is a misconception regarding the .env file and the env_file option in the docker-compose.yml, as it is very ambiguous. Shin points it out very nicely in the github issue docker-compose doesn't use env_file. I will just quote his summary:
Variable substitution in your docker-compose.yml file will be pulled (in decreasing order of priority) from your shell's environment and your .env file.
Variables available in your container are a combination of values found in your env_file files and values described in the environment section of the service.
Those are two entirely separate sets of features.
while reading this page: https://docs.docker.com/compose/environment-variables/
and from my understanding, you should do the following:
for the global variables(that should not change) make an env file like so:
VAR1=VALUE1
VAR2=VALUE2
and for the others(that might change) you should add their name under environment in docker-compose.yml like this:
environment:
- VAR1
- VAR2
this will take the VAR1 and VAR2 values from the shell you are running docker-compose.
I hope this helps.