I want to set credentials to use Google Translate Api Client so I have to set environment variable GOOGLE_APPLICATION_CREDENTIALS that value is path to credential file (from Google Cloud).
When I have been used docker build and docker run it was pretty easy.
I have been used docker run
--env GOOGLE_APPLICATION_CREDENTIALS=/usr/src/app/CryptoTraderBot-901d31d199ce.json and environment variable has been set.
More difficult things come when I tried to set it in docker-compose. I have to use docker-compose because I need few containers so it is only way to achieve this.
Based on Docker compose environment variables documentation I created my docker-compose.yml file that looks like this:
version: "3"
services:
redis:
image: redis:4-alpine
crypto-bot:
build: .
depends_on:
- redis
environment:
- GOOGLE_APPLICATION_CREDENTIALS = /usr/src/app/CryptoTraderBot-901d31d199ce.json
I also have been tried multiple combination of path to .json file but none of this has been worked properly.
Have you got any idea how can I set it properly ?
While creating this question I have been resolve this problem in a funny and easy way but I have been thought that I post answer to help someone in the future with similiar problem.
All you have to do is remove " " (space) next = sign so two last lines of docker-compose.yml should looks like this:
environment:
- GOOGLE_APPLICATION_CREDENTIALS=/usr/src/app/CryptoTraderBot-901d31d199ce.json
Docker Compose has a newer feature called secrets. You can bind the credentials like this:
services:
secret-service:
build:
context: secret-service
environment:
- GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gcp-credentials
secrets:
- gcp-credentials
secrets:
gcp-credentials:
file: ./gcp-credentials.json
Reference: https://docs.docker.com/compose/compose-file/#secrets
Related
I am trying to setup a cluster of 3 RavenDB instances using docker-compose and I am having problems with the RavenDB server not picking up the values in the RAVEN_ environment variables.
At first, I was running a single instance, using this docker-compose file:
version: '3'
services:
ravendb:
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:8080"
- "38888:38888"
volumes:
- ../data:/opt/RavenDB/Server/RavenData
With a simple Dockerfile that used the latest raven image and simply copied a settings.json file into the container.
FROM ravendb/ravendb
COPY settings.json /opt/RavenDB/Server/settings.json
{
"License.Eula.Accepted": true,
"License": {/*License here*/},
"Setup.Mode": "Unsecured",
"Security.UnsecuredAccessAllowed": "PublicNetwork",
"ServerUrl": "http://0.0.0.0:8080",
"ServerUrl.Tcp": "tcp://0.0.0.0:38888"
}
Now that I am trying to setup 3 instances, I wanted to avoid this way of creating the containers, since I would have to have different Dockerfiles and a settings.json file for each one.
Therefore, I thought of using a single docker-compose file that creates three containers and configures each one with enviroment variables.
I started with a single instance, to see if any problems would arise:
version: '3'
services:
raven1:
container_name: raven1
image: ravendb/ravendb
ports:
- "8080:8080"
- "38888:38888"
environment:
- RAVEN_Security_UnsecuredAccessAllowed=PublicNetwork
- RAVEN_Setup_Mode=Unsecured
- RAVEN_License_Eula_Accepted=true
- "RAVEN_ServerUrl=http://0.0.0.0:8080"
- "RAVEN_ServerUrl_Tcp=tcp://0.0.0.0:38888"
volumes:
- ../data:/opt/RavenDB/Server/RavenData
And arise they did! Despite the environment variables being set correctly, they are not picked up by the server, and the settings.json file is the default one.
root#8ad95cc439d4:/opt/RavenDB/Server# env
RAVEN_ARGS=
RAVEN_Security_UnsecuredAccessAllowed=PublicNetwork
RAVEN_AUTO_INSTALL_CA=true
RAVEN_ServerUrl=http://0.0.0.0:8080
RAVEN_SETTINGS=
RAVEN_ServerUrl_Tcp=tcp://0.0.0.0:38888
RAVEN_IN_DOCKER=true
RAVEN_Setup_Mode=Unsecured
RAVEN_License_Eula_Accepted=true
RAVEN_DataDir=RavenData
root#8ad95cc439d4:/opt/RavenDB/Server# cat settings.json
{
"Security.UnsecuredAccessAllowed": "PrivateNetwork"
}
Any idea why this might be happening? I can't seem to find any mention of issues regarding this.
Do I understand correctly you expected the configuration from environment variables to be included in the settings.json file in the container?
If that's the case I would like to clarify that passing environment variables does not modify RavenDB's settings.json file. Instead RavenDB loads them up directly from the environment.
Configuration options are loaded in the following order of precedence:
command line arguments
settings.json configuration file
RAVEN_ prefixed environment variables
So if you wanted to override the configuration option Security.UnsecuredAccessAllowed found in settings.json file you would need to either change the file on the container or pass it as a CLI argument --Security.UnsecuredAccessAllowed PublicNetwork.
Both cases are supported by RavenDB docker images:
to clear the default settings.json you can pass RAVEN_SETTINGS={} environment variable to the container.
to pass command line arguments to the RavenDB server binary you can use RAVEN_ARGS environment variable. E.g. RAVEN_ARGS=--Security.UnsecuredAccessAllowed PublicNetwork
I am trying to deploy my django app to app engine using dockerfile and for that after following a few blogs such as these, I created a docker-compose.yml file but when I run the docker compose up command or docker-compose -f docker-compose-deploy.yml run --rm gcloud sh -c "gcloud app deploy" I get an error key cannot contain a space. See below:
For example:
$ docker compose up
key cannot contain a space
$ cat docker-compose.yml
version: '3.7'
services:
app:
build:
context: .
ports: ['8000:8000']
volumes: ['./app:/app']
Can someone please help me to fix this error? I have tried yamllint to validate the yaml file for any space/indentation type of error and it doesn't show any error to me.
EDIT:
Here is the content for file in the longer command:
version: '3.7'
services:
gcloud:
image: google/cloud-sdk:338.0.0
volumes:
- gcp-creds:/creds
- .:/app
working_dir: /app
environment:
- CLOUDSDK_CONFIG=/creds
volumes:
gcp-creds:
Ok this is resolved finally! After beating my head around, I was able to finally resolve this issue by doing the following things:
Unchecked the option to use "Docker Compose v2" from my docker desktop settings. Here is the setting in Docker Desktop
Closed the docker desktop app and restarted it.
Please try these steps in case you face the issue. Thanks!
Just adding another alt answer here that I confirmed worked for me when following the steps above did not. My case is slightly different, but as Google brought me here first I thought I'd leave a note.
Check your env var values for spaces!
This may only be applicable if you are using env_var files (appreciate that OP is not in the minimal example, hence saying this is different).
Unescaped spaces in variables will cause this cryptic error message.
So, given a compose file like this:
version: '3.7'
services:
gcloud:
image: google/cloud-sdk:338.0.0
volumes:
- gcp-creds:/creds
- .:/app
working_dir: /app
env_file:
- some_env_file.env
If some_env_file.env looks like this:
MY_VAR=some string with spaces
then we get the cryptic key cannot contain a space.
If instead we change some_env_file.env to be like this:
MY_VAR="some string with spaces"
then all is well.
The issue has been reported to docker-compose.
Google brought me here first, and when your suggestion sadly didn't work for me, it then took me to this reddit thread, where I found out the above.
Docker Compose (at least since v2) automatically parses .env files before processing the docker-compose.yml file, regardless of any env_file setting within the yaml file. If any of the variables inside your .env file contains spaces, then you will get the error key cannot contain a space.
Two workarounds exist at this time:
Rename your .env file to something else, or
Create an alternate/empty .env file, e.g. .env.docker and then explicitly set the --env-file parameter, i.e. docker compose --env-file .env.docker config.
Track the related issues here:
https://github.com/docker/compose/issues/6741
https://github.com/docker/compose/issues/8736
https://github.com/docker/compose/issues/6951
https://github.com/docker/compose/issues/4642
https://github.com/docker/compose/commit/ed18cefc040f66bb7f5f5c9f7b141cbd3afbbc89
https://docs.docker.com/compose/env-file/
One more thing to be care about - since Compose V2, this error may be raise in case you have inline comments in the env file used by Compose. For example
---
version: "3.7"
services:
backend:
build:
context: .
dockerfile: Dockerfile
env_file: .app.env
and that .app.env is like this
RABBIT_USER=user # RabbitMQ user
the same error may occur. To fix just move comment to its own line
# RabbitMQ user
RABBIT_USER=user
I have a custom build image (Python + project) which is working fine, I just want to deploy multiple container using docker-compose only environment variables will be changing rest is same can it be done from one service in docker-compose.yml
docker-compose.yml
version: '3'
services:
logsService:
image: logs:latest # logs image is already built and working fine
env_file:
- /home/controller/test/.env # File contains url, api_key, api_secret
by using docker-compose up I can provision one container but I want to provision 4 container with same code but different env variables (i.e. url, api_key, api_secret), any ideas how to achieve this will be appriciated
NOTE: I am new to docker, please pardon me if this is a silly question.
you can do this with the docker-compose command
docker-compose --scale --name="numberofcontainers"
so for if I wanted to launch my "webserver" container 3 times it would be
docker-compse --scale --webserver=3
as you have said you are new I would advise looking into a load balancer aswell. Or your other containers will be usless
So, something like this perhaps:
version: '3'
services:
logsService:
image: logs:latest
env_file:
- /home/controller/test/${ENV:-}.env
ENV=prod docker-compose --project-name prod up
assuming that your production config is in "prod.env"
I want to pass environment variables that is readable by applications spin up by docker-compose up.
What is the proper way of using docker-compose up with varying configuration settings?
I don't want to use .env & environment: config as the environment variables are changing frequently & it is insecure to save tokens in a file.
docker-compose run -e does work a bit, but loses many.
It does not map the ports that defined in docker-compose.yml services.
Also multiple services are defined in docker-compose.yml and I don't want to use depends_on just because docker-compose up doesn't work.
Let's say I define service in docker-compose.yml
service-a:
build:
context: .
dockerfile: DockerfileA
command: node serviceA.js
In my serviceA.js, I simply use the environment variable:
console.log("This is ", process.env.KEY, "running in service A");
When I run docker-compose run -e KEY=DockerComposeRun service-a
I do get the environment variable KEY read by serviceA.js
This is DockerComposeRun running in service A
However I could only get one single service running.
I could have use environment: in docker-compose.yml
environment:
- KEY=DockerComposeUp
But in my use case, each docker compose would have different environment variable values, meaning I would need to edit the file each time before I do docker-compose.
Also, not only single service would use the same environment variable, .env even done a better job, but it is not desired.
There doesn't seem to be a way to do the same for docker-compose up
I have tried KEY=DockerComposeUp docker-compose up,
but what I get is undefined .
Export doesn't work for me as well, it seems they are all about using environment variable for docker-compose.yml instead of for the applications in container
To safely pass sensitive configuration data to your containers you can use Docker secrets. Everything passed through Secrets is encrypted.
You can create and manage secrets using the commands below:
docker secret create
docker secret inspect
docker secret ls
docker secret rm
And use them in your docker-compose file, either referring to existing secrets (external) or use a file:
secrets:
my_first_secret:
file: ./secret_data
my_second_secret:
external: true
You can use environment like this:
service-a:
build:
context: .
dockerfile: DockerfileA
command: node serviceA.js
environment:
KEY=DockerComposeRun
Refer at: https://docs.docker.com/compose/environment-variables/
I would like to pull and push my images to different repositories depending on the environment I am deploying them to.
My .env file obviously only works in a single environment and currently i have to use multiple docker-compose files to get around this.
Simply I would like to do something like this:
version: '2.1'
services:
my_service:
build:
dockerfile: Dockerfile.somedockerfile
image: ${docker_repo_host}:${docker_repo_port}/myimage:latest
Where I could pass in docker_repo_host and docker_repo_port dynamically when I am building. I am unsure if there is a way to do this neatly but any suggestion would be appreciated.
Export the two variables
export docker_repo_host=your_host
export docker_repo_port=your_port
Now run the docker-compose command