I'm struggling with providing environment-specific configuration (say dev/qa/prod) to an application container. The closest I've got to my goal is this docker-compose.yml:
version: '2'
services:
app:
image: my-application:latest
tty: true
volumes_from:
- configs:ro
configs:
image: my-configs:latest
tty: true
volumes:
- /configs/$ENV
If, say, we're deploying in the QA environment, the relevant configuration will be accessible at /configs/qa (from within the app service), meaning that in order to access these configs application will have to be aware of the environment it is running in, and I don't think it is something the application developers should be concerned about.
So my goal is to have config's /configs/$ENV accessible as simply /configs from within the app service. How can I achieve that? My current idea is rebuilding config's image:
FROM my-configs:latest
ARG env
RUN cp -rf /configs/$env /tmp/configs && rm -rf /configs && cp -rf /tmp/configs /configs
ENTRYPOINT /bin/sh
Then updated docker-compose.yml will look like this:
version: '2'
services:
app:
image: my-application:latest
tty: true
volumes_from:
- configs:ro
configs:
build:
context: .
args:
env: $ENV
tty: true
volumes:
- /configs
Are there any better options or should I go through with my idea?
You can directly use environment variables in the volumes: specification
volumes:
- /config/$ENV:/config
I'd put this in the service definition that's actually using it. You don't need to put this into its own image. I'd also tend to avoid volumes_from: in favor of being explicit about what volumes are actually getting mounted.
version: '3'
services:
app:
image: my-application:latest
volumes:
- /config/$ENV:/config
Related
I think I'm officially going mad as I can't seem to figure out what I am doing wrong with my docker configuration.
What I'm trying to achieve:
set up a Jupyter server using Jupyter/datascience-notebook as a base
copy a bunch if start scripts into the /home/my-user/.ipython/profile_default/startup folder
I don't want to use the default jovyan user but override to my own name
What I have:
a Dockerfile
a docker-compose.yml
the startup file I'm trying to copy
Dockerfile
FROM jupyter/datascience-notebook
USER $NB_USER
ENV JUPYTER_ENABLE_LAB=yes
COPY --chown=${NB_UID}:${NB_GID} notebook-startup-scripts/00-s3fs.py .ipython/profile_default/startup
docker-compose.yml
version: "3.9"
services:
jupyter-1:
build:
context: ./jupyter-lab
user: root
working_dir: /home/my-user/work
ports:
- "8888:8888"
volumes:
environment:
NB_USER: "my-user"
CHOWN_HOME: "yes"
restartable: "yes"
stdin_open: true
tty: true
command: /usr/local/bin/start-notebook.sh --ServerApp.allow_origin="*" --ServerApp.open_browser=False --ServerApp.allow_remote_access=True --ServerApp.trust_xheaders=True --ServerApp.password=${JUPYTER_PASSWORD}
I start the beast using docker compose up.
For some reason I always end up with my 00-s3fs.py file being synced to /home/jovyan/.../ as opposed to /home/my-user/.../ and it drives me nuts.
I tried all kinds of things, e.g. using
COPY --chown=${NB_UID}:${NB_GID} notebook-startup-scripts/00-s3fs.py /home/${NB_USER}.ipython/profile_default/startup
but nothing works :(.
If you move NB_USER within your docker-compose.yml:
version: "3.9"
services:
jupyter-1:
build:
context: ./jupyter-lab
args:
NB_USER: "my-user"
user: root
working_dir: /home/my-user/work
ports:
- "8888:8888"
volumes:
environment:
CHOWN_HOME: "yes"
restartable: "yes"
stdin_open: true
tty: true
command: /usr/local/bin/start-notebook.sh --ServerApp.allow_origin="*" --ServerApp.open_browser=False --ServerApp.allow_remote_access=True --ServerApp.trust_xheaders=True --ServerApp.password=${JUPYTER_PASSWORD}
This updates $NB_USER before the build process, unlike setting it within environment, which only sets $NB_USER after the image has already been build (and the file COPY already occurring).
I use docker and also use docker-compose for tie each container.
In my python flask code, refer environment variable like this.
import os
from app import db, create_app
app = create_app(os.getenv('FLASK_CONFIGURATION') or 'development')
if __name__ == '__main__':
print(os.getenv('FLASK_CONFIGURATION'))
app.run(host='0.0.0.0', debug=True)
And docker-compose.yml here.
version: '3.7'
services:
nginx:
build:
context: .
dockerfile: docker/nginx/dockerfile
container_name: nginx
hostname: nginx-prod
ports:
- '80:80'
networks:
- backend
links:
- web_project
depends_on:
- web_project
environment:
- FLASK_CONFIGURATION=production
mongodb:
build:
context: .
dockerfile: docker/mongodb/dockerfile
container_name: mongodb
hostname: mongodb-prod
ports:
- '27017:27017'
networks:
- backend
web_project:
build:
context: .
dockerfile: docker/web-prod/dockerfile
container_name: web_project
hostname: web_project_prod
ports:
- '5000:5000'
networks:
- backend
tty: true
depends_on:
- mongodb
links:
- mongodb
environment:
- FLASK_CONFIGURATION=production
networks:
backend:
driver: 'bridge'
I set FLASK_CONFIGURATION=production via environment command.
But when I execute, maybe FLASK_CONFIGURATION=production doesn't work.
I also tried to ENV FLASK_CONFIGURATION production to each dockerfile. (doesn't work too)
Strange thing is, When I enter to my container via bash(docker exec -it bash) and check the environment variable with export, it was set perfectly.
Is there any wrong code in my docker settings?
Thanks.
[SOLVED]
It is caused by supervisor.
When using supervisor, it's shell is isolated with original.
So we have to define our environment variables into supervisor.conf
Your flask code is looks ok, and as you said ... in bash this ENV variable exists,
My advice to you is to find way to put this variable to .env file in your project.
I will explain why i'm saying it regarding similar issue that i had with cron:
The cron run in his "own world" because the system run and execute it, and because of it he don't share those ENV variables that the bash of the main container process holding.
So i assume (please give feed back if not) that flask run too in similar way in his "own world" and don't have access to those ENV that Docker set.
So, there for, i created bash script that read all ENV variable and write them to the .env file of the project, this script run after the container created.
In this way, no matter from where and how you run the code/script ... those ENV variables will always be exists.
I'm working on a group project involving Docker that has a .env file, which looks like this:
DATABASE_URL=xxx
DJANGO_SETTINGS_MODULE=xxx
SECRET_KEY=xxx
Couldn't this just be declared inside the Dockerfile? If so, what is the advantage of making a .env file?
Not sure if I'm going in the right direction with this, but this Docker Docs page says (emphasis my own):
Your configuration options can contain environment variables. Compose
uses the variable values from the shell environment in which
docker-compose is run. For example, suppose the shell contains
POSTGRES_VERSION=9.3 and you supply this configuration:
db:
`image: "postgres:${POSTGRES_VERSION}"`
When you run docker-compose up with this configuration, Compose looks for the POSTGRES_VERSION environment variable in the shell and substitutes its value in. For this example, Compose resolves the image to postgres:9.3 before running the configuration.
If an environment variable is not set, Compose substitutes with an empty string. In the example above, if POSTGRES_VERSION is not set, the value for the image option is postgres:.
You can set default values for environment variables using a .env file, which Compose automatically looks for. Values set in the shell environment override those set in the .env file.
If we're using a .env file, then wouldn't I see some ${...} syntax in our docker-compose.yml file? I don't see anything like that, though.
Here's our docker-compose.yml file:
version: '3'
services:
server:
build:
context: ./server
dockerfile: Dockerfile
env_file: .env.dev
command: python3 manage.py runserver 0.0.0.0:8000
volumes:
- ./server:/app
ports:
- "8500:8000"
depends_on:
- db
stdin_open: true
tty: true
db:
image: postgres
client:
build:
context: ./client
dockerfile: Dockerfile
command: bash -c "npm install; npm run start"
volumes:
- ./client:/app
- /app/node_modules
ports:
- "3000:3000"
depends_on:
- server
Idea there is probably to have a place to keep secrets separated from docker-compose.yml, which you then can keep in VCS and/or share.
I'm using nestjs for my backend and using typeorm as ORM.
I tried to define my database and my application in an docker-compose file.
If I'm running my database as a container and my application from my local machine it works well. My program connects and creates the tables etc.
But if I try to connect the database from within my container or to start the container with docker-compose up it fails.
Always get an ECONNREFUSED Error.
Where is my mistake ?
docker-compose.yml
version: '3.1'
volumes:
dbdata:
services:
db:
image: postgres:10
volumes:
- ./dbData/:/var/lib/postgresql/data
restart: always
environment:
- POSTGRES_PASSWORD=${TYPEORM_PASSWORD}
- POSTGRES_USER=${TYPEORM_USERNAME}
- POSTGRES_DB=${TYPEORM_DATABASE}
ports:
- ${TYPEORM_PORT}:5432
backend:
build: .
ports:
- "3001:3000"
command: npm run start
volumes:
- .:/src
Dockerfile
FROM node:10.5
WORKDIR /home
# Bundle app source
COPY . /home
# Install app dependencies
#RUN npm install -g nodemon
# If you are building your code for production
# RUN npm install --only=production
RUN npm i -g #nestjs/cli
RUN npm install
EXPOSE 3000
.env
# .env
HOST=localhost
PORT=3000
NODE_ENV=development
LOG_LEVEL=debug
TYPEORM_CONNECTION=postgres
TYPEORM_HOST=localhost
TYPEORM_USERNAME=postgres
TYPEORM_PASSWORD=postgres
TYPEORM_DATABASE=mariokart
TYPEORM_PORT=5432
TYPEORM_SYNCHRONIZE=true
TYPEORM_DROP_SCHEMA=true
TYPEORM_LOGGING=all
TYPEORM_ENTITIES=src/database/entity/*.ts
TYPEORM_MIGRATIONS=src/database/migrations/**/*.ts
TYPEORM_SUBSCRIBERS=src/database/subscribers/**/*.ts
I tried to use links but it don't work in the container.
Take a look at your /etc/hosts inside the backend container. You will see
192.0.18.1 dir_db_1
or something like that. The IP will be different and dir will represent the dir you're in. Therefore, you must change TYPEORM_HOST=localhost to TYPEORM_HOST=dir_db_1.
Although, I suggest you set static names to your containers.
services:
db:
container_name: project_db
...
backend:
container_name: project_backend
In this case you can always be sure, that your container will have a static name and you can set TYPEORM_HOST=project_db and never worry about the name ever again.
You can create a network and share among two services.
Create network for db and backend services:
networks:
common-net: {}
and add the network to these two services. So your .yml file would like below after edit:
version: '3.1'
volumes:
dbdata:
services:
db:
image: postgres:10
volumes:
- ./dbData/:/var/lib/postgresql/data
restart: always
environment:
- POSTGRES_PASSWORD=${TYPEORM_PASSWORD}
- POSTGRES_USER=${TYPEORM_USERNAME}
- POSTGRES_DB=${TYPEORM_DATABASE}
ports:
- ${TYPEORM_PORT}:5432
networks:
- common-net
backend:
build: .
ports:
- "3001:3000"
command: npm run start
volumes:
- .:/src
networks:
- common-net
networks:
common-net: {}
Note1: After this change, there is no need to expose the Postgres port externally unless you have a reason for it. You can remove that section.
Note2: TYPEORM_HOST should be renamed to db. Docker would resolve the IP address of db service by itself.
I have the following docker-compose.yml file:
version: "3.3"
services:
api:
build: ./api
expose:
- '8080'
container_name: 'api'
ports:
- "8080:8080"
depends_on:
- db
stdin_open: true
tty: true
networks:
- api-net
db:
build: ./db
expose:
- '27017'
container_name: 'mongo'
ports:
- "27017:27017"
networks:
- api-net
networks:
api-net:
driver: bridge
and the Dockerfile for the api container is as follows:
FROM iron/go:dev
RUN mkdir /app
COPY src/main/main.go /app/
ENV SRC_DIR=/app
ADD . $SRC_DIR
RUN go get goji.io
RUN go get gopkg.in/mgo.v2
# RUN cd $SRC_DIR; go build -o main
CMD ["go", "run", "/app/main.go"]
If I run the code for main.go outside of a container it runs as expected, however if I try to run the container as part of docker-compose I get an exit 0. I have seen other threads on stackoverflow that have suggested using stdin_open and tty, but these have not helped. I have also tried creating an .env file in the same directory I issue docker-compose up from with COMPOSE_HTTP_TIMEOUT=8000 in it and this has not worked either. I am looking for helped and suggestions as to what I need to do in order for my api container to stay up.
I know that --verbose can be issued with docker-compose, however I'm not sure what I should be looking for in the output that this produces.
I finally managed to get to the bottom of this, in my code which worked outside of a container I had:
http.ListenAndServe("localhost:8080", mux)
the fix was to simply remove localhost such that I now have:
http.ListenAndServe(":8080", mux)