I have a simple docker compose file to create a Mysql database for my app. But I cannot interpolate the environment variable MYSQL_PORT to set a custom port. Running docker compose up with the configuration below results in a random port being assigned to mysql.
The path to the env file does work, since I have env variables configuring the database.
docker-compose.yml
version: '3'
services:
mysql:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
volumes:
- mysql_data:/var/lib/mysql
env_file:
- ../../.env
ports:
- ${MYSQL_PORT}:3306
volumes:
mysql_data:
.env
MYSQL_PORT=3306
MYSQL_ROOT_PASSWORD=root
MYSQL_DATABASE=final_project_database
MYSQL_USER=db_user
MYSQL_PASSWORD=some_db_user_password
Use --env-file option with docker-compose up command. env_file declared in your MySQL service applies only for container env
Move your .env file to the same directory as the docker-compose file and change the env_file to point to it. That way both docker-compose and the container will use the same environment file.
Right now it's only the container that's using it.
version: '3'
services:
mysql:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
volumes:
- mysql_data:/var/lib/mysql
env_file:
- ./.env
ports:
- ${MYSQL_PORT}:3306
volumes:
mysql_data:
Related
I get the below error when I run docker-compose up, any pointers why I am getting this error
service "mysqldb-docker" refers to undefined volume mysqldb: invalid compose project
Also, is there a way to pass the $ENV value in CLI to docker-compose up , currently I have a ENV variable that specified dev, uat or prod that I use to specify the db name. Are there better alternatives to do this other than create a .env file explicitly for this
version: '3.8'
services:
mysqldb-docker:
image: '8.0.27'
restart: 'unless-stopped'
ports:
- "3309:3306"
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_PASSWORD=root
- MYSQL_DATABASE=reco-tracker-$ENV
volumes:
- mysqldb:/var/lib/mysql
reco-tracker-docker:
image: 'reco-tracker-docker:v1'
ports:
- "8083:8083"
environment:
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=root
- SPRING_DATASOURCE_URL="jdbc:mysql://mysqldb-docker:3309/reco-tracker-$ENV"
depends_on: [mysqldb-docker]
You must define volumes at the top level like this:
version: '3.8'
services:
mysqldb-docker:
# ...
volumes:
- mysqldb:/var/lib/mysql
volumes:
mysqldb:
You can pass environment variables from your shell straight through to a service’s containers with the ‘environment’ key by not giving them a value
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers
web:
environment:
- ENV
but from my tests you cant write $ENV in the compose file and expect it to read your env
for this you need to call docker-compose that way :
docker-compose run -e ENV web python console.py
see this : https://docs.docker.com/compose/environment-variables/#set-environment-variables-with-docker-compose-run
I have a compose file at /opt/myapp/docker-compose.yml (upon others at the same place).
I want to run docker-compose stop but from an other location on the host machine, let's say /home/username/scripts/. When I run:
docker-compose -f /opt/myapp/docker-compose.yml stop
it raises this error:
ERROR: The Compose file '/opt/myapp/docker-compose.yml' is invalid because:
services.myapp-db.ports contains an invalid type, it should be a number, or an object
services.myapp-nginx.ports contains an invalid type, it should be a number, or an object
I suspect this is because I've set up these port values using environment variables which are defined in a .env file that stands nearby the docker-compose.yml file. E.g.:
version: '3.8'
volumes:
dbdata: {}
services:
db:
image: postgres
environment:
POSTGRES_DB:
POSTGRES_USER:
POSTGRES_PASSWORD:
volumes:
- dbdata:/var/lib/postgresql/data
ports:
- ${PORT}:5432
web:
image: nginx
environment:
NGINX_HOST:
NGINX_PORT:
volumes:
- ./templates:/etc/nginx/templates
ports:
- "${PORT2}:80"
Question
How could I properly stop (and after that I'll need to start it again from the same location) the services defined in that docker-compose file, taking into account the correct environment variables, but from an other location on the host machine?
option 1:
export value of MY_APP_PORT variable directly in shell:
export MY_APP_PORT='put_your_ip_here' && docker-compose -f /opt/myapp/docker-compose.yml stop
option 2:
use --env-file parameter
docker-compose --env-file /opt/myapp/.env -f /opt/myapp/docker-compose.yml stop
https://docs.docker.com/compose/environment-variables/
option 3:
define path to your .env file in docker-compose.yaml
version: '3.8'
volumes:
dbdata: {}
services:
db:
image: postgres
environment:
POSTGRES_DB:
POSTGRES_USER:
POSTGRES_PASSWORD:
volumes:
- dbdata:/var/lib/postgresql/data
env_file:
- /opt/myapp/.env
ports:
- ${PORT}:5432
web:
image: nginx
environment:
NGINX_HOST:
NGINX_PORT:
volumes:
- ./templates:/etc/nginx/templates
env_file:
- /opt/myapp/.env
ports:
- "${PORT2}:80"
https://medium.com/better-programming/using-variables-in-docker-compose-265a604c2006
option 4 :
send value of MY_APP_PORT variable via -e flag
docker-compose -f -e MY_APP_PORT='put_your_ip_here' /opt/myapp/docker-compose.yml stop
I have two Rails 6 application and I am trying to deploy in aws ec2 instance with different port 8080 and 8081 but when I trying to run docker-compose up -d it start one rails application successfully and if I tries to run docker-compose up -d for second application, It make first application down and make another application up on particular Port
Below is my docker configuration for two applications.
Application 1
version: "3.4"
services:
app:
image: "dockerhub_repo/a_api:${TAG}"
# build:
# context: .
# dockerfile: Dockerfile
container_name: a_api_container
depends_on:
- database
- redis
- sidekiq
ports:
- "8080:8080"
volumes:
- .:/app
env_file: .env
environment:
RAILS_ENV: staging
database:
image: postgres:12.1
container_name: a_database_container
restart: always
volumes:
- db_data:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
sidekiq:
image: "dockerhub_repo/a_api:${STAG}"
container_name: a_sidekiq_container
environment:
RAILS_ENV: staging
env_file: .env
depends_on:
- redis
volumes:
- ".:/app"
redis:
image: redis:4.0-alpine
container_name: a_redis_container
volumes:
- "redis:/data"
volumes:
redis:
db_data:
Application 2
version: "3.4"
services:
app:
image: "dockerhub_repo/b_api:${PPTAG}"
build:
context: .
dockerfile: Dockerfile
container_name: b_api
depends_on:
- database
- redis
ports:
- "8081:8081"
volumes:
- .:/app
env_file: .env
environment:
RAILS_ENV: development
database:
image: postgres:12.1
container_name: pp_database
restart: always
volumes:
- db_data:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
redis:
image: redis:4.0-alpine
container_name: pp_redis
volumes:
db_data:
This Configuration works very well in local machine. It start both application in local on different port but it has some issue on aws ec2. I am not sure is any thing wrong in configuration?
Compose has the notion of a project name. If you add or delete containers from a docker-compose.yml file, it looks for existing containers that are labeled with the project name to figure out what needs to change. The project name is also included in the Docker names of containers, networks, and volumes.
You can configure the project name with the COMPOSE_PROJECT_NAME environment variable or the docker-compose -p option. If you don't configure it, it defaults to the base name of the current directory.
You clarify in a comment that the two docker-compose.yml files are in directories app1/backend and app2/backend. Since the base name of those directories are both backend, they have the same project name; so if you run docker-compose up in the app2/backend directory, it finds the existing containers for the backend project, sees they don't match what's in the docker-compose.yml file, and deletes them (even though you as the operator think they belong to the other project).
There are a couple of ways to get around this:
Rename one or the other directory; maybe move the docker-compose.yml files up to the top-level app1 and app2 directories.
In one or both directories, create a .env file that sets COMPOSE_PROJECT_NAME=app1. (Note that file is checked in the current directory, not necessarily the directory that contains the docker-compose.yml file.)
Set and change an environment variable export COMPOSE_PROJECT_NAME=app1.
Consistently use an option docker-compose -p app1 ... with all Compose commands.
I have a docker-compose file which looks like:
version: '3.8'
services:
scheduler:
image: apache/airflow
command: scheduler
restart: on-failure
env_file:
- .env
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
webserver:
image: apache/airflow
entrypoint: ./scripts/entrypoint.sh
restart: on-failure
depends_on:
- scheduler
env_file:
- .env
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./scripts:/opt/airflow/scripts
ports:
- "8080:8080"
It works as expected on running docker-compose up.
But if I remove the env_file option from yml and pass it in CLI - docker-compose --env-file .env up, then it doesn't pick the value of environment variables from the file.
This happens because the --env-file in CLI and env_file in service section are not exactly the same thing.
The former is actually a specification of the file that compose will use to substitute variables inside docker-compose.yml (the default value is .env).
The latter is specifying the file from which the variables will be loaded and passed to the container of the service.
And yes, it's kinda confusing.
Reference:
Read this and this.
I can't seem to get MySQL data to persist if I run $ docker-compose down with the following .yml
version: '2'
services:
# other services
data:
container_name: flask_data
image: mysql:latest
volumes:
- /var/lib/mysql
command: "true"
mysql:
container_name: flask_mysql
restart: always
image: mysql:latest
environment:
MYSQL_ROOT_PASSWORD: 'test_pass' # TODO: Change this
MYSQL_USER: 'test'
MYSQL_PASS: 'pass'
volumes_from:
- data
ports:
- "3306:3306"
My understanding is that in my data container using volumes: - /var/lib/mysql maps it to my local machines directory where mysql stores data to the container and because of this mapping the data should persist even if the containers are destroyed. And the mysql container is just a client interface into the db and can see the local directory because of volumes_from: - data
Attempted this answer and it did not work. Docker-Compose Persistent Data Trouble
EDIT
Changed my .yml as shown below and created a the dir ./data but now when I run docker-compose up --build the mysql container wont start throws error saying
data:
container_name: flask_data
image: mysql:latest
volumes:
- ./data:/var/lib/mysql
command: "true"
mysql:
container_name: flask_mysql
restart: always
image: mysql:latest
environment:
MYSQL_ROOT_PASSWORD: 'test_pass' # TODO: Change this
MYSQL_USER: 'test'
MYSQL_PASS: 'pass'
volumes_from:
- data
ports:
- "3306:3306"
flask_mysql | mysqld: Can't create/write to file '/var/lib/mysql/is_writable' (Errcode: 13 - Permission denied)
flask_mysql | 2016-08-26T22:29:21.182144Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
flask_mysql | 2016-08-26T22:29:21.185392Z 0 [ERROR] --initialize specified but the data directory exists and is not writable. Aborting.
The data container is a superfluous workaround. Data-volumes would do the trick for you. Alter your docker-compose.yml to:
version: '2'
services:
mysql:
container_name: flask_mysql
restart: always
image: mysql:latest
environment:
MYSQL_ROOT_PASSWORD: 'test_pass' # TODO: Change this
MYSQL_USER: 'test'
MYSQL_PASS: 'pass'
volumes:
- my-datavolume:/var/lib/mysql
volumes:
my-datavolume:
Docker will create the volume for you in the /var/lib/docker/volumes folder. This volume persist as long as you are not typing docker-compose down -v
There are 3 ways:
First way
You need specify the directory to store mysql data on your host machine. You can then remove the data container. Your mysql data will be saved on you local filesystem.
Mysql container definition must look like this:
mysql:
container_name: flask_mysql
restart: always
image: mysql:latest
environment:
MYSQL_ROOT_PASSWORD: 'test_pass' # TODO: Change this
MYSQL_USER: 'test'
MYSQL_PASS: 'pass'
volumes:
- /opt/mysql_data:/var/lib/mysql
ports:
- "3306:3306"
Second way
Would be to commit the data container before typing docker-compose down:
docker commit my_data_container
docker-compose down
Third way
Also you can use docker-compose stop instead of docker-compose down (then you don't need to commit the container)
first, you need to delete all old mysql data using
docker-compose down -v
after that add two lines in your docker-compose.yml
volumes:
- mysql-data:/var/lib/mysql
and
volumes:
mysql-data:
your final docker-compose.yml will looks like
version: '3.1'
services:
php:
build:
context: .
dockerfile: Dockerfile
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
mysql-data:
after that use this command
docker-compose up -d
now your data will persistent and will not be deleted even after using this command
docker-compose down
extra:- but if you want to delete all data then you will use
docker-compose down -v
You have to create a separate volume for mysql data.
So it will look like this:
volumes_from:
- data
volumes:
- ./mysql-data:/var/lib/mysql
And no, /var/lib/mysql is a path inside your mysql container and has nothing to do with a path on your host machine. Your host machine may even have no mysql at all. So the goal is to persist an internal folder from a mysql container.
Adding on to the answer from #Ohmen, you could also add an external flag to create the data volume outside of docker compose. This way docker compose would not attempt to create it. Also you wouldn't have to worry about losing the data inside the data-volume in the event of $ docker-compose down -v.
The below example is from the official page.
version: "3.8"
services:
db:
image: postgres
volumes:
- data:/var/lib/postgresql/data
volumes:
data:
external: true
Actually this is the path and you should mention a valid path for this to work. If your data directory is in current directory then instead of my-data you should mention ./my-data, otherwise it will give you that error in mysql and mariadb also.
volumes:
./my-data:/var/lib/mysql
Feasible bind mount solution:
mariadb:
image: mariadb:latest
restart: unless-stopped
environment:
- MARIADB_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD}
volumes:
- type: bind
source: /host/dir
target: /var/lib/mysql