docker-compose not finding environment variable $PWD on Ubuntu WSL 2 - docker

I am relatively new to docker. I have been trying to compose the file below:
version: "3"
services:
postgres:
restart: always
image: postgres
ports:
- "5432:5432"
environment:
- POSTGRES_DB=test_db
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD_FILE=/run/secrets/postgres-passwd
volumes:
- test_db:${PWD}
pgweb:
restart: always
image: sosedoff/pgweb
ports:
- "8081:8081"
environment:
- DATABASE_URL=postgres://postgres:POSTGRES_USER#POSTGRES_PASSWORD_FILE:5432/POSTGRES_DB?sslmode=disable
depends_on:
- postgres
volumes:
test_db:
What I am trying to do is mount the volume test_db to my current working directory by using the environment variable $PWD. When I run docker-compose up in my terminal I get the following warning:
The PWD variable is not set. Defaulting to a blank string.
Now it is important to note that I am currently using Ubuntu running on WSL2 on windows 10. Another thing to note is that I am running ZSH and not BASH.
I followed the exact steps mentioned in the documentation.
I also checked another question which seemed to be similar to mine but not quite the same, as it was possible to replace ${PWD} with ./ which simply does not work in my case.
When using ./ instead of $PWD I get the following error:
for pg_test_postgres_1 Cannot create container for service postgres:\
invalid volume specification: 'pg_test_test_db:.:rw': invalid mount config\
for type "volume": invalid mount path: '.' mount path must be absolute

If you are trying to see what you can an cannot do, this is something you cannot do. Docker does not load an environment variable that would normally be set by the shell, zsh or bash, doesn't matter. And yes, it's the shell that sets $PWD and $OLDPWD. Docker CAN define a variable that will be passed to the distro as an environment variable and also be used by Docker at the time of the container build. Also volumes need to be defined using absolute paths.
Also, like David Maze mentions, your PostgreSQL data folder needs to be specifically in /var/lib/postgresql/data or that folder needs to be symlinked to a different arbitrary folder where PostgreSQL has read-write access. The point of a container is to build it for your needs so under normal circumstances you should know where everything goes and set volumes' paths explicitly.

Related

persist %USERPROFILE% folder using docker compose volume

I am searching on how to persist user profile folder in volume mounting
I have folder C:\Users\ABEL\source\repos which needs to be persisted for a windows container. The username should be from the host. It is unknown.
Below is my docker-compose file, The volume section is not correct.
Any comments will be helpful. Thanks in advance
version: '3.4'
services:
directoryservice:
image: abc-directoryservice:latest
build: .
ports:
- "44309:44309"
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=https://+:44309;
- ASPNETCORE_Kestrel__Certificates__Default__Password=welcome123#
- ASPNETCORE_Kestrel__Certificates__Default__Path=/https/aspnetapp.pfx
volumes:
- ./devops/https/abccert.pfx:/https/aspnetapp.pfx:ro
# - "$env:USERPROFILE/source:$env:USERPROFILE/source"
- ${Env:USERPROFILE}\source:${Env:USERPROFILE}\source
I get below error
invalid interpolation format for services.directoryservice.volumes.[]: "${Env:USERPROFILE}\\source:${Env:USERPROFILE}\\source". You may need to escape any $ with another $.
The $env:USERPROFILE/ ${env:USERPROFILE} syntax is specific to PowerShell.
Judging by the docs, docker-compose uses its own syntax: $USERPROFILE / ${USERPROFILE}
You report a follow-up problem, namely that the Windows-style path stored in $USERPROFILE (%USERPROFILE%) (e.g. C:\Users\jdoe\source) isn't converted to a Unix-style path (e.g. c/Users/jdoe/source)
This answer suggests that you must set environment variable COMPOSE_CONVERT_WINDOWS_PATHS to 1, before running your docker-compose command.
E.g., in a PowerShell session:
$env:COMPOSE_CONVERT_WINDOWS_PATHS=1
Consider adding this statement to your $PROFILE file so that it takes effect in future PowerShell sessions too.

define volumes in docker-compose.yaml

I am writing a docker-compose.yaml file for my project. I have checked the volumes documentation here .
I also understand the concept of volume in docker that I can mount a volume e.g. -v my-data/:/var/lib/db where my-data/ is a directory on my host machine while /var/lib/db is the path inside database container.
My confuse is with the link I put above. There it has the following sample:
version: "3.9"
services:
db:
image: db
volumes:
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
- data-volume:/var/lib/backup/data
volumes:
data-volume:
I wonder does it mean that I have to create a directory named data-volume on my host machine? What if I have a directory on my machine with path temp/my-data/ and I want to mount that path to the database container /var/lib/db ? Should I do something like below?
version: "3.9"
services:
db:
image: db
volumes:
- temp/my-data/:/var/lib/db
volumes:
temp/my-data/:
My main confusion is the volumes: section at the bottom, I am not sure whether the volume name should be the path of my directory or should be just literally a name I give & if it is the latter case then how could the given name be mapped with temp/my-data/ on my machine? The sample doesn't indicate that & is ambiguous to clarify that.
Could someone please clarify it for me?
P.S. I tried with above docker-compose I guessed, ended up with the error:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
volumes value 'temp/my-data/' does not match any of the regexes: '^[a-zA-Z0-9._-]+$'
Mapped volumes can either be files/directories on the host machine (sometimes called bind mounts in the documentation) or they can be docker volumes that can be managed using docker volume commands.
The volumes: section in a docker-compose file specify docker volumes, i.e. not files/directories. The first docker-compose in your post uses such a volume.
If you want to map a file or directory (like in your last docker-compose file), you don't need to specify anything in the volumes: section.
Docker volumes (the ones specified in the volumes: section or created using docker volume create) are of course also stored somewhere on your host computer, but docker manages that and you shouldn't normally need to know where or what the format is.
This part of the documentation is pretty good about explaining it, I think https://docs.docker.com/storage/volumes/
As #HansKilian mentions, you don't need both volumes and services.volumes. To use services.volumes, map the host directory to the container directory like this:
services:
db:
image: db
volumes:
- /host/path/lib/db:/container/path/lib/db
With that, the directory /host/path/lib/db on the host machine will be used by the container and available at /container/path/lib/db.
Now, if you're like me, I get really confused with fake examples, so let's say the real directory on your host machine is /var/lib/db and you just want to see it at /db when you run a shell in Docker (i.e., docker exec -it /bin/bash container-id).
docker-compose.yaml would look like this:
services:
db:
image: db
volumes:
- /var/lib/db:/db
Now when you run the shell, cd /logs and ls, you'll see the same results as if you'd cd /var/lib/db on the host.
If you want to use the volumes section to indicate a global volume to use, you first have to create that volume using docker volume create. The documentation Hans linked includes steps to do this. The syntax of /host/path:/container/path is replaced by volume-name:/container/path. Then, once defined, you'd alter your docker-compose.yaml to be more like this:
services:
db:
image: db
volumes:
- your-global-volume-name:/db
volumes:
your-global-volume-name:
external: true
Note that I have not tested or used the this configuration. I'm assuming it's correct based on the other method working and the few changes I can identify in the docs.

Dockerfiles not found when running docker-compose on windows (boot2docker)

I'm hoping that I've just missed something terribly obvious, but here's the situation I'm faced with.
Problem
Running docker-compose on windows after following docker-compose install steps from the website
docker-compose.yml file works fine on unix systems (have tested on Mac)
Currently fails immediately on Windows when it cannot locate any Dockerfiles for the services defined in the yml file. Here's the error:
NOTE: The image above might be a bit confusing. The environment variable below is called GOPATH, but the folder on my colleague's computer is also called GOPATH. This gives the impression that the env var isn't set correctly, but it is indeed.
version: '3'
services:
renopost:
depends_on:
- reno-cassandra
- reno-kafka
- reno-consul
build:
context: ${GOPATH}/src/renopost
dockerfile: ${GOPATH}/src/renopost/docker/dev/Dockerfile
container_name: renopost
image: renopost
ports:
- "4000:4000"
volumes:
- ${GOPATH}/src/renopost:/go/src/renopost
Above is a snippet of the docker-compose.yml file that is being run. The GOPATH env variable is indeed set and when following the directory path listed, I can confirm the file exists in that location.
Is there some interaction here with the OracleBoxVM that boot2docker uses where it isn't actually finding that file?

File in docker-entrypoint-initdb.d never get executed when using docker compose

I'm using Docker Toolbox on Windows 10
I can access the php part succesfully via http://192.168.99.100:8000, I have been working around with the mariadb part but still having several problems
I have an sql file as /mariadb/initdb/abc.sql so I should be copied into /docker-entrypoint-initdb.d, after the container is created I use docker-compose exec mariadb to access the container, there is the file as /docker-entrypoint-initdb.d/abc.sql but the file never get executed, I also have tested to import the sql file to the container manually, it was succesful so the sql file is valid
I don't quite understand about the data folder mapping, and what to do to get the folder sync with the container, I always get the warning when recreate the container using docker-compose up -d
WARNING: Service "mariadb" is using volume "/var/lib/mysql" from the previous container. Host mapping "/.../mariadb/data" has no effect. Remove the existing containers (with docker-compose rm mariadb) to use the Recreating db ... done
Questions
How to get the sql file in /docker-entrypoint-initdb.d to be executed ?
What is the right way to map the data folder with the mariadb container ?
Please guide
Thanks
This is my docker-compose.yml
version: "3.2"
services:
php:
image: php:7.1-apache
container_name: web
restart: always
volumes:
- /.../php:/var/www/html
ports:
- "8000:80"
mariadb:
image: mariadb:latest
container_name: db
restart: always
environment:
- MYSQL_ROOT_PASSWORD=12345
volumes:
- /.../mariadb/initdb:/docker-entrypoint-initdb.d
- /.../mariadb/data:/var/lib/mysql
ports:
- "3306:3306"
For me the issue was the fact that Docker didn't clean up my mounted volumes from previous runs.
Doing a:
docker volume ls
Will list any volumes, and if previous exist, then run 'rm' command on the volume to remove it.
As stated on docker mysql docks, scripts in the '/docker-entrypoint-initdb.d' folder is only evalutated the first time the container runs, and if a previous volume remains, it won't run the scripts.
As for the mapping, you simply need to mount your script folder to the '/docker-entrypoint-initdb.d' folder in the image:
volumes:
- ./db/:/docker-entrypoint-initdb.d
I have a single script file in a folder named db, relative to my docker-compose file.
In your Docker file for creating mariaDB, at the end add the abc.sql file to your docker entry point like so:
COPY abc.sql /docker-entrypoint-initdb.d/
Remove the - /.../mariadb/initdb:/docker-entrypoint-initdb.d mapping as any file copied into the entry point will be executed.
Note: Windows containers do not execute anything in docker-entrypoint-initdb.d/

Why does variables substitution not work when using multiple env_files in docker-compose.yml?

I'm trying to get a docker-compose file working with multiple .env files, and I'm not having any luck. I'm trying to setup three .env files:
global settings that are the same across all container instances
environment-specific settings (stuff just for test or dev)
local settings - overridable things that a developer might need to change in case they have conflicts with, say, a port number
My docker-compose.yml file looks like this:
version: '2'
services:
db:
env_file:
- ./.env
- ./.env.${ENV}
- ./.env.local
image: postgres
ports:
- ${POSTGRES_PORT}:5432
.env looks like this:
POSTGRES_USER=myapp
and the .env.development looks like this:
POSTGRES_PASSWORD=supersecretpassword
POSTGRES_HOST=localhost
POSTGRES_PORT=25432
POSTGRES_DB=myapp_development
.env.local doesn't exist in this case.
After running ENV=development docker-compose up, I receive the following output:
$ ENV=development docker-compose up
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.db.ports is invalid: Invalid port ":5432", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
From that error message, it looks like none of my environment variables are being used. I just upgraded to the newest available docker-compose as well - same errors:
$ docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6
Any ideas here? Would be nice to have a single docker-compose.yml that would work across multiple environments.
In order to apply different/multiple env_files depending on the running environment, such as development/staging/production, I think a better way for docker-compose is to use multiple docker-compose yml files.
For example:
1. Start with a base file that defines the canonical configuration for the services.
docker-compose.yml
web:
image: example/my_web_app:latest
env_file:
- .env
2. Add the override file for development, as its name implies, can contain configuration overrides for existing services or entirely new services.
docker-compose.override.yml
web:
build: .
volumes:
- '.:/code'
ports:
- 8883:80
env_file:
- .env.dev
When you run docker-compose up it reads the overrides automatically.
3. Create another override file for the production environment.
docker-compose.prod.yml
web:
ports:
- 80:80
env_file:
- .env.prod
To deploy with this production Compose file you can run
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up
Note
My Docker version:
$ docker -v
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose -v
docker-compose version 1.22.0, build f46880fe
Reference: https://docs.docker.com/compose/extends/
Keep in mind that there are 2 different environments where you are defining variables. The host machine where you are executing the docker-compose command, and the container itself (running the db service in your case).
Your docker-compose.yml file has access to your host's environment variables. Hence ENV is reachable from the docker-compose command, but not these in your .env files.
On the contrary, the value for ENV is not reachable inside the container, but all variables defined in your .env files will.
I don't know if you really need your db container to access the variables defined on your .env.development. But at least seem that your host machine needs to have the content of that file, so when the docker-compose command is called, the POSTGRES_PORT variable is defined.
To fix your specific problem you would need to define the environment variables on your host machine too, not only for the container. You could do something like this:
#Set for host
ENV=development
#Also sets the variables on the host
source ./.env.$ENV
#POSTGRES_PORT defined in .env.development is used here
docker-compose up
#since env_file also contains .env.development, the variables will be reachable from the container.
Hope that helps.
There is a misconception regarding the .env file and the env_file option in the docker-compose.yml, as it is very ambiguous. Shin points it out very nicely in the github issue docker-compose doesn't use env_file. I will just quote his summary:
Variable substitution in your docker-compose.yml file will be pulled (in decreasing order of priority) from your shell's environment and your .env file.
Variables available in your container are a combination of values found in your env_file files and values described in the environment section of the service.
Those are two entirely separate sets of features.
while reading this page: https://docs.docker.com/compose/environment-variables/
and from my understanding, you should do the following:
for the global variables(that should not change) make an env file like so:
VAR1=VALUE1
VAR2=VALUE2
and for the others(that might change) you should add their name under environment in docker-compose.yml like this:
environment:
- VAR1
- VAR2
this will take the VAR1 and VAR2 values from the shell you are running docker-compose.
I hope this helps.

Resources