I am trying to set an arg for docker compose using an output of linux command as my example:
args:
ID_GITLAB: $(id -u $USER)
but when I run my compose I get following error:
ERROR: Invalid interpolation format for "build" option in service "gpc-fontes-ci": "$(id -u $USER)"
Just do
USER_ID=$(id -u $USER) docker-compose
With the Compose file using a regular variable
args:
ID_GITLAB: $USER_ID
You need to escape $ character with another $, e.g.
args:
ID_GITLAB: $$(id -u $USER)
It's the same rule for command.
See: Variable substitution.
You can use a $$ (double-dollar sign) when your configuration needs a literal dollar sign. This also prevents Compose from interpolating a value, so a $$ allows you to refer to environment variables that you don’t want processed by Compose.
For example:
web:
build: .
command: "$$VAR_NOT_INTERPOLATED_BY_COMPOSE"
Related
version: '3.7'
services:
pgdump:
image: postgres:alpine
command: pg_dump -f "backup-`date -u -Iseconds`.pg_restore" $DATABASE_URL
This produces a file named
backup-`date -u -Iseconds`.pg_restore
instead of the desired
backup-2021-04-14T16:42:54+00:00.pg_restore.
I also tried:
command: pg_dump -f backup-`date -u -Iseconds`.pg_restore $DATABASE_URL
command: pg_dump -f "backup-${date -u -Iseconds}.pg_restore" $DATABASE_URL
command: pg_dump -f backup-${date -u -Iseconds}.pg_restore $DATABASE_URL
All of them yield different errors.
As of April 2021 command substitution is not supported by docker-compose according to this GitHub issue.
As a workaround in my use case, one could either use native docker run commands, where substitution works or use an .env file.
Current command
The date command itself is incorrect. Try running it on its own
date -u -Iseconds
echo `date -u -Iseconds`
From your command, I presume you want date in UTC seconds since epoch? Epoch by itself is UTC. So you just need seconds since Epoch. No need for -u parameter.
Solution
Here's the correct command in two forms:
A.
command: pg_dump -f "backup-`date +'%s'`.pg_restore" $DATABASE_URL
B.
command: pg_dump -f "backup-$(date +'%s').pg_restore" $DATABASE_URL
Explanation
There are multiple things to watch out for in the command you provided:
Notice the double quotes around the file name? This means you cannot nest another double-quote within the original outer pair without escaping the inner ones with \. Another alternative option is to use as many single-quote pairs you want within a pair of double-quotes. See this answer and this excerpt about 2.2.2 Single-Quotes and 2.2.3 Double-Quotes.
For string interpolation, you can use either $() or `` notation. But NOT within single-quotes as I said.
As a dry-run test, create a file directly with said notations:
vi "backup-`date +'%s'`.txt"
vi "backup-$(date +'%s').txt"
As for date format. Both GNU/date BSD/date accept %s to represent seconds since Epoch. Find "%s" in ss64 or man7 or cyberciti.
Docker-related, watch out what command does. Source:
command overrides the the default command declared by the container image (i.e. by Dockerfile's CMD).
You can create the filename and store it as a variable with shell command before doing the pg_dump:
version: '3.7'
services:
pgdump:
image: postgres:alpine
entrypoint: ["/bin/sh","-c"]
command: >
"FILENAME=backup-`date -u -Iseconds`.pg_restore
&& pg_dump -f $$FILENAME $$DATABASE_URL"
Successfully tested against Docker image for postgres 13.6.
I would like to export the environment variable in the Makefile. In this case, is to get the IP for debugging with docker
Makefile
start:
export XDEBUG_REMOTE_HOST=$$(/sbin/ip route|awk '/kernel.*metric/ { print $$9 }') \
; docker-compose up -d
Update from answers:
version: '3.5'
services:
php:
build:
context: .
dockerfile: docker/php/Dockerfile
environment:
- XDEBUG_CONFIG="idekey=docker"
- XDEBUG_REMOTE_HOST=${XDEBUG_REMOTE_HOST}
output:
$ make start
export XDEBUG_REMOTE_HOST=$(/sbin/ip route|awk '/kernel.*metric/ { print $9 }') \
; docker-compose up -d
Starting service_php ... done
$ docker-compose exec php bash
WARNING: The XDEBUG_REMOTE_HOST variable is not set. Defaulting to a blank string.
You need to make sure the variable assignment and the docker command run in the same shell. Trivially, put them in the same rule:
start:
XDEBUG_REMOTE_HOST=$$(/sbin/ip route|awk '/kernel.*metric/ { print $$9 }') \
docker-compose up -d
I took out the # because it's probably simply a bad idea, especially if you need to understand what's going on here. You can use make -s once your Makefile is properly tested if you don't want to see what it's doing.
The purpose of export is to expose a variable to subprocesses, but that's not necessary here. Instead, we use the shell's general
variable=value anothervar=anothervalue command
syntax to set the value of a variable for the duration of a single command.
If the internals of docker-compose require the variable to be exported, then of course, you can do that too:
start:
export XDEBUG_REMOTE_HOST=$$(/sbin/ip route|awk '/kernel.*metric/ { print $$9 }') \
; docker-compose up -d
Notice how the backslash at the end of the first line of the command list joins the two commands on a single logical line, so they get passed to the same shell instance, and the ; command separator is required to terminate the first command. (I put the semicolon at beginning of line as an ugly reminder to the reader that this is all one command line.)
Specifically for docker-compose, the customary way to set a variable from the command line is with a specific named option;
start:
docker-compose up -e XDEBUG_REMOTE_HOST=$$(/sbin/ip route|awk '/kernel.*metric/ { print $$9 }') -d
There are other ways to solve this such as the GNU Make .ONESHELL directive but this is simple and straightforward, and portable to any Make.
If you assume that the route exists when make is first invoked, you can assign a make variable as opposed to a shell variable as follows:
export XDRH_MAKE_VAR:=$(shell /sbin/ip route|awk '/kernel.*metric/ { print $$9 }')
start:
#echo XDHR_MAKE_VAR=$(XDRH_MAKE_VAR)
XDEBUG_REMOTE_HOST=$(XDRH_MAKE_VAR) docker-compose up -d
XDRH_FILE:
echo $(XDRH_MAKE_VAR) > $#
someother_target:
XDEBUG_REMOTE_HOST=$(XDRH_MAKE_VAR) some_other_command
command_that_uses_it_as_param $(XDRH_MAKE_VAR)
NOTE_does_not_work:
XDEBUG_REMOTE_HOST=$(XDRH_MAKE_VAR) echo $$XDEBUG_REMOTE_HOST
The last one does not work, because the bash shell will expand $XDEBUG_REMOTE_HOST before assigning it (See here). Also, the variable is set at make parse time, so if any of your rules effect the route, then this will not be reflected in its value.
If you want to access the value in the shell afterwards, you would want to do something like:
bash> make start XDRH_FILE
bash> XDEBUG_REMOTE_HOST=`cat XDRH_FILE`
bash> docker-compose exec php bash
I have a line in my Dockerfile:
&& echo "xdebug.remote_host=192.168.0.216" >> /usr/local/etc/php/conf.d/xdebug.ini`
I want to make the IP dynamic. How would I get the host IP in there?
You need to use build-time variables (–build-arg).
This flag allows you to pass the build-time variables that are accessed like regular environment variables in the RUN instruction of the Dockerfile.
So, Dockerfile is modified to:
ARG IP_ADDRESS
RUN ... && echo "xdebug.remote_host=$IP_ADDRESS" >> /usr/local/etc/php/conf.d/xdebug.ini`
And you just need to define build-time variable IP_ADDRESS during image building:
$ docker build --build-arg IP_ADDRESS=<IP_ADDRESS> .
If you use docker-compose:
1. Create file .env with the following content:
IP_ADDRESS="<IP_ADDRESS>"
You can make it every time like (example is for a linux machine):
IP_ADDRESS=$(ip a | grep <interface> | grep inet | awk '{print $2}' | awk -F'/' '{print $1}')
echo "IP_ADDRESS=$IP_ADDRESS" > .env
2. Use the following docker-compose.yaml to build your image:
version: '3'
services:
myservice:
build:
context: .
args:
IP_ADDRESS: ${IP_ADDRESS}
3. Build the above image:
docker-compose build
There's no simple in built way to get the Docker host IP (unless you are using Docker for Mac)
Entrypoint
It's best not to set a Docker host IP at build time, otherwise the image will be tied to the host it was built on and won't work anywhere else.
An ENTRYPOINT can be used to do the config setup based on an environment variable and then pass through all commands to the container:
#!/bin/sh
if [ -n "$IP_ADDRESS" ]; then
echo "xdebug.remote_host=$IP_ADDRESS" >> /usr/local/etc/php/conf.d/xdebug.ini
else
echo "No environment variable IP_ADDRESS set for xdebug"
fi
exec "$#"
Then run with:
docker run -e IP_ADDRESS=192.168.51.5 me/app-debug
Docker for Mac
On Docker for Mac 17.12+ you can use the host name docker.for.mac.host.internal
Xdebug
Another option is setting xdebug.remote_connect_back = 1 so you don't need a specific remote_host for xdebug.
Build
Nicolay's answer covers the build time setup.
Running the command docker-compose run -e TYPE=result mongo_db_backup should give me the value of the given TYPE variable:
mongo_db_backup:
image: 'mongo:3.4'
volumes:
- '/backup:/backup'
command: sh -c '$$(echo $TYPE)'
But instead I get the error The TYPE variable is not set. Defaulting to a blank string. What am I doing wrong
It happens that Compose expands $TYPE before it gets to the inside of the container. Compose looks for the $TYPE environment variable in the shell or host environment and substitutes its value in.
This will work with the following terminal command:
docker-compose.yml
command: sh -c 'echo $TYPE'
terminal command
TYPE='hello world' docker-compose run web
When there is no $TYPE environment variable in the host machine, Compose sets the value of $TYPE to an empty string and outputs a warning.
Compose needs to be informed not to expand $TYPE since we want it expanded inside of the shell running in the container.
For this use
docker-compose.yml
command: sh -c "echo $$TYPE"
Prepending a dollar symbol to $TYPE escapes it.
Reference:
Variable Substitution in Compose
I am using Docker 17.04.0-ce, build 4845c56 with docker-compose 1.12.0, build b31ff33 on Ubuntu 16.04.2 LTS. I simply want to pass an environment variable and display it from my script running in a container. I am doing this according to the documentation https://docs.docker.com/compose/compose-file/#environment . The problem is that the variable is not passed to the container.
My docker-compose.yml file:
env-file-test:
build: .
dockerfile: Dockerfile
environment:
- DEMO_VAR
My Dockerfile:
FROM alpine
COPY docker-start.sh /
CMD ["/docker-start.sh"]
And the docker-start.sh file:
#!/bin/sh
echo "DEMO_VAR Var Passed in: $DEMO_VAR"
I try to set the variable in my current terminal session and pass it to the container:
$ export DEMO_VAR=aabbdd
$ echo $DEMO_VAR
aabbdd
$ sudo docker-compose up
Starting envfiletest_env-file-test_1
Attaching to envfiletest_env-file-test_1
env-file-test_1 | DEMO_VAR Var Passed in:
envfiletest_env-file-test_1 exited with code 0
So you can see that the variable DEMO_VAR is empty!
I also tried using variables in docker-compose.yml like this: DEMO_VAR=${DEMO_VAR} but then when I run sudo docker-compose up, I get a warning: "WARNING: The DEMO_VAR variable is not set. Defaulting to a blank string.".
What am I doing wrong? What should I do to pass the variable to the container?
I found a solution. Answering my own question...
The problem was with the sudo command. It turned out that it does not pass environment variables by default. There are some possible solutions:
Use sudo -E. Demo:
$ export DEMO_VAR=aabbdd
$ echo $DEMO_VAR
aabbdd
$ sudo -E docker-compose up
env-file-test_1 | DEMO_VAR Var Passed in: aabbdd
Use sudo VAR=value:
sudo DEMO_VAR=$DEMO_VAR docker-compose up
Add environment variables to the sudoers file (https://stackoverflow.com/a/8636711)
Use docker without sudo (https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo)
you should use ENV in your Dockerfile, and avoid export.
See the doc
https://docs.docker.com/engine/reference/builder/#env