I want to make docker-compose (v2.12.2) load env variables from both .env and .env.local
I tried the method mentioned here:
docker-compose --env-file <(cat "./.env" && ([ -f "./.env.local" ] && cat "./.env.local" || echo '')) up -d
But it seems that no envs are loaded and plenty of warnings are thrown:
WARN[0000] The "PGADMIN_DEFAULT_EMAIL" variable is not set. Defaulting to a blank string.
WARN[0000] The "PGADMIN_DEFAULT_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "PGADMIN_HOST_PORT" variable is not set. Defaulting to a blank string.
...
Don't know why. Any help would be appreciated.
vim <(cat "./.env" && ([ -f "./.env.local" ] && cat "./.env.local" || echo ''))
vim shows that two env files are concatenated correctly:
PGADMIN_DEFAULT_EMAIL=example#example.com
PGADMIN_DEFAULT_PASSWORD=example
PGADMIN_HOST_PORT=8080
PGADMIN_HOST_PORT=8081
OS: centOS 7
You could try to use a little script like this:
I have created two test files with the image and version.
In the command shown i create a test3 file which combines those two and uses it in the docker-compose as --env-file, afterwards test3 gets deleted again.
Note: test3 does not get deleted if there is a problem starting the docker container.
Related
The Dockerfile contains:
ENV VAR 1
COPY ./setup.exp /tmp/
RUN chmod a+x /tmp/setup.exp
The expect file:
#!/usr/bin/expect
set timeout -1
spawn setup -v
expect "Enter variable: "
send -- "$env(VAR)\r"
The shell file (main.sh):
#!/bin/sh
/tmp/setup.exp $VAR
When I run ./main.sh from the shell inside the container, it works perfectly fine.
However, when I run docker-compose up with entrypoint: ./main.sh, it prints this error:
send: spawn id exp4 not open
while executing
"send -- "$env(VAR)\r""
(file "/tmp/setup.exp" line 5)
If I pass the variable directly as entrypoint: /tmp/setup.exp ${VAR}, it prints this warning:
WARNING: The VAR variable is not set. Defaulting to a blank string.
I also tried set VAR [lindex $argv 0]; and then send -- "$VAR\r" without any success.
Seems like from inside the container the script is able to load docker's env variables.
Any suggestions?
Thanks #glennjackman for pointing that out. I ran expect -d for a more verbose output.
It turns out the reason the program exited is because python was raising this exception:
ValueError: invalid width 0 (must be > 0)
Setting the variable say ENV COLUMNS 100 in the Dockerfile solved the issue for me.
I have a Dockerfile in which files in a directory are downloaded:
RUN wget https://www.classe.cornell.edu/~cesrulib/downloads/tarballs/ -r -l1 --no-parent -A tgz \
--cut=99 -nH -nv --show-progress --progress=bar:force:noscroll
I know that there is exactly one file here of the form "bmad_dist_YYYY_MMDD.tgz" where "YYYY_MMDD" is a date. For example, the file might be named "bmad_dist_2020_0707.tgz". I want to set a bash variable to the file name without the ".tgz" extension. If this was outside of docker I could use:
FULLNAME=$(ls -1 bmad_dist_*.tgz)
BMADDIST="${FULLNAME%.*}"
So I tried in the dockerfile:
ENV FULLNAME $(ls -1 bmad_dist_*.tgz)
ENV BMADDIST "${FULLNAME%.*}"
But this does not work. Is it possible to do what I want?
Shell expansion does not happen in Dockerfile ENV. Then workaround that you can try is to pass the name during Docker build.
Grab the filename during build name and discard the file or you can try --spider for wget to just get the filename.
ARG FULLNAME
ENV FULLNAME=${FULLNAME}
Then pass the full name dynamically during build time.
For example
docker build --build-args FULLNAME=$(wget -nv https://upload.wikimedia.org/wikipedia/commons/5/54/Golden_Gate_Bridge_0002.jpg 2>&1 |cut -d\" -f2) -t my_image .
The ENV ... ... syntax is mainly for plaintext content, docker build arguments, or other environment variables. It does not support a subshell like your example.
It is also not possible to use RUN export ... and have that variable defined in downstream image layers.
The best route may be to write the name to a file in the filesystem and read from that file instead of an environment variable. Or, if an environment variable is crucial, you could set an environment variable from the contents of that file in an ENTRYPOINT script.
I would like to export the environment variable in the Makefile. In this case, is to get the IP for debugging with docker
Makefile
start:
export XDEBUG_REMOTE_HOST=$$(/sbin/ip route|awk '/kernel.*metric/ { print $$9 }') \
; docker-compose up -d
Update from answers:
version: '3.5'
services:
php:
build:
context: .
dockerfile: docker/php/Dockerfile
environment:
- XDEBUG_CONFIG="idekey=docker"
- XDEBUG_REMOTE_HOST=${XDEBUG_REMOTE_HOST}
output:
$ make start
export XDEBUG_REMOTE_HOST=$(/sbin/ip route|awk '/kernel.*metric/ { print $9 }') \
; docker-compose up -d
Starting service_php ... done
$ docker-compose exec php bash
WARNING: The XDEBUG_REMOTE_HOST variable is not set. Defaulting to a blank string.
You need to make sure the variable assignment and the docker command run in the same shell. Trivially, put them in the same rule:
start:
XDEBUG_REMOTE_HOST=$$(/sbin/ip route|awk '/kernel.*metric/ { print $$9 }') \
docker-compose up -d
I took out the # because it's probably simply a bad idea, especially if you need to understand what's going on here. You can use make -s once your Makefile is properly tested if you don't want to see what it's doing.
The purpose of export is to expose a variable to subprocesses, but that's not necessary here. Instead, we use the shell's general
variable=value anothervar=anothervalue command
syntax to set the value of a variable for the duration of a single command.
If the internals of docker-compose require the variable to be exported, then of course, you can do that too:
start:
export XDEBUG_REMOTE_HOST=$$(/sbin/ip route|awk '/kernel.*metric/ { print $$9 }') \
; docker-compose up -d
Notice how the backslash at the end of the first line of the command list joins the two commands on a single logical line, so they get passed to the same shell instance, and the ; command separator is required to terminate the first command. (I put the semicolon at beginning of line as an ugly reminder to the reader that this is all one command line.)
Specifically for docker-compose, the customary way to set a variable from the command line is with a specific named option;
start:
docker-compose up -e XDEBUG_REMOTE_HOST=$$(/sbin/ip route|awk '/kernel.*metric/ { print $$9 }') -d
There are other ways to solve this such as the GNU Make .ONESHELL directive but this is simple and straightforward, and portable to any Make.
If you assume that the route exists when make is first invoked, you can assign a make variable as opposed to a shell variable as follows:
export XDRH_MAKE_VAR:=$(shell /sbin/ip route|awk '/kernel.*metric/ { print $$9 }')
start:
#echo XDHR_MAKE_VAR=$(XDRH_MAKE_VAR)
XDEBUG_REMOTE_HOST=$(XDRH_MAKE_VAR) docker-compose up -d
XDRH_FILE:
echo $(XDRH_MAKE_VAR) > $#
someother_target:
XDEBUG_REMOTE_HOST=$(XDRH_MAKE_VAR) some_other_command
command_that_uses_it_as_param $(XDRH_MAKE_VAR)
NOTE_does_not_work:
XDEBUG_REMOTE_HOST=$(XDRH_MAKE_VAR) echo $$XDEBUG_REMOTE_HOST
The last one does not work, because the bash shell will expand $XDEBUG_REMOTE_HOST before assigning it (See here). Also, the variable is set at make parse time, so if any of your rules effect the route, then this will not be reflected in its value.
If you want to access the value in the shell afterwards, you would want to do something like:
bash> make start XDRH_FILE
bash> XDEBUG_REMOTE_HOST=`cat XDRH_FILE`
bash> docker-compose exec php bash
I need to fill a variable in dockerfile with the result of a command
Like in bash var=$(date)
EDIT 1
date is a example.
in my case i use FROM phusion/baseimage:0.9.17 so i want at each building use the last version so i use this
curl -v --silent api.github.com/repos/phusion/baseimage-docker/tags 2>&1 | grep -oh 'rel-.*",' | head -1 | sed 's/",//' | sed 's/rel-//' ==> 0.9.17.
but i don't know how i parse it in var with dockerfile for this result
ENV verbaseimage=curl...
FROM phusion/baseimage:$verbaseimage
RESULT
In my use case
FROM phusion/baseimage:latest
But the question remains unresolved for other case
I had same issue and found way to set environment variable as result of function by using RUN command in dockerfile.
For example i need to set SECRET_KEY_BASE for Rails app just once without changing as would when i run:
docker run -e SECRET_KEY_BASE="$(openssl rand -hex 64)"
Instead it i write to Dockerfile string like:
RUN bash -l -c 'echo export SECRET_KEY_BASE="$(openssl rand -hex 64)" >> /etc/bash.bashrc'
and my env variable available from root, even after bash login.
or may be
RUN /bin/bash -l -c 'echo export SECRET_KEY_BASE="$(openssl rand -hex 64)" > /etc/profile.d/docker_init.sh'
then it variable available in CMD and ENTRYPOINT commands
Docker cache it as layer and change only if you change some strings before it.
You also can try different ways to set environment variable.
The old workaround is mentioned here (issue 2637: Feature request: expand Dockerfile ENV $VARIABLES in WORKDIR):
One work around that I've used, is to have a file in my context called "build-env". What I do is source it and run my desired command in the same RUN step. So for example:
build-env:
VERSION=stable
Dockerfile:
FROM radial/axle-base:latest
ADD build-env /build-env
RUN source build-env && mkdir /$VERSION
RUN ls /
But for date, that might not be as precise as you want.
Other workarounds are in issue 2022 "Dockerfile with variable interpolation".
In docker 1.9 (end of October 2015), you will have "support for build-time environment variables to the 'build' API (PR 9176)" and "Support for passing build-time variables in build context (PR 15182)".
docker build --build-arg=[]: Set build-time variables
You can use ENV instructions in a Dockerfile to define variable values. These values persist in the built image. However, often persistence is not what you want. Users want to specify variables differently depending on which host they build an image on.
A good example is http_proxy or source versions for pulling intermediate files. The ARG instruction lets Dockerfile authors define values that users can set at build-time using the ---build-arg flag:
$ docker build --build-arg HTTP_PROXY=http://10.20.30.2:1234 .
This flag allows you to pass the build-time variables that are accessed like regular environment variables in the RUN instruction of the Dockerfile.
Also, these values don't persist in the intermediate or final images like ENV values do.
so I want at each building use the last version so I use this
curl -v --silent api.github.com/repos/phusion/baseimage-docker/tags 2>&1 | grep -oh 'rel-.*",' | head -1 | sed 's/",//' | sed 's/rel-//' ==> 0.9.17.
If you want to use the last version of that image, all you need to do is use the tag 'latest' with the FROM directive:
FROM phusion/baseimage:latest
See also "The misunderstood Docker tag: latest": it doesn't always reference the actual latest build, but in this instance, it should work.
If you really want to use the curl|parse option, use it to generate a Dockerfile with the right value (as in a template processed to generate the right file).
Don't try to use it directly in the Dockerfile.
I wanted to set an ENV or LABEL variable from a computation in the Dockerfile, e.g. to make some computed installation options visible in docker inspect.
There does not seem to be any way to do that, and this issue suggests that it's a security design choice.
A Dockerfile can set an ENV variable to $X, ${X:-default}, or ${X:+substitute} where that $X must be another ENV or ARG variable.
A single RUN command can set and use shell variables, but that goes away at the end of the RUN command when that container layer shuts down.
A RUN command can write computed data into files, but the Dockerfile still can't get that data into an ENV or LABEL even if the file is ~/.bashrc. (File contents can, of course, be used by code running in the Container.)
The build can at least RUN echo $X to record choices to the build log -- unless that step comes from the build cache, in which case the RUN step doesn't run.
Please do correct me if there's a way out.
Partially connected to question. If one wants to use the result of some command later on it is possible within single RUN statement as follows:
RUN CUR_DIR=`pwd` && \
echo $CUR_DIR
I want to set $PS1 environment variable to the container. It helps me to identify multilevel or complex docker environment setup. Currently docker container prompts with:
root#container-id#
If I can change it as following , I can identify the container by looking at the $PS1 prompt itself.
[Level-1]root#container-id#
I did experiments by exporting $PS1 by making my own image (Dockerfile), .profile file etc. But it's not reflecting.
I had the same problem but in docker-compose context.
Here is how I managed to make it work:
# docker-compose.yml
version: '3'
services:
my_service:
image: my/image
environment:
- "PS1=$$(whoami):$$(pwd) $$ "
Just pass PS1 value as an environment variable in docker-compose.yml configuration file.
Notice how dollars signs need to be escaped to prevent docker-compose from interpolating values (documentation).
This Dockerfile sets PS1 by doing:
RUN echo 'export PS1="[\u#docker] \W # "' >> /root/.bash_profile
We use a similar technique for tracking inputs and outputs in complex container builds.
https://github.com/ianmiell/shutit/blob/master/shutit_global.py#L1338
This line represents the product of hard-won experience dealing with docker/(p)expect combinations:
"SHUTIT_BACKUP_PS1_%s=$PS1 && PS1='%s' && unset PROMPT_COMMAND"
Backing up the prompt is handy if you want to revert, setting the PS1 with PS1= sets the PS1, and unsetting the PROMPT_COMMAND removes any nasty surprises with the terminal being reset etc.. for the expect.
If the question is about how to ensure it's set when you run the container up (as opposed to building), then you may need to add something to your .bashrc / .profile files depending on how you run up your container. As far as I know there's no way to ensure it with a dockerfile directive and make it persist.
I normally create /home/USER/.bashrc or /root/.bashrc, depending on who the USER of the Dockerfile is. That works well. I've tried
ENV PS1 '# '
but that never worked for me.
Here's a way to set the PS1 when you run the container:
docker run -it \
python:latest \
bash -c "echo \"export PS1='[python:latest] \w$ '\" >> ~/.bashrc && bash"
I made a little wrapper script, to be able to run any image with my custom prompt:
#!/usr/bin/env bash
# ~/bin/docker-run
set -eu
image=$1
docker run -it \
-v $(pwd):/opt/app
-w /opt/app ${image} \
bash -c "echo \"export PS1='[${image}] \w$ '\" >> ~/.bashrc && bash"
In debian 9, for running bash, this worked:
RUN echo 'export PS1="[\$ENV_VAR] \W # "' >> /root/.bashrc
It's generally running as root and I generally know I am in docker, so I wanted to have a prompt that indicated what the container was, so I used an environment variable. And I guess the bash I use loads .bashrc preferentially.
Try setting environment variables using docker options
Example:
docker run \
-ti \
--rm \
--name ansibleserver-debug \
-w /githome/axel-ansible/ \
-v /home/lordjea/githome/:/githome/ \
-e "PS1=DEBUG$(pwd)# " \
lordjea/priv:311 bash
docker --help
Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
Run a command in a new container
Options:
...
-e, --env list Set environment variables
...
You should set that in .profile, not .bashrc.
Just open .profile from your root or home and replace PS1='\u#\h:\w\$ ' with PS1='\e[33;1m\u#\h: \e[31m\W\e[0m\$ ' or whatever you want.
Note that you need to restart your container.
On my MAC I have an alias named lxsh that will start a bash shell using the ubuntu image in my current directory (details). To make the shell's prompt change, I mounted a host file onto /root/.bash_aliases. It's a dirty hack, but it works. The full alias:
alias lxsh='echo "export PS1=\"lxsh[\[$(tput bold)\]\t\[$(tput sgr0)\]\w]\\$\[$(tput sgr0)\] \"" > $TMPDIR/a5ad217e-0f2b-471e-a9f0-a49c4ae73668 && docker run --rm --name lxsh -v $TMPDIR/a5ad217e-0f2b-471e-a9f0-a49c4ae73668:/root/.bash_aliases -v $PWD:$PWD -w $PWD -it ubuntu'
The below solution assumes that you've used Dockerfile USER to set a non-root Linux user for Bash.
What you might have tried without success:
ENV PS1='[docker]$' ## may not work
Using ENV to set PS1 can fail because the value can be overridden by default settings in a preexisting .bashrc when an interactive shell is started. Some Linux distributions are opinionated about PS1 and set it in an initial .bashrc for each user (Ubuntu does this, for example).
The fix is to modify the Dockerfile to set the desired value at the end of the user's .bashrc -- overriding any earlier settings in the script.
FROM ubuntu:20.04
# ...
USER myuser ## the username
RUN echo "PS1='\n[ \u#docker \w ]\n$ '" >>.bashrc