As part of a Dockerfile, I am attempting to modify a text file (ssh_config) to contain a variable passed through by the user (as an ARG) to the docker container at build time.
In my Dockerfile I have (this is not the entire file):
ARG key_name
RUN echo 'Host geoserver\n\
User user\n\
HostName 38.191.191.111\n\
IdentityFile /root/$key_name' >> /etc/ssh/ssh_config
This collects the argument key_name, and then appends some text to the ssh_config text file.
This is run as follows:
docker build --build-arg key_name=GC -t pyramid .
When I check to see what has been written, the key_name variable hasn't been parsed, and instead has been written as text (so literally as $key_name). Obviously I want it to be replaced with the variable passed through ARG.
I have tried using ${key_file} instead of just $key_file, I just get the same text in the text file but with curly braces included.
So my question is, how can I use the ARG variable correctly within the RUN echo statement?
First: Make sure, your ARG comes after your FROM. See: https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
Second: As you can see here, variables won't be interpretad inside '', so use "" instead.
When you surround the variable with single quotes it doesn't get replaced.
If you need the single qoutes in the file just surround everything with double quotes, otherwise just remove the single quotes all together.
ARG key_name
RUN echo "'Host geoserver\n\
User user\n\
HostName 38.191.191.111\n\
IdentityFile /root/$key_name'" >> /etc/ssh/ssh_config
Related
I am using a base.env as an env_file for several of my docker services.In this base.env I have several parts of the environment variable that repeat throughout the file. For example, port and ip are the same for three different environment variables.
I would like to specify these in an environment variable and reuse those variables to fill out the other environment variables.
Here is base.env:
### Kafka
# kafka's port is 9092 by default in the docker-compose file
KAFKA_PORT_NUMBER=9092
KAFKA_TOPIC=some-topic
KAFKA_IP=kafka
KAFKA_CONN: //$KAFKA_IP:$KAFKA_PORT_NUMBER/$KAFKA_TOPIC
# kafka topic that is to be created. Note that ':1:3' should remain the same.
KAFKA_CREATE_TOPICS=$KAFKA_TOPIC:1:3
# the url for connecting to kafka
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://$KAFKA_IP:$KAFKA_PORT_NUMBER
I have tried writing
KAFKA_CONN: //$${KAFKA_IP}:$${KAFKA_PORT_NUMBER}/$${KAFKA_TOPIC}
in the environment section of the appropriate service in the docker-compose.yml, but this gets interpreted as a literal string in the container.
Is there a way to do what I want in the base.env file?
Thank you for your help!
You can actually do it like this (at least in vlucas/dotenv package (php), not sure about others, please check it yourself)
MAIL_NAME=${MAIL_FROM}
Read more about it here
There is no way to do this in an env_file since it is not run as a bash command. This means that the variable is not created and then concatenated into the next variable it appears in. The values are just read in as they are in the env_file.
I used $ in Node.js and React.js , and both worked
POSTGRES_PORT=5432
DATABASE_URL="postgresql://root#localhost:${POSTGRES_PORT}/dbname"
and in react
REACT_APP_DOMAIN=domain.com
#API Configurations
REACT_APP_API_DOMAIN=$REACT_APP_DOMAIN
I know that I am a little late to the party, but I had the same question and I found a way to do it. There is a package called env-cmd, which allows you to use a .js file as an .env file. The file simply needs to export an object with the keys being your environment variable names and the values being, well, the values. This now allows you to run javascript before the environment variables are exported and thus use environment variables to set others.
I temporarly managed to deal with this where I create a script to replace env file vars from another env file vars like so:
.env.baseurl:
BASEURL1=http://127.0.0.1
BASEURL2=http://192.168.1.10
.env.uris.default:
URI1=${BASEURL1}/uri1
URI2=${BASEURL2}/uri2
URI3=${BASEURL2}/uri3
convert-env.sh:
#!/bin/bash
# To allow using sed correctly from same file multiple times
cp ./.env.uris.default ./.env.uris
# Go through each variable in .env.baseurl and store them as key value
for VAR in $(cat ./.env.baseurl); do
key=$(echo $VAR | cut -d "=" -f1)
value=$(echo $VAR | cut -d "=" -f2)
# Replace env vars by values in ./.env.uris
sed -i "s/\${$key}/$value/g" ./.env.uris
done
then you can run docker run command to start the container and load it with your env vars (from .env.baseurl to .env.uris) :
docker run -d --env-file "./.env.uris" <image>
This is not the best solution but helped me for now.
Using Nextjs, in the .env.local file I have the following variables:
NEXT_PUBLIC_BASE_URL = http://localhost:5000
NEXT_PUBLIC_API_USERS_URL_REGISTER = ${NEXT_PUBLIC_BASE_URL}/api/users/register
it works well, I used the variable NEXT_PUBLIC_BASE_URL in the variable NEXT_PUBLIC_API_USERS_URL_REGISTER.
There is a simple way to do this you will just need to run:
env >>/root/.bashrc && source /root/.bashrc
this will append all environment variables inside /root/.bashrc then convert those if they have not been converted while passing the env-file
you can use something like this ${yourVar}
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${KAFKA_IP}:${$KAFKA_PORT_NUMBER}
I test this on PHP / Laravel .env it's working fine
For our GitLab runner we have some variables saved on GitLab. One of them is a base64 encoded USER_DB_PASSWORD_ENCODED variable.
I pass the variable to the Docker build command used for our tests and can access it like this in the Dockerfile:
ARG USER_DB_PASSWORD_ENCODED
ENV USER_DB_PASSWORD_ENCODED=${USER_DB_PASSWORD_ENCODED}
From here my app could access this ENV variable USER_DB_PASSWORD_ENCODED, but I need to decode it to be able to use it in the app. For this purpose I tried this sequence:
RUN echo "$USER_DB_PASSWORD_ENCODED" | base64 --decode > /temp
RUN USER_DB_PASSWORD_ENCODED=$(cat /temp); echo "Output: $USER_DB_PASSWORD_ENCODED"
ENV USER_DB_PASSWORD=$USER_DB_PASSWORD_ENCODED
RUN echo $USER_DB_PASSWORD
I decode the encoded variable into a /temp dir, try to assign that value to the existing variable, and try to assign that existing variable to a new variable, with the name that is actually used in the app.
The decoding works, the output echo shows me the decoded value correctly, but when I echo the new variable, it still shows me the decoded value.
How can I properly deal with an encoded variable and overwrite an existing/create a new ENV variable in a Dockerfile?
An alternative idea was to not define a separate ENV variable, but decode the value directly into a .env file in the directory where it is needed, e.g.
RUN echo "$USER_DB_PASSWORD_ENCODED" | base64 --decode > /api/.env
But then I have the problem of only getting the decoded value into the file, while I also need to prepend the value with USER_DB_PASSWORD_ENCODED for it to be recognized by the app
In the end, I went with my alternative approach:
RUN echo -n "USER_DB_PASSWORD=" > api/.env
RUN echo -n "$USER_DB_PASSWORD_ENCODED" | base64 --decode >> api/.env
The first command creates a .env where it can be read from the application and writes the string USER_DB_PASSWORD= to it (without newline).
The second command appends the decoded value of the encoded and masked GitLab variable to the existing line in the existing .env file.
This way, the cleartext value is never visible in the job log and the container is destroyed after a few minutes of running the tests.
I'm trying to create a script that starts a docker container and mounts a local folder that contains spaces in the name. I can get it to work fine when I run a *.bat file with the docker run command:
docker run -p 8081:8081 -v "C:\Test Folder With Blanks":/zzz myimage jupyter lab --notebook-dir=/zzz--ip=0.0.0.0 --port=8081 --allow-root
But when I try to do the same in a Powershell script file, I get an error:
$CMD = 'docker run -p 8081:8081 -v "C:\Test Folder With Blanks":/zzz myimage jupyter lab --notebook-dir=/zzz--ip=0.0.0.0 --port=8081 --allow-root'
Invoke-Expression $CMD
docker: invalid reference format.
See 'docker run --help'.
I'm on Win10 and running Powershell in Visual Studio Code IDE.
Thanks for ideas.
First, the obligatory warning: Unless you have a command line stored as a single string somewhere and you either fully control or trust the content of the string, Invoke-Expression should generally be avoided.
You're seeing an inconsistency in how PowerShell treats compound tokens composed of directly concatenated quoted and unquoted parts.
Specifically, argument "C:\Test Folder With Blanks":/zzz is unexpectedly broken in two, i.e passed as two arguments.
The workaround is to quote the entire argument, i.e.
"C:\Test Folder With Blanks:/zzz"
Note: I'm assuming that docker doesn't actually require partial quoting in its arguments, which it shouldn't; however, there are high-profile CLIs on Windows that do, notably msiexec.
Alternatively, use an expression enclosed in (...) to compose your string; e.g.
("C:\Test Folder With Blanks" + ':/zzz')
There's no good reason to do so in this case, but it could be helpful if you need string interpolation in one part of your string ("..."), but not in another ('...').
General caveats:
Compared to cmd.exe and also POSIX-compatible shells such as bash, PowerShell has several additional metacharacters, notably # (at the start of a token), { / }, and ;. Therefore, you cannot always expect command lines written for these shells to work as-is in PowerShell.
As of PowerShell 7.2.2, passing arguments to external programs (such as docker) is fundamentally broken with respect to arguments that have embedded " characters and empty-string arguments - see this answer.
The general pattern of the inconsistency, as of PowerShell 7.2.2, is as follows:
If an argument:
starts with a quoted token - whether single- ('...') or double-quoted ("...") -
and has additional characters directly following the closing quote,
the first such character starts a separate argument.
E.g. "foo":bar / "foo"=bar / "foo"'bar' are passed as separate arguments foo and :bar / foo and =bar / foo and bar, respectively.
In other words:
You cannot compose a single string argument from a mix of quoted and unquoted / differently quoted tokens if the first token is quoted.
Conversely, it does work if the first token is unquoted, including an unquoted simple variable reference such as $HOME.
# OK: First token is unquoted.
PS> cmd /c echo foo"bar"'baz'last
foobarbazlast
# !! BROKEN: First token is quoted.
# !! Because each successive token is quoted too,
# !! each becomes its own argument.
PS> cmd /c echo 'foo'"bar"'baz'last
foo bar baz last
GitHub issue #6467 discusses this inconsistency; however, it has been closed, because the behavior is - surprisingly - considered by design.
This does not happen if the first token is unquoted; however, there are related bugs that start with unquoted tokens that similarly break arguments in two, related to their looking like named arguments to PowerShell (which is a PowerShell concept that doesn't apply when calling external programs):
GitHub issue #11646: an argument such as -foo=1,2 breaks parsing.
GitHub issue #6291: an argument such as -foo=bar.baz is broken in two at the (first) .
Docker env files look similar to shell scripts defining variables.
Unfortunately in env files, values cannot be quoted and so simply sourcing them only works if there are no "special" characters in the values.
VAR1=This_works
VAR2=This will not work
VAR3=This won't either
Is there any way you can use these files in a shell script?
My current approach is this:
eval $( perl -ne '
s/\x27/\x27\\\x27\x27/g;
s/^(\w+)=(.+)$/export $1=\x27$2\x27/ and print
' "path/to/env_file" )
So I'm searching for any quote in each line of the env file and replace it by '\''.
Then I'm modifying each line which starts with an identifier (\w+) followed by a = and any text (.+). So the VAR3 will become: export VAR3='This won'\''t either.
The modified line is then printed.
Everything which was printed is eval-ed and so the variables will be available in my shell's environment.
Are there other proposals how to achieve this?
I want to compose two environment variables: first define a "root" and in the same line use that to create a composed one. In example, filename and append extension.
Doing this container,
FROM centos:7
ENV ROOT_VAR=stringy ROOT_VAR_TGZ=${ROOT_VAR}.tar.gz
RUN echo ${ROOT_VAR} $ ${ROOT_VAR_TGZ}
The output for echo is
stringy $ .tar.gz
But when splitting each variable in an individual ENV command is composed correctly.
Is this the expected behaviour?
The behaviour is clearly explained in the docker reference document:
Environment variable substitution will use the same value for each variable throughout the entire instruction. In other words, in this example:
ENV abc=hello
ENV abc=bye def=$abc
ENV ghi=$abc
will result in def having a value of hello, not bye. However, ghi will have a value of bye because it is not part of the same instruction that set abc to bye.