I have a YAML scalar that is throwing the following error when I try to evaluate my docker-compose.yml file:
ERROR: Invalid interpolation format for "environment" option in
service "time_service": "${Time.now}"
YAML:
---
version: '2'
services:
time_service:
build: "."
environment:
TIME: "${Time.now}"
How can I maintain the same string output as written, but avoid having the docker-compose interpret it as faulty string interpolation?
You can use a $$ (double-dollar sign) when your configuration needs
a literal dollar sign.
You are hitting the docker-compose variable substitution, which is well documented here:
Both $VARIABLE and ${VARIABLE} syntax are supported. Extended shell-style features, such as ${VARIABLE-default} and ${VARIABLE/foo/bar}, are not supported.
You can use a $$ (double-dollar sign) when your configuration needs a literal dollar sign. This also prevents Compose from interpolating a value, so a $$ allows you to refer to environment variables that you don’t want processed by Compose.
docker-compose is written in Python, as you see on github, the doubling mechanism to get the original meaning of special characters can be found in many programs, I needed to use this myself, while programming, as far back in 1984.
Found the answer by copying the suggestion for % characters in this post
It requires a double dollar sign $$.
So I needed "$${Time.now}", which evaluates to "${Time.now}"
Related
Let's say I need to change confluent.security.event.logger.exporter.kafka.topic.replicas or confluent.metrics.reporter.topic.replicas.
How can I set them in my docker-compose file via ENVs?
It's described here https://docs.confluent.io/platform/current/installation/docker/config-reference.html#confluent-enterprise-ak-configuration
For the Enterprise Kafka (cp-server) image, convert the kafka.properties file variables as below and use them as environment variables:
Prefix with KAFKA_ for Apache Kafka.
Prefix with CONFLUENT_ for Confluent components.
Convert to upper-case.
Replace a period (.) with a single underscore (_).
Replace a dash (-) with double underscores (__).
Replace an underscore (_) with triple underscores (___).
It seems like I have to use
KAFKA_CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
and
KAFKA_CONFLUENT_SECURITY_EVENT_LOGGER_EXPORTER_KAFKA_TOPIC_REPLICAS: 1
I'm trying to create a script that starts a docker container and mounts a local folder that contains spaces in the name. I can get it to work fine when I run a *.bat file with the docker run command:
docker run -p 8081:8081 -v "C:\Test Folder With Blanks":/zzz myimage jupyter lab --notebook-dir=/zzz--ip=0.0.0.0 --port=8081 --allow-root
But when I try to do the same in a Powershell script file, I get an error:
$CMD = 'docker run -p 8081:8081 -v "C:\Test Folder With Blanks":/zzz myimage jupyter lab --notebook-dir=/zzz--ip=0.0.0.0 --port=8081 --allow-root'
Invoke-Expression $CMD
docker: invalid reference format.
See 'docker run --help'.
I'm on Win10 and running Powershell in Visual Studio Code IDE.
Thanks for ideas.
First, the obligatory warning: Unless you have a command line stored as a single string somewhere and you either fully control or trust the content of the string, Invoke-Expression should generally be avoided.
You're seeing an inconsistency in how PowerShell treats compound tokens composed of directly concatenated quoted and unquoted parts.
Specifically, argument "C:\Test Folder With Blanks":/zzz is unexpectedly broken in two, i.e passed as two arguments.
The workaround is to quote the entire argument, i.e.
"C:\Test Folder With Blanks:/zzz"
Note: I'm assuming that docker doesn't actually require partial quoting in its arguments, which it shouldn't; however, there are high-profile CLIs on Windows that do, notably msiexec.
Alternatively, use an expression enclosed in (...) to compose your string; e.g.
("C:\Test Folder With Blanks" + ':/zzz')
There's no good reason to do so in this case, but it could be helpful if you need string interpolation in one part of your string ("..."), but not in another ('...').
General caveats:
Compared to cmd.exe and also POSIX-compatible shells such as bash, PowerShell has several additional metacharacters, notably # (at the start of a token), { / }, and ;. Therefore, you cannot always expect command lines written for these shells to work as-is in PowerShell.
As of PowerShell 7.2.2, passing arguments to external programs (such as docker) is fundamentally broken with respect to arguments that have embedded " characters and empty-string arguments - see this answer.
The general pattern of the inconsistency, as of PowerShell 7.2.2, is as follows:
If an argument:
starts with a quoted token - whether single- ('...') or double-quoted ("...") -
and has additional characters directly following the closing quote,
the first such character starts a separate argument.
E.g. "foo":bar / "foo"=bar / "foo"'bar' are passed as separate arguments foo and :bar / foo and =bar / foo and bar, respectively.
In other words:
You cannot compose a single string argument from a mix of quoted and unquoted / differently quoted tokens if the first token is quoted.
Conversely, it does work if the first token is unquoted, including an unquoted simple variable reference such as $HOME.
# OK: First token is unquoted.
PS> cmd /c echo foo"bar"'baz'last
foobarbazlast
# !! BROKEN: First token is quoted.
# !! Because each successive token is quoted too,
# !! each becomes its own argument.
PS> cmd /c echo 'foo'"bar"'baz'last
foo bar baz last
GitHub issue #6467 discusses this inconsistency; however, it has been closed, because the behavior is - surprisingly - considered by design.
This does not happen if the first token is unquoted; however, there are related bugs that start with unquoted tokens that similarly break arguments in two, related to their looking like named arguments to PowerShell (which is a PowerShell concept that doesn't apply when calling external programs):
GitHub issue #11646: an argument such as -foo=1,2 breaks parsing.
GitHub issue #6291: an argument such as -foo=bar.baz is broken in two at the (first) .
How can I use variable substitution for a list, map, or array value in a docker-compose.yml file.
For example:
graylog:
image: graylog2/server2
extra_hosts: ${EXTRA_HOSTS}
and
export EXTRA_HOSTS="['host1:10.10.10.1','host2:10.10.10.2']"
gives the following error: graylog.extra_hosts must be a mapping
I've tried different variations of the above with no luck.
I do see that there's an open issue about this here: https://github.com/docker/compose/issues/4249
Is it just not possible? Does anyone know of a workaround?
At least as of this time (June 2018), Docker still doesn't support this. I was able to work around the issue utilizing envsubst.
envsubst is part of gettext and it can be used to replace only environment variables you tell it to.
Tweak the docker-compose.yml value to look like an array or map (either brackets or curly braces) but have the value be an environment variable.
For example
graylog:
image: graylog2/server2
extra_hosts: [ ${EXTRA_HOSTS} ]
Then, define your environment variable without brackets or curly braces.
For example:
export EXTRA_HOSTS="'host1:10.10.10.1','host2:10.10.10.2'"
Then utilize envsubst
envsubst '${EXTRA_HOSTS}' < docker-compose.yml > docker-compose.subst.yml && docker stack deploy -c docker-compose.subst.yaml foobar
Notice that you pass '${EXTRA_HOSTS}' to envsubst. This tells it to only replace this environment variable. This ensures it doesn't accidentally replace some other variable that's utilizing the variable substitution syntax of Docker compose files.
I read this link which explains the valid directives. And it says "Parser directive is a special form of comment"
I really don't understand what is the need for this and if it is comment, why docker parses it?
I didn't find any good explanations related to this.
Is it only escape directive? Because of windows path issues.
The "parser directives" instruct the Dockerfile parser to handle the content of Dockerfile as specified in the directives.
"Parser directives" direct the "way" how the content of DockerFile should be treated/interpreted. For this reason, its must be at the top of a Dockerfile.
Currently, "escape" is the only supported directive.
This can be useful for you: https://www.safaribooksonline.com/library/view/learning-docker-/9781786462923/b203797f-2c5f-4f24-a189-9e1780adfbbb.xhtml
It basically tell the docker daemon how to read the dockerfile, the way it intended to. For an example, I'll use the escape parser as it's the only supported one right now.
COPY sample.txt c:\\
This will fail as the second '\' at the end of the line interpreted as an escape for the newline, instead of a target of the escape from the first '\'
#escape=`
By adding an escape parser, we can tackle this. However, you have to be careful when using parsers as it's very easy to get them invalidated. You can see the common mistakes stated in Docker docs
Parser directive affect the way in which subsequent lines in a Dockerfile are handled / executed. Parser directive must be added at the top of Dockerfile.
For example, \ (backslash) is path separator in windows, whereas in Docker file it is used as escape character. So if we want to change the escape character in Docker files then we can use parser directive
# escape=`
FROM microsoft/nanoserver
COPY testfile.txt c:\
RUN dir c:\
In the above example backtick will be act as escape character instead of backslash.
Given the following configuration:
mysql:
environment:
MY_MYSQL_PORT:
image: mysql
ports:
- "${MY_MYSQL_PORT}:3306"
There's a way to provide a fallback value for MY_MYSQL_PORT without relying on wrapper scripts? I already tested like bash ${MY_MYSQL_PORT-3306} but it doesn't work.
They implemented that feature with compose 1.9 release:
Added support for shell-style inline defaults in variable interpolation.
The supported forms are ${FOO-default} (fall back if FOO is unset) and ${FOO:-default} (fall back if FOO is unset or empty).
Release Notes Docker Compose 1.9
An explanation in the Dockerfile documentation can be found at:
https://docs.docker.com/engine/reference/builder/#environment-replacement
To quote from there:
The ${variable_name} syntax also supports a few of the standard bash modifiers as specified below:
${variable:-word} indicates that if variable is set then the result will be that value. If variable is not set then word will be the result.
${variable:+word} indicates that if variable is set then word will be the result, otherwise the result is the empty string.
In all cases, word can be any string, including additional environment variables.
Escaping is possible by adding a \ before the variable: \$foo or \${foo}, for example, will translate to $foo and ${foo} literals respectively.