Let's say I need to change confluent.security.event.logger.exporter.kafka.topic.replicas or confluent.metrics.reporter.topic.replicas.
How can I set them in my docker-compose file via ENVs?
It's described here https://docs.confluent.io/platform/current/installation/docker/config-reference.html#confluent-enterprise-ak-configuration
For the Enterprise Kafka (cp-server) image, convert the kafka.properties file variables as below and use them as environment variables:
Prefix with KAFKA_ for Apache Kafka.
Prefix with CONFLUENT_ for Confluent components.
Convert to upper-case.
Replace a period (.) with a single underscore (_).
Replace a dash (-) with double underscores (__).
Replace an underscore (_) with triple underscores (___).
It seems like I have to use
KAFKA_CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
and
KAFKA_CONFLUENT_SECURITY_EVENT_LOGGER_EXPORTER_KAFKA_TOPIC_REPLICAS: 1
Related
I have a YAML scalar that is throwing the following error when I try to evaluate my docker-compose.yml file:
ERROR: Invalid interpolation format for "environment" option in
service "time_service": "${Time.now}"
YAML:
---
version: '2'
services:
time_service:
build: "."
environment:
TIME: "${Time.now}"
How can I maintain the same string output as written, but avoid having the docker-compose interpret it as faulty string interpolation?
You can use a $$ (double-dollar sign) when your configuration needs
a literal dollar sign.
You are hitting the docker-compose variable substitution, which is well documented here:
Both $VARIABLE and ${VARIABLE} syntax are supported. Extended shell-style features, such as ${VARIABLE-default} and ${VARIABLE/foo/bar}, are not supported.
You can use a $$ (double-dollar sign) when your configuration needs a literal dollar sign. This also prevents Compose from interpolating a value, so a $$ allows you to refer to environment variables that you don’t want processed by Compose.
docker-compose is written in Python, as you see on github, the doubling mechanism to get the original meaning of special characters can be found in many programs, I needed to use this myself, while programming, as far back in 1984.
Found the answer by copying the suggestion for % characters in this post
It requires a double dollar sign $$.
So I needed "$${Time.now}", which evaluates to "${Time.now}"
I was running the following command:
docker run -e 'SOME_ENV_VARIABLE=somevalue' somecommand
And this did not pass the env variable to somecommand (Note: SOME_ENV_VARIABLE and somevalue did not have un-escaped single quotes, they were capital snake case and lowercase respectively without special character.
Later, I tried:
docker run -e "SOME_ENV_VARIABLE=somevalue" somecommand
And this somehow worked (note the double quotes).
Why is this? I have seen documentation with single quotes.
I tried this on Windows.
Quotation is handled by your shell - since you are on windows, that is cmd.exe. And, from what I understand, since cmd.exe does not handle single quotes, it will pass that argument with the quotes to docker.
It would work on Linux though (which usually has the bash shell), being the the environment many people would use docker on, and as such the reason it is present in a lot of documentation.
How can I use variable substitution for a list, map, or array value in a docker-compose.yml file.
For example:
graylog:
image: graylog2/server2
extra_hosts: ${EXTRA_HOSTS}
and
export EXTRA_HOSTS="['host1:10.10.10.1','host2:10.10.10.2']"
gives the following error: graylog.extra_hosts must be a mapping
I've tried different variations of the above with no luck.
I do see that there's an open issue about this here: https://github.com/docker/compose/issues/4249
Is it just not possible? Does anyone know of a workaround?
At least as of this time (June 2018), Docker still doesn't support this. I was able to work around the issue utilizing envsubst.
envsubst is part of gettext and it can be used to replace only environment variables you tell it to.
Tweak the docker-compose.yml value to look like an array or map (either brackets or curly braces) but have the value be an environment variable.
For example
graylog:
image: graylog2/server2
extra_hosts: [ ${EXTRA_HOSTS} ]
Then, define your environment variable without brackets or curly braces.
For example:
export EXTRA_HOSTS="'host1:10.10.10.1','host2:10.10.10.2'"
Then utilize envsubst
envsubst '${EXTRA_HOSTS}' < docker-compose.yml > docker-compose.subst.yml && docker stack deploy -c docker-compose.subst.yaml foobar
Notice that you pass '${EXTRA_HOSTS}' to envsubst. This tells it to only replace this environment variable. This ensures it doesn't accidentally replace some other variable that's utilizing the variable substitution syntax of Docker compose files.
I read this link which explains the valid directives. And it says "Parser directive is a special form of comment"
I really don't understand what is the need for this and if it is comment, why docker parses it?
I didn't find any good explanations related to this.
Is it only escape directive? Because of windows path issues.
The "parser directives" instruct the Dockerfile parser to handle the content of Dockerfile as specified in the directives.
"Parser directives" direct the "way" how the content of DockerFile should be treated/interpreted. For this reason, its must be at the top of a Dockerfile.
Currently, "escape" is the only supported directive.
This can be useful for you: https://www.safaribooksonline.com/library/view/learning-docker-/9781786462923/b203797f-2c5f-4f24-a189-9e1780adfbbb.xhtml
It basically tell the docker daemon how to read the dockerfile, the way it intended to. For an example, I'll use the escape parser as it's the only supported one right now.
COPY sample.txt c:\\
This will fail as the second '\' at the end of the line interpreted as an escape for the newline, instead of a target of the escape from the first '\'
#escape=`
By adding an escape parser, we can tackle this. However, you have to be careful when using parsers as it's very easy to get them invalidated. You can see the common mistakes stated in Docker docs
Parser directive affect the way in which subsequent lines in a Dockerfile are handled / executed. Parser directive must be added at the top of Dockerfile.
For example, \ (backslash) is path separator in windows, whereas in Docker file it is used as escape character. So if we want to change the escape character in Docker files then we can use parser directive
# escape=`
FROM microsoft/nanoserver
COPY testfile.txt c:\
RUN dir c:\
In the above example backtick will be act as escape character instead of backslash.
Given the following configuration:
mysql:
environment:
MY_MYSQL_PORT:
image: mysql
ports:
- "${MY_MYSQL_PORT}:3306"
There's a way to provide a fallback value for MY_MYSQL_PORT without relying on wrapper scripts? I already tested like bash ${MY_MYSQL_PORT-3306} but it doesn't work.
They implemented that feature with compose 1.9 release:
Added support for shell-style inline defaults in variable interpolation.
The supported forms are ${FOO-default} (fall back if FOO is unset) and ${FOO:-default} (fall back if FOO is unset or empty).
Release Notes Docker Compose 1.9
An explanation in the Dockerfile documentation can be found at:
https://docs.docker.com/engine/reference/builder/#environment-replacement
To quote from there:
The ${variable_name} syntax also supports a few of the standard bash modifiers as specified below:
${variable:-word} indicates that if variable is set then the result will be that value. If variable is not set then word will be the result.
${variable:+word} indicates that if variable is set then word will be the result, otherwise the result is the empty string.
In all cases, word can be any string, including additional environment variables.
Escaping is possible by adding a \ before the variable: \$foo or \${foo}, for example, will translate to $foo and ${foo} literals respectively.