Docker-compose variable-substitution mandatory variables - docker

I have a docker-compose file that uses variable substitution for some secrets and I want to get an error if they are not supplied or empty, for this purpose I have tried this:
environment:
- >-
JAVA_OPTS=
-DMYSQL_USER=${MYSQL_USER:?MYSQL_USER_NOT_SET}
-DMYSQL_PASSWORD=${MYSQL_PASSWORD:?MYSQL_PASSWORD_NOT_SET}
-DMYSQL_URL=db:3306/${MYSQL_DATABASE:?MYSQL_DATABASE_NOT_SET}
However, it gives me the error:
ERROR: Invalid interpolation format for "environment" option in service "myservice": "JAVA_OPTS= -DMYSQL_USER=${MYSQL_USER:?MYSQL_USER_NOT_SET}...
According to https://docs.docker.com/compose/compose-file/#variable-substitution this should work since it has this snippet:
Similarly, the following syntax allows you to specify mandatory
variables:
${VARIABLE:?err} exits with an error message containing err if
VARIABLE is unset or empty in the environment. ${VARIABLE?err} exits
with an error message containing err if VARIABLE is unset in the
environment.
I also have version: "3.4" in my docker-compose so that shouldn't be the issue.
Already tried it with just ${MY_VAR?MY_ERROR} but it didn't work either.
I have even gone as far as to look at the source code but found nothing helpful.
EDIT :
I tried to make a minimum size reproduction:
docker-compose.yml
version: "3.4"
services:
hello:
image: hello-world
environment:
- TEST=${TEST?err}
docker-compose up
ERROR: Invalid interpolation format for "environment" option in service "hello": "TEST=${TEST?err}

This depends on your docker-compose version.
With docker-compose 1.17.1 you will get
ERROR: Invalid interpolation format for "environment" option in service "my-service": ...
if you use ${TEST?"My error message"} but with
e.g. docker-compose 1.29.2 it works as expected
ERROR: Missing mandatory value for "environment" option interpolating ... in service "my-service": "My error message"

Related

IoT-Agent OPC-UA Docker-compose setting for NGSI ld or NGSI v2

In the docker-composer files of the OPC-UA IoT-Agent there are some comments unclear to me, in particular at the line is told to comment if you want to use NGSI-LD or to comment the line if you want to use NGSI-V2.
Reading the strings that should be commented out however, it would seem that it is necessary to remove the comments from both the lines to use NGSI-LD, and comment both of them to use NGS-V2.
Is my interpretation correct? Thanks for clearing it up.
PS: the same issue is present to the file docker-compose-external-server.yml
Setting up NGSI-v2 vs NGSI-LD is common to all IoT Agents. The Installation Guide describes the required configuration - default operation is NGSI-v2.
If you want to operate NGSI-LD, the ngsiVersion and jsonLdContext must be defined.
{
host: '192.168.56.101',
port: '1026',
ngsiVersion: 'ld',
jsonLdContext: 'http://context.json-ld'
}
ngsiVersion can be v2, ld or mixed.
Both settings can also be set up using Environment Variables which is more convenient when using Docker
Therefore, for NGSI-LD the following minimal set-up is required:
iotage:
hostname: iotage
image: iotagent4fiware/iotagent-opcua:latest
environment:
- IOTA_CB_NGSI_VERSION=ld
- IOTA_JSON_LD_CONTEXT=https://path-to-context-file
- IOTA_FALLBACK_TENANT=opcua_car
- IOTA_RELAX_TEMPLATE_VALIDATION=true
For NGSI-v2 the following is required:
iotage:
hostname: iotage
image: iotagent4fiware/iotagent-opcua:latest
environment:
- IOTA_CB_NGSI_VERSION=v2
- IOTA_RELAX_TEMPLATE_VALIDATION=true
IOTA_RELAX_TEMPLATE_VALIDATION is required for OPC-UA to allow the provisioning of OPC-UA topics with = within them which would normally be disallowed.

docker-compose contains an invalid type

In my docker-compose.yml I'm getting an error when I run docker-compose up
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.pulsar.ports contains an invalid type, it should be a number, or an object
I believe the problem lies with this line:
ports:
- "${HOST_PULSAR_PORT}:8999"
I'm not understanding the syntax of ${PORT_NAME} or what the error is taking issue with exactly...

How can you conditionally set services in the docker-compose.yml file?

I'm new to using Docker and Cake. At the moment we have a simple Cake task that runs the DockerComposeUp() method that takes a DockerComposeUpSettings object. The docker-compose.yaml file holds some info on a service that I want to conditionally run (serviceA):
version: "1.0"
services:
serviceA:
image: someImage
ports:
-"000000"
-"000001"
serviceB:
image: someOtherImage
anotherProperty: somethingElse
ports:
-"111111"
I've tried splitting out serviceA into a separate docker-compose file called 'docker-compose.serviceA.yaml' and calling it by adding to the DockerComposeUpSettings.ArgumentCustomization the following:
if(some setting)
{
dockerComposeUpSettings.ArgumentCustomization = builder => builder.Append("-f docker-compose.yaml -f docker-compose.serviceA.yaml");
}
However, Cake throws the following error:
"unknown shorthand flag: 'f' in -f"
How can I merge to docker-compose files as part of the DockerComposeUp method using Cake?
Update
I've found there is a 'Files' property on the DockerComposeUpSettings object (inherited from DockerComposeSettings object), where you can declare the configuration files. So I've added:
if(some flag)
{
dockerComposeSettings.Files = new[]{ "docker-compose.yaml", "docker-compose.serviceA.yaml" };
}
I don't know much about docker, but looking at the docs here and here it seems it would be important to have the -f option set before the command specified on the commandline. Your customization (builder.Append()) puts them at the end of the commandline.
Have you tried setting the Files property of the DockerComposeUpSettings? That looks like what you are looking for.

What is the syntax for ELASTICSEARCH_HOSTS in docker-compose?

Trying to figure out Kibana's ELASTICSEARCH_HOSTS syntax, but I receive either:
kib01 | FATAL Error: [config validation of [elasticsearch].hosts]: types that failed validation:
kib01 | - [config validation of [elasticsearch].hosts.0]: expected URI with scheme [http|https].
kib01 | - [config validation of [elasticsearch].hosts.1]: could not parse array value from json input
from Kibana itself or:
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.kibana.environment.ELASTICSEARCH_HOSTS contains ["http://es01:9200", "http://es02:9200", "http://es03:9200", "http://es04:9200"], which is an invalid type, it should be a string, number, or a null
from Docker compose.
My latest iteration is:
environment:
ELASTICSEARCH_HOSTS=["http://es01:9200", "http://es02:9200", "http://es03:9200", "http://es04:9200"]
I have also tried:
environment:
- ELASTICSEARCH_HOSTS: '["http://es01:9200", "http://es02:9200", "http://es03:9200", "http://es04:9200"]'
I tried turning that into a list:
environment:
ELASTICSEARCH_HOSTS:
- "<host1>"
- "<host2>"
I tried removing quotes in various places in the above variation and various combinations on the list.
I also tried a combination from the official documentation using both : and =
with ELASTICSEARCH_HOST = http://es01:9200, http://es02:9200...
All have been rejected. Does anyone know the magic syntax to get this to work?
Updated List of failures:
ELASTICSEARCH_HOSTS: ['http://es01:9200','http://es02:9200','http://es03:9200','http://es04:9200']
ELASTICSEARCH_HOSTS="['http://es01:9200','http://es02:9200','http://es03:9200','http://es04:9200']"
- ELASTICSEARCH_HOSTS=['http://es01:9200','http://es02:9200','http://es03:9200','http://es04:9200']
- ELASTICSEARCH_HOSTS="["http://es01:9200","http://es02:9200","http://es03:9200","http://es04:9200"]"
ELASTICSEARCH_HOSTS:
- "<host1>"
- "<host2>"
There are some things to note to solve your problem:
In docker-compose, your environment variables must be written as an object, or an array:
# A native yaml approach to define key-value objects
environment:
KEY1: VAL1
KEY2: VAL2
# OR
# Some special way for compose yaml parser
# that can split key and value from a "KEY=VAL" string
environment:
- KEY1=VAL1
- KEY2=VAL2
The value of an environment variable must be a string (the above example), a number, or null (empty value). Note that [foo, bar] is parsed as a list in the first format (and if you want it to be parsed as a string, you need to wrap it inside quotation marks), but the second format parses it as a string.
In this forum question, there is an example to how pass multiple elasticsearch hosts as an environment variable (ELASTICSEARCH_HOSTS).
So this must be a valid example that docker-compose and Kibana can both understand:
environment:
ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200","http://es04:9200"]'
# OR
environment:
- ELASTICSEARCH_HOSTS=["http://es01:9200","http://es02:9200","http://es03:9200","http://es04:9200"]

spring-cloud-skipper : How to delete and re-deploy a package

I want to redeploy a package but I am getting an error:
skipper:>package install --package-name sg-cloud-MbakTestworld
--package-version 0.0.1 --release-name MbakTestworld --file E:\skipper\apps\MbakTestworld-upgrade-local.yml
Result:
Release with the name [] already exists and it is not deleted. Details
of the error have been omitted. You can use the stacktrace command to
print the full stacktrace.
My yml File:
spec:
applicationProperties:
server.port: 8029
spring.profiles.active: mbakCloud
deploymentProperties:
spring.cloud.deployer.memory: 512m
I found the answer:
skipper:>release delete --release-name MbakTestworld
MbakTestworld has been deleted.
Source: https://docs.spring.io/spring-cloud-skipper/docs/current/reference/htmlsingle/

Resources