What is the syntax for ELASTICSEARCH_HOSTS in docker-compose? - docker

Trying to figure out Kibana's ELASTICSEARCH_HOSTS syntax, but I receive either:
kib01 | FATAL Error: [config validation of [elasticsearch].hosts]: types that failed validation:
kib01 | - [config validation of [elasticsearch].hosts.0]: expected URI with scheme [http|https].
kib01 | - [config validation of [elasticsearch].hosts.1]: could not parse array value from json input
from Kibana itself or:
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.kibana.environment.ELASTICSEARCH_HOSTS contains ["http://es01:9200", "http://es02:9200", "http://es03:9200", "http://es04:9200"], which is an invalid type, it should be a string, number, or a null
from Docker compose.
My latest iteration is:
environment:
ELASTICSEARCH_HOSTS=["http://es01:9200", "http://es02:9200", "http://es03:9200", "http://es04:9200"]
I have also tried:
environment:
- ELASTICSEARCH_HOSTS: '["http://es01:9200", "http://es02:9200", "http://es03:9200", "http://es04:9200"]'
I tried turning that into a list:
environment:
ELASTICSEARCH_HOSTS:
- "<host1>"
- "<host2>"
I tried removing quotes in various places in the above variation and various combinations on the list.
I also tried a combination from the official documentation using both : and =
with ELASTICSEARCH_HOST = http://es01:9200, http://es02:9200...
All have been rejected. Does anyone know the magic syntax to get this to work?
Updated List of failures:
ELASTICSEARCH_HOSTS: ['http://es01:9200','http://es02:9200','http://es03:9200','http://es04:9200']
ELASTICSEARCH_HOSTS="['http://es01:9200','http://es02:9200','http://es03:9200','http://es04:9200']"
- ELASTICSEARCH_HOSTS=['http://es01:9200','http://es02:9200','http://es03:9200','http://es04:9200']
- ELASTICSEARCH_HOSTS="["http://es01:9200","http://es02:9200","http://es03:9200","http://es04:9200"]"
ELASTICSEARCH_HOSTS:
- "<host1>"
- "<host2>"

There are some things to note to solve your problem:
In docker-compose, your environment variables must be written as an object, or an array:
# A native yaml approach to define key-value objects
environment:
KEY1: VAL1
KEY2: VAL2
# OR
# Some special way for compose yaml parser
# that can split key and value from a "KEY=VAL" string
environment:
- KEY1=VAL1
- KEY2=VAL2
The value of an environment variable must be a string (the above example), a number, or null (empty value). Note that [foo, bar] is parsed as a list in the first format (and if you want it to be parsed as a string, you need to wrap it inside quotation marks), but the second format parses it as a string.
In this forum question, there is an example to how pass multiple elasticsearch hosts as an environment variable (ELASTICSEARCH_HOSTS).
So this must be a valid example that docker-compose and Kibana can both understand:
environment:
ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200","http://es04:9200"]'
# OR
environment:
- ELASTICSEARCH_HOSTS=["http://es01:9200","http://es02:9200","http://es03:9200","http://es04:9200"]

Related

IoT-Agent OPC-UA Docker-compose setting for NGSI ld or NGSI v2

In the docker-composer files of the OPC-UA IoT-Agent there are some comments unclear to me, in particular at the line is told to comment if you want to use NGSI-LD or to comment the line if you want to use NGSI-V2.
Reading the strings that should be commented out however, it would seem that it is necessary to remove the comments from both the lines to use NGSI-LD, and comment both of them to use NGS-V2.
Is my interpretation correct? Thanks for clearing it up.
PS: the same issue is present to the file docker-compose-external-server.yml
Setting up NGSI-v2 vs NGSI-LD is common to all IoT Agents. The Installation Guide describes the required configuration - default operation is NGSI-v2.
If you want to operate NGSI-LD, the ngsiVersion and jsonLdContext must be defined.
{
host: '192.168.56.101',
port: '1026',
ngsiVersion: 'ld',
jsonLdContext: 'http://context.json-ld'
}
ngsiVersion can be v2, ld or mixed.
Both settings can also be set up using Environment Variables which is more convenient when using Docker
Therefore, for NGSI-LD the following minimal set-up is required:
iotage:
hostname: iotage
image: iotagent4fiware/iotagent-opcua:latest
environment:
- IOTA_CB_NGSI_VERSION=ld
- IOTA_JSON_LD_CONTEXT=https://path-to-context-file
- IOTA_FALLBACK_TENANT=opcua_car
- IOTA_RELAX_TEMPLATE_VALIDATION=true
For NGSI-v2 the following is required:
iotage:
hostname: iotage
image: iotagent4fiware/iotagent-opcua:latest
environment:
- IOTA_CB_NGSI_VERSION=v2
- IOTA_RELAX_TEMPLATE_VALIDATION=true
IOTA_RELAX_TEMPLATE_VALIDATION is required for OPC-UA to allow the provisioning of OPC-UA topics with = within them which would normally be disallowed.

docker-compose contains an invalid type

In my docker-compose.yml I'm getting an error when I run docker-compose up
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.pulsar.ports contains an invalid type, it should be a number, or an object
I believe the problem lies with this line:
ports:
- "${HOST_PULSAR_PORT}:8999"
I'm not understanding the syntax of ${PORT_NAME} or what the error is taking issue with exactly...

How can you conditionally set services in the docker-compose.yml file?

I'm new to using Docker and Cake. At the moment we have a simple Cake task that runs the DockerComposeUp() method that takes a DockerComposeUpSettings object. The docker-compose.yaml file holds some info on a service that I want to conditionally run (serviceA):
version: "1.0"
services:
serviceA:
image: someImage
ports:
-"000000"
-"000001"
serviceB:
image: someOtherImage
anotherProperty: somethingElse
ports:
-"111111"
I've tried splitting out serviceA into a separate docker-compose file called 'docker-compose.serviceA.yaml' and calling it by adding to the DockerComposeUpSettings.ArgumentCustomization the following:
if(some setting)
{
dockerComposeUpSettings.ArgumentCustomization = builder => builder.Append("-f docker-compose.yaml -f docker-compose.serviceA.yaml");
}
However, Cake throws the following error:
"unknown shorthand flag: 'f' in -f"
How can I merge to docker-compose files as part of the DockerComposeUp method using Cake?
Update
I've found there is a 'Files' property on the DockerComposeUpSettings object (inherited from DockerComposeSettings object), where you can declare the configuration files. So I've added:
if(some flag)
{
dockerComposeSettings.Files = new[]{ "docker-compose.yaml", "docker-compose.serviceA.yaml" };
}
I don't know much about docker, but looking at the docs here and here it seems it would be important to have the -f option set before the command specified on the commandline. Your customization (builder.Append()) puts them at the end of the commandline.
Have you tried setting the Files property of the DockerComposeUpSettings? That looks like what you are looking for.

Docker-compose variable-substitution mandatory variables

I have a docker-compose file that uses variable substitution for some secrets and I want to get an error if they are not supplied or empty, for this purpose I have tried this:
environment:
- >-
JAVA_OPTS=
-DMYSQL_USER=${MYSQL_USER:?MYSQL_USER_NOT_SET}
-DMYSQL_PASSWORD=${MYSQL_PASSWORD:?MYSQL_PASSWORD_NOT_SET}
-DMYSQL_URL=db:3306/${MYSQL_DATABASE:?MYSQL_DATABASE_NOT_SET}
However, it gives me the error:
ERROR: Invalid interpolation format for "environment" option in service "myservice": "JAVA_OPTS= -DMYSQL_USER=${MYSQL_USER:?MYSQL_USER_NOT_SET}...
According to https://docs.docker.com/compose/compose-file/#variable-substitution this should work since it has this snippet:
Similarly, the following syntax allows you to specify mandatory
variables:
${VARIABLE:?err} exits with an error message containing err if
VARIABLE is unset or empty in the environment. ${VARIABLE?err} exits
with an error message containing err if VARIABLE is unset in the
environment.
I also have version: "3.4" in my docker-compose so that shouldn't be the issue.
Already tried it with just ${MY_VAR?MY_ERROR} but it didn't work either.
I have even gone as far as to look at the source code but found nothing helpful.
EDIT :
I tried to make a minimum size reproduction:
docker-compose.yml
version: "3.4"
services:
hello:
image: hello-world
environment:
- TEST=${TEST?err}
docker-compose up
ERROR: Invalid interpolation format for "environment" option in service "hello": "TEST=${TEST?err}
This depends on your docker-compose version.
With docker-compose 1.17.1 you will get
ERROR: Invalid interpolation format for "environment" option in service "my-service": ...
if you use ${TEST?"My error message"} but with
e.g. docker-compose 1.29.2 it works as expected
ERROR: Missing mandatory value for "environment" option interpolating ... in service "my-service": "My error message"

Use MongoDB with docker-compose: create database and user

Is possible to create a database using docker-compose? I'm trying to run mongodb on docker but I'm not able to create user and initial database :(
version: '3.1'
services:
mongo:
image: mongo
restart: always
environment:
- MONGO_INITDB_DATABASE=test
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=admin
volumes:
- ~/docker/volumes/mongodb:/data/db
ports:
- 27017:27017
What is missing in my script?
test code:
from pymongo import MongoClient
mongo_client = MongoClient('mongodb://%s:%s#127.0.0.1' % ('admin', 'admin'))
cursor = mongo_client.list_databases()
for db in cursor:
print(db)
The above docker-compose seems fine, it will create a user named admin and the DB named will be test if your *.js file contain insert data script. otherwise, it will not create Database because so if you do not insert data with your JavaScript files, then no database is created.
You can log in with these credential that specifies in env.
- MONGO_INITDB_DATABASE=test
- MONGO_INITDB_ROOT_USERNAME=test
- MONGO_INITDB_ROOT_PASSWORD=admin
To initialize DB, you need to mount you DB init script.
volumes:
- ~/docker/volumes/mongodb:/data/db
- youdbscript/init.js:/docker-entrypoint-initdb.d/
MONGO_INITDB_DATABASE
This variable allows you to specify the name of a database to be used
for creation scripts in /docker-entrypoint-initdb.d/*.js (see
Initializing a fresh instance below). MongoDB is fundamentally
designed for "create on first use", so if you do not insert data with
your JavaScript files, then no database is created.
Initializing a fresh instance
When a container is started for the first time it will execute files
with extensions .sh and .js that are found in
/docker-entrypoint-initdb.d. Files will be executed in alphabetical
order. .js files will be executed by mongo using the database
specified by the MONGO_INITDB_DATABASE variable, if it is present, or
test otherwise. You may also switch databases within the .js script.
mongo-docker
Update:
For your information the code you pasted code is not js file. its python script. Also, change the user name from admin might conflict with admin.
from pymongo import MongoClient
mongo_client = MongoClient('mongodb://%s:%s#127.0.0.1' % ('admin', 'admin'))
cursor = mongo_client.list_databases()
for db in cursor:
print(db)
You init script will some thing like
youdbscript/init.js
You can try this.
db = db.getSiblingDB("test");
db.article.drop();
db.article.save( {
title : "this is my title" ,
author : "bob" ,
posted : new Date(1079895594000) ,
pageViews : 5 ,
tags : [ "fun" , "good" , "fun" ] ,
comments : [
{ author :"joe" , text : "this is cool" } ,
{ author :"sam" , text : "this is bad" }
],
other : { foo : 5 }
});
Just like #Adiii answer. If you just declare a db name and not insert some data. The db will not created actually.
As your question. You can add one simple script (such as .sh or .js script) to the path /docker-entrypoint-initdb.d. Like the docs on mongo Initializing a fresh instance section.

Resources