How can you conditionally set services in the docker-compose.yml file? - docker

I'm new to using Docker and Cake. At the moment we have a simple Cake task that runs the DockerComposeUp() method that takes a DockerComposeUpSettings object. The docker-compose.yaml file holds some info on a service that I want to conditionally run (serviceA):
version: "1.0"
services:
serviceA:
image: someImage
ports:
-"000000"
-"000001"
serviceB:
image: someOtherImage
anotherProperty: somethingElse
ports:
-"111111"
I've tried splitting out serviceA into a separate docker-compose file called 'docker-compose.serviceA.yaml' and calling it by adding to the DockerComposeUpSettings.ArgumentCustomization the following:
if(some setting)
{
dockerComposeUpSettings.ArgumentCustomization = builder => builder.Append("-f docker-compose.yaml -f docker-compose.serviceA.yaml");
}
However, Cake throws the following error:
"unknown shorthand flag: 'f' in -f"
How can I merge to docker-compose files as part of the DockerComposeUp method using Cake?
Update
I've found there is a 'Files' property on the DockerComposeUpSettings object (inherited from DockerComposeSettings object), where you can declare the configuration files. So I've added:
if(some flag)
{
dockerComposeSettings.Files = new[]{ "docker-compose.yaml", "docker-compose.serviceA.yaml" };
}

I don't know much about docker, but looking at the docs here and here it seems it would be important to have the -f option set before the command specified on the commandline. Your customization (builder.Append()) puts them at the end of the commandline.
Have you tried setting the Files property of the DockerComposeUpSettings? That looks like what you are looking for.

Related

IoT-Agent OPC-UA Docker-compose setting for NGSI ld or NGSI v2

In the docker-composer files of the OPC-UA IoT-Agent there are some comments unclear to me, in particular at the line is told to comment if you want to use NGSI-LD or to comment the line if you want to use NGSI-V2.
Reading the strings that should be commented out however, it would seem that it is necessary to remove the comments from both the lines to use NGSI-LD, and comment both of them to use NGS-V2.
Is my interpretation correct? Thanks for clearing it up.
PS: the same issue is present to the file docker-compose-external-server.yml
Setting up NGSI-v2 vs NGSI-LD is common to all IoT Agents. The Installation Guide describes the required configuration - default operation is NGSI-v2.
If you want to operate NGSI-LD, the ngsiVersion and jsonLdContext must be defined.
{
host: '192.168.56.101',
port: '1026',
ngsiVersion: 'ld',
jsonLdContext: 'http://context.json-ld'
}
ngsiVersion can be v2, ld or mixed.
Both settings can also be set up using Environment Variables which is more convenient when using Docker
Therefore, for NGSI-LD the following minimal set-up is required:
iotage:
hostname: iotage
image: iotagent4fiware/iotagent-opcua:latest
environment:
- IOTA_CB_NGSI_VERSION=ld
- IOTA_JSON_LD_CONTEXT=https://path-to-context-file
- IOTA_FALLBACK_TENANT=opcua_car
- IOTA_RELAX_TEMPLATE_VALIDATION=true
For NGSI-v2 the following is required:
iotage:
hostname: iotage
image: iotagent4fiware/iotagent-opcua:latest
environment:
- IOTA_CB_NGSI_VERSION=v2
- IOTA_RELAX_TEMPLATE_VALIDATION=true
IOTA_RELAX_TEMPLATE_VALIDATION is required for OPC-UA to allow the provisioning of OPC-UA topics with = within them which would normally be disallowed.

Docker-compose variable-substitution mandatory variables

I have a docker-compose file that uses variable substitution for some secrets and I want to get an error if they are not supplied or empty, for this purpose I have tried this:
environment:
- >-
JAVA_OPTS=
-DMYSQL_USER=${MYSQL_USER:?MYSQL_USER_NOT_SET}
-DMYSQL_PASSWORD=${MYSQL_PASSWORD:?MYSQL_PASSWORD_NOT_SET}
-DMYSQL_URL=db:3306/${MYSQL_DATABASE:?MYSQL_DATABASE_NOT_SET}
However, it gives me the error:
ERROR: Invalid interpolation format for "environment" option in service "myservice": "JAVA_OPTS= -DMYSQL_USER=${MYSQL_USER:?MYSQL_USER_NOT_SET}...
According to https://docs.docker.com/compose/compose-file/#variable-substitution this should work since it has this snippet:
Similarly, the following syntax allows you to specify mandatory
variables:
${VARIABLE:?err} exits with an error message containing err if
VARIABLE is unset or empty in the environment. ${VARIABLE?err} exits
with an error message containing err if VARIABLE is unset in the
environment.
I also have version: "3.4" in my docker-compose so that shouldn't be the issue.
Already tried it with just ${MY_VAR?MY_ERROR} but it didn't work either.
I have even gone as far as to look at the source code but found nothing helpful.
EDIT :
I tried to make a minimum size reproduction:
docker-compose.yml
version: "3.4"
services:
hello:
image: hello-world
environment:
- TEST=${TEST?err}
docker-compose up
ERROR: Invalid interpolation format for "environment" option in service "hello": "TEST=${TEST?err}
This depends on your docker-compose version.
With docker-compose 1.17.1 you will get
ERROR: Invalid interpolation format for "environment" option in service "my-service": ...
if you use ${TEST?"My error message"} but with
e.g. docker-compose 1.29.2 it works as expected
ERROR: Missing mandatory value for "environment" option interpolating ... in service "my-service": "My error message"

pyhdfs.HdfsIOException: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

I am trying to run hadoop using docker provided here:
https://github.com/big-data-europe/docker-hadoop
I use the following command:
docker-compose up -d
to up the service and am able to access it and browse file system using: localhost:9870. Problem rises whenever I try to use pyhdfs to put file on HDFS. Here is my sample code:
hdfs_client = HdfsClient(hosts = 'localhost:9870')
# Determine the output_hdfs_path
output_hdfs_path = 'path/to/test/dir'
# Does the output path exist? If not then create it
if not hdfs_client.exists(output_hdfs_path):
hdfs_client.mkdirs(output_hdfs_path)
hdfs_client.create(output_hdfs_path + 'data.json', data = 'This is test.', overwrite = True)
If test directory does not exist on HDFS, the code is able to successfully create it but when it gets to the .create part it throws the following exception:
pyhdfs.HdfsIOException: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null
What surprises me is that my code is able to create the empty directory but fails to put the file on HDFS. My docker-compose.yml file is exactly the same as the one provided in the github repo. The only change I've made is in the hadoop.env file where I change:
CORE_CONF_fs_defaultFS=hdfs://namenode:9000
to
CORE_CONF_fs_defaultFS=hdfs://localhost:9000
I have seen this other post on sof and tried the following command:
hdfs dfs -mkdir hdfs:///demofolder
which works fine in my case. Any help is much appreciated.
I would keep the default CORE_CONF_fs_defaultFS=hdfs://namenode:9000 setting.
Works fine for me after adding a forward slash to the paths
import pyhdfs
fs = pyhdfs.HdfsClient(hosts="namenode")
output_hdfs_path = '/path/to/test/dir'
if not fs.exists(output_hdfs_path):
fs.mkdirs(output_hdfs_path)
fs.create(output_hdfs_path + '/data.json', data = 'This is test.')
# check that it's present
list(fs.walk(output_hdfs_path))
[('/path/to/test/dir', [], ['data.json'])]

Use MongoDB with docker-compose: create database and user

Is possible to create a database using docker-compose? I'm trying to run mongodb on docker but I'm not able to create user and initial database :(
version: '3.1'
services:
mongo:
image: mongo
restart: always
environment:
- MONGO_INITDB_DATABASE=test
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=admin
volumes:
- ~/docker/volumes/mongodb:/data/db
ports:
- 27017:27017
What is missing in my script?
test code:
from pymongo import MongoClient
mongo_client = MongoClient('mongodb://%s:%s#127.0.0.1' % ('admin', 'admin'))
cursor = mongo_client.list_databases()
for db in cursor:
print(db)
The above docker-compose seems fine, it will create a user named admin and the DB named will be test if your *.js file contain insert data script. otherwise, it will not create Database because so if you do not insert data with your JavaScript files, then no database is created.
You can log in with these credential that specifies in env.
- MONGO_INITDB_DATABASE=test
- MONGO_INITDB_ROOT_USERNAME=test
- MONGO_INITDB_ROOT_PASSWORD=admin
To initialize DB, you need to mount you DB init script.
volumes:
- ~/docker/volumes/mongodb:/data/db
- youdbscript/init.js:/docker-entrypoint-initdb.d/
MONGO_INITDB_DATABASE
This variable allows you to specify the name of a database to be used
for creation scripts in /docker-entrypoint-initdb.d/*.js (see
Initializing a fresh instance below). MongoDB is fundamentally
designed for "create on first use", so if you do not insert data with
your JavaScript files, then no database is created.
Initializing a fresh instance
When a container is started for the first time it will execute files
with extensions .sh and .js that are found in
/docker-entrypoint-initdb.d. Files will be executed in alphabetical
order. .js files will be executed by mongo using the database
specified by the MONGO_INITDB_DATABASE variable, if it is present, or
test otherwise. You may also switch databases within the .js script.
mongo-docker
Update:
For your information the code you pasted code is not js file. its python script. Also, change the user name from admin might conflict with admin.
from pymongo import MongoClient
mongo_client = MongoClient('mongodb://%s:%s#127.0.0.1' % ('admin', 'admin'))
cursor = mongo_client.list_databases()
for db in cursor:
print(db)
You init script will some thing like
youdbscript/init.js
You can try this.
db = db.getSiblingDB("test");
db.article.drop();
db.article.save( {
title : "this is my title" ,
author : "bob" ,
posted : new Date(1079895594000) ,
pageViews : 5 ,
tags : [ "fun" , "good" , "fun" ] ,
comments : [
{ author :"joe" , text : "this is cool" } ,
{ author :"sam" , text : "this is bad" }
],
other : { foo : 5 }
});
Just like #Adiii answer. If you just declare a db name and not insert some data. The db will not created actually.
As your question. You can add one simple script (such as .sh or .js script) to the path /docker-entrypoint-initdb.d. Like the docs on mongo Initializing a fresh instance section.

How to properly setup docker-sync to exclude folders

I am trying to setup docker-sync to exclude my app/cache and app/logs folder but it is not working.
Things I've tried:
Using sync_excludes: ['.idea', 'app/cache/', 'app/logs/'] but it will be translated to something like this
command unison -ignore='Name .idea' -ignore='Name app/cache/*'
-ignore='Name app/logs/*'
So then I tried using sync_args and set like this:
sync_args:
- "-debug verbose"
- "-ignore='Path app/cache'"
- "-ignore='Path app/logs'"
command unison -ignore='Name .idea' -ignore='Name systems' \
-debug verbose -ignore='Path app/cache/*' -ignore='Path app/logs/*'
looking at the logs I can see this
[unox][DEBUG]: recvCmd: DIR
[unox][DEBUG]: sendCmd: OK
[fswatch+] >> OK
[pred] immutable 'app/cache' = false
[pred] ignore 'app/cache/prod' = true
[pred] ignorenot 'app/cache/prod' = false
[ignore] buildUpdateChildren: ignoring path app/cache/prod
But this still seeing the events being triggered and it is still syncing to my host machine.
[pred] ignore 'app/cache/prod/annotations/5702f47f407ddb07532bfd60d8ea2919489ef4bc#__construct.cache.php' = false
[pred] ignore 'app/cache/prod/annotations/f21b469a2214195ff16e2af43f249bdbfa245c25#findPublishedOr404.cache.php' = false
Anyone know what I am missing?
my last version look like this:
version: "2"
options:
# optional, activate this if you need to debug something, default is false
# IMPORTANT: do not run stable with this, it creates a memory leak, turn off verbose when you are done testing
verbose: true
syncs:
#IMPORTANT: ensure this name is unique
dt-akeneo-unison-sync:
notify_terminal: true
# which folder to watch / sync from - you can use tilde (~), it will get expanded. Be aware that the trailing slash makes a difference
# if you add them, only the inner parts of the folder gets synced, otherwise the parent folder will be synced as top-level folder
src: './'
# the files should be own by root in the target cointainer
sync_userid: 1000
sync_strategy: 'unison'
# optional, a list of regular expressions to exclude from the fswatch - see fswatch docs for details
watch_excludes: ['\.git', '\.gitignore', '.*\.md']
sync_args:
- "-debug verbose" #force Unison to choose the file with the later (earlier) modtime
- "-ignore='Path app/cache'"
- "-ignore='Path app/logs'"
- "-ignore='Path .git'"
- "-ignore='Path .git'"
- "-ignore='Path vendor'"
- "-ignore='Path upgrades'"
- "-ignore='Path systems'"
# optional: use this to switch to fswatch verbose mode
watch_args: '-v'
My Env:
{11:41}~ ➭ docker -v
Docker version 17.09.0-ce, build afdb6d4
{11:42}~ ➭ docker-sync -v
0.4.6
SO: OSx 10.11.6
Running my projects on a mounted Filesystem Case-Sensitive created using https://gist.github.com/scottsb/479bebe8b4b86bf17e2d
/dev/disk2s2 on /Users/neisantos/src (hfs, local, nodev, nosuid, journaled, noowners, nobrowse)
Don't know if it's too late.
I have almost the same setup:
OSx 10.12.6
Docker version 18.06.0-ce, build 0ffa825
docker-sync v 0.5.7
My project structure looks like this
./
build
changes
docker
etc
var
some-yml-files.yml
My docker-sync.yml looks like this
version: "2"
options:
verbose: true
syncs:
my-appcode-sync: # tip: add -sync and you keep consistent names as a convention
src: '.'
# sync_strategy: 'native_osx' # not needed, this is the default now
sync_excludes: ['docker/var', 'changes', '.git', '.idea']
I took this from https://github.com/EugenMayer/docker-sync/issues/421#issuecomment-309244156
Note the src: '.'. Initially I used src: './' like you did and ignoring folders didn't work either. After removing the / it worked for me.

Resources