I have a lot of services, which use the same basic configuration in docker-compose. Actually most of the configuration is the same, with some minor tweaks.
I have seen that it is somehow possible to inherit values in YAML. Can I use this in docker-compose to define a "default-service" and use this all over in the other services for e.g. docker-compose run? How would I do this?
No, you cannot do that using YAML. The only inheritance like feature in YAML is the Merge Key Language Independent Type and that only works withing one YAML document, not between multiple documents in the same YAML file (separated by ---) and certainly not between different YAML files.
However docker-compose reads docker-compose.yml and if available docker-compose.override.yml, where the values in the second file (if available) override the ones in the first. Combined with the -f option to specify an input YAML file for docker-compose you can use a shared base file with different overrides.
This is a feature of docker-compose and is done on the data loaded from the YAML files, not by combining the YAML files and then loading them.
Related
My goal is to put my telegraf config into source control. To do so, I have a repo in my user's home directory with the appropriate config file which has already been tested and proven working.
I have added the path to the new config file in the "default" environment variables file:
/etc/default/telegraf
like this:
TELEGRAF_CONFIG_PATH="/home/ubuntu/some_repo/telegraf.conf"
... as well as other required variables such as passwords.
However, when I attempt to run
telegraf --test
It says No config file specified, and could not find one in $TELEGRAF_CONFIG_PATH etc.
Further, if I force it by
telegraf --test --config /home/ubuntu/some_repo/telegraf.conf
Then the process fails because it is missing the other required variables.
Questions:
What am I doing wrong?
Is there not also a way of specifying a config directory too (I would like to break my file down into separate input files)?
Perhaps as an alternative to all of this... is there not a way of specifying additional configuration files to be included from within the default /etc/telegraf/telegraf.conf file? (I've been unable to find any mention of this in documentation).
What am I doing wrong?
See what user:group owns /etc/default/telegraf. This file is better used when running telegraf as a service via systemd. Additionally, if you run env do you see the TELEGRAF_CONFIG_PATH variable? What about your other variables? If not, then you probably need to source the file first.
Is there not also a way of specifying a config directory too (I would like to break my file down into separate input files)?
Yes! Take a look at all the options of telegraf with telegraf --help and you will find:
--config-directory <directory> directory containing additional *.conf files
Perhaps as an alternative to all of this... is there not a way of specifying additional configuration files to be included from within the default /etc/telegraf/telegraf.conf file? (I've been unable to find any mention of this in documentation).
That is not the method I would suggest going down. Check out the config directory option above I mentioned.
Ok, after a LOT of trial and error, I figured everything out. For those facing similar issues, here is your shortcut to the answer:
Firstly, remember that when adding variables to the /etc/default/telegraf file, it must effectively be reloaded. So for example using ubuntu systemctl, that requires a restart.
You can verify that the variables have been loaded successfully using this:
$ sudo strings /proc/<pid>/environ
where <pid> is the "Main PID" from the telegraf status output
Secondly, when testing (eg telegraf --test) then (this is the part that is not necessarily intuitive and isn't documented) you will have to ALSO load the same environmental variables into the current user (eg: SET var=value) such that running
$ env
shows the same results as the previous command.
Hint: This is a good method for loading the current env file directly rather than doing it manually.
Yml depends on formatting for nested structures, and for inaccurate filled files it is very hard to find out all format errors that violate original meaning. Is there any alternative to yml format for docker-compose configuration file?
TLDR; No.
From Documentation
The Compose file is a YAML file defining services, networks and volumes. The default path for a Compose file is ./docker-compose.yml
Now to answer on your arguments regarding the format and the difficulty to find errors violating original meaning:
yamllint is a first tool that can help your validate your overall yaml syntax (whatever the target expected format).
docker-compose config will read your docker-compose.yml file in and report errors if it does not comply with the expected compose file format.
We have a fairly complex system using docker-compose with a lot of different microservices. I want to be able to run an individual microservice via visual studio with one docker-compose configuration (Debug). Alternatively, I have another configuration (lets call it Debug2) where I want a slightly different docker-compose configuration.
Right now my "docker-compose.yml" file has the basics, and my "docker-compose.override.yml" has some development specific things. I made a "docker-compose.debug.yml". When I run the project in Debug mode, it launches all 3 of those files.
All is well so far, right?
Well, then I tried making a "docker-compose.debug2.yml". I added a new configuration to the project and solution called "Debug2". When I try running from Visual Studio in that mode, it only launches with the first 2 files, and doesn't attempt to use the "debug2" file at all.
Is the system hardcoded to only allow Debug and Release override files? Did I do something wrong or is there an oversight? Any other ideas?
When you are running the services via compose, are you passing the optional override file as well?
For example,
docker-compose -f docker-compose.debug.yml -f docker-compose.debug2.yml
By default, compose only looks for a docker-compose.overrides.yml to my knowledge. Therefore, you would have to pass it as an optional argument when you spin up your environment.
"By default, Compose reads two files, a docker-compose.yml and an optional docker-compose.override.yml file. By convention, the docker-compose.yml contains your base configuration. The override file, as its name implies, can contain configuration overrides for existing services or entirely new services."
For more information: https://docs.docker.com/compose/extends/
For anyone else coming across this issue you can find documentation here:
https://learn.microsoft.com/en-us/visualstudio/containers/docker-compose-properties?view=vs-2019
The two specific file names for "debug" and "release" are:
docker-compose.vs.debug.yml
docker-compose.vs.release.yml
When using the integrated docker-compose structure in Visual Studio 2017, the system provides visibility to a two-tiered docker-compose structure.
docker-compose.yml and docker-compose.override.yml
Settings in the override take precedent over those in the former file. When the file is actually executed, it includes a third, auto-generated Docker compose file ...
docker-compose.vs.*configuration*.g.yml
This latter file contains mostly values related to debugging interactions and mapping volumes in for code you want. Generally, you wouldn't want to change any of these.
One thing that it does by default is to set the entrypoint which ends up becoming the command for the container. As this file is applied last (after the compose file and the override, it is holding precedence over the other two resulting in not being able to override that entrypoint.
Is there a way around this?
Some of my Docker containers require configuration files (think: JSON, YAML or INI files) that depend on environment variables and/or container arguments.
What is the standard approach to automatically generate those environment files?
Solutions I have considered so far include:
Tiller: but I'm not excited by the idea of adding a ruby gem to my containers.
A custom string replacement script written in the same language as my other applications: but I don't want to include an extra script in all my docker projects.
sed -i s/ENVVAR/${ENVVAR}/g config.json: it works but it's a bit too "raw" for my taste.
Check out confd, highly recommend it for this particular use case of "I want templating with minimal clutter in my image". confd allows you to use a local environment variable backend to template out whatever config file format you want.