Some of my Docker containers require configuration files (think: JSON, YAML or INI files) that depend on environment variables and/or container arguments.
What is the standard approach to automatically generate those environment files?
Solutions I have considered so far include:
Tiller: but I'm not excited by the idea of adding a ruby gem to my containers.
A custom string replacement script written in the same language as my other applications: but I don't want to include an extra script in all my docker projects.
sed -i s/ENVVAR/${ENVVAR}/g config.json: it works but it's a bit too "raw" for my taste.
Check out confd, highly recommend it for this particular use case of "I want templating with minimal clutter in my image". confd allows you to use a local environment variable backend to template out whatever config file format you want.
Related
My goal is to put my telegraf config into source control. To do so, I have a repo in my user's home directory with the appropriate config file which has already been tested and proven working.
I have added the path to the new config file in the "default" environment variables file:
/etc/default/telegraf
like this:
TELEGRAF_CONFIG_PATH="/home/ubuntu/some_repo/telegraf.conf"
... as well as other required variables such as passwords.
However, when I attempt to run
telegraf --test
It says No config file specified, and could not find one in $TELEGRAF_CONFIG_PATH etc.
Further, if I force it by
telegraf --test --config /home/ubuntu/some_repo/telegraf.conf
Then the process fails because it is missing the other required variables.
Questions:
What am I doing wrong?
Is there not also a way of specifying a config directory too (I would like to break my file down into separate input files)?
Perhaps as an alternative to all of this... is there not a way of specifying additional configuration files to be included from within the default /etc/telegraf/telegraf.conf file? (I've been unable to find any mention of this in documentation).
What am I doing wrong?
See what user:group owns /etc/default/telegraf. This file is better used when running telegraf as a service via systemd. Additionally, if you run env do you see the TELEGRAF_CONFIG_PATH variable? What about your other variables? If not, then you probably need to source the file first.
Is there not also a way of specifying a config directory too (I would like to break my file down into separate input files)?
Yes! Take a look at all the options of telegraf with telegraf --help and you will find:
--config-directory <directory> directory containing additional *.conf files
Perhaps as an alternative to all of this... is there not a way of specifying additional configuration files to be included from within the default /etc/telegraf/telegraf.conf file? (I've been unable to find any mention of this in documentation).
That is not the method I would suggest going down. Check out the config directory option above I mentioned.
Ok, after a LOT of trial and error, I figured everything out. For those facing similar issues, here is your shortcut to the answer:
Firstly, remember that when adding variables to the /etc/default/telegraf file, it must effectively be reloaded. So for example using ubuntu systemctl, that requires a restart.
You can verify that the variables have been loaded successfully using this:
$ sudo strings /proc/<pid>/environ
where <pid> is the "Main PID" from the telegraf status output
Secondly, when testing (eg telegraf --test) then (this is the part that is not necessarily intuitive and isn't documented) you will have to ALSO load the same environmental variables into the current user (eg: SET var=value) such that running
$ env
shows the same results as the previous command.
Hint: This is a good method for loading the current env file directly rather than doing it manually.
We have a fairly complex system using docker-compose with a lot of different microservices. I want to be able to run an individual microservice via visual studio with one docker-compose configuration (Debug). Alternatively, I have another configuration (lets call it Debug2) where I want a slightly different docker-compose configuration.
Right now my "docker-compose.yml" file has the basics, and my "docker-compose.override.yml" has some development specific things. I made a "docker-compose.debug.yml". When I run the project in Debug mode, it launches all 3 of those files.
All is well so far, right?
Well, then I tried making a "docker-compose.debug2.yml". I added a new configuration to the project and solution called "Debug2". When I try running from Visual Studio in that mode, it only launches with the first 2 files, and doesn't attempt to use the "debug2" file at all.
Is the system hardcoded to only allow Debug and Release override files? Did I do something wrong or is there an oversight? Any other ideas?
When you are running the services via compose, are you passing the optional override file as well?
For example,
docker-compose -f docker-compose.debug.yml -f docker-compose.debug2.yml
By default, compose only looks for a docker-compose.overrides.yml to my knowledge. Therefore, you would have to pass it as an optional argument when you spin up your environment.
"By default, Compose reads two files, a docker-compose.yml and an optional docker-compose.override.yml file. By convention, the docker-compose.yml contains your base configuration. The override file, as its name implies, can contain configuration overrides for existing services or entirely new services."
For more information: https://docs.docker.com/compose/extends/
For anyone else coming across this issue you can find documentation here:
https://learn.microsoft.com/en-us/visualstudio/containers/docker-compose-properties?view=vs-2019
The two specific file names for "debug" and "release" are:
docker-compose.vs.debug.yml
docker-compose.vs.release.yml
I have a lot of services, which use the same basic configuration in docker-compose. Actually most of the configuration is the same, with some minor tweaks.
I have seen that it is somehow possible to inherit values in YAML. Can I use this in docker-compose to define a "default-service" and use this all over in the other services for e.g. docker-compose run? How would I do this?
No, you cannot do that using YAML. The only inheritance like feature in YAML is the Merge Key Language Independent Type and that only works withing one YAML document, not between multiple documents in the same YAML file (separated by ---) and certainly not between different YAML files.
However docker-compose reads docker-compose.yml and if available docker-compose.override.yml, where the values in the second file (if available) override the ones in the first. Combined with the -f option to specify an input YAML file for docker-compose you can use a shared base file with different overrides.
This is a feature of docker-compose and is done on the data loaded from the YAML files, not by combining the YAML files and then loading them.
Is there ways to specify path to schema in cowboy app? Maybe it's possible to set in my_app.app.src or any config file?
The path to the mnesia directory has to be provided to erlang VM before mnesia application is started through application configuration parameters. In Mnesia tutorial, this is done with the -Application par val VM arguments syntax.
What you call a cowboy application is probably an Erlang OTP release (built by relx as per cowboy tutorial). The solutions, quickly described in Cowboy issue #595, are as follows.
The choice between solutions really depends on style as well as some constraints. Any sufficiently complex release would use a configuration file, so it would be a good choice. vm.args seems easier to deal with. Eventually, you might need to alter the start script (for example to run several nodes from a single deployment), and include some logic to define the mnesia directory.
Provide relx with a configuration file (sys_config option)
To do so, add the following term to relx.config as documented.
{sys_config, "./config/sys.config"}.
sys.config actually is a standard Erlang configuration file, also documented. Specifying mnesia dir is done by adding a section for mnesia application. If no other configuration is required, the file would be:
[{mnesia, [{dir, "/path/to/dir"}]}].
Get relx to pass arguments to the vm (vm_args option)
The vm.args file is actually passed to the VM through -args_file option. This is a simple text file with arguments.
You would add the following term to relx.config as documented.
{vm_args, "./config/vm.args"}.
And put the following content in the vm.args file:
-mnesia dir foo
Write your own start script
relx actually creates a start script, which passes -config sys.config and args_file vm.args to the VM as required. You could modify this script or roll your own to actually pass the -mnesia dir argument to the VM.
in case of vm.args, planit text -mnesia dir foo is invalid, please use format:
-mnesia {dir,foo}
I have an erlang application I have been writing which uses the erldis library for communicating with redis.
Being a bit of a newbie with actually deploying erlang applications to production, I wanted to know if there was anyway to 'bundle' these external libraries with the application rather than installing into my system wide /usr/lib/erlang/lib/ folder.
Currently my directory structure looks like...
\
--\conf
--\ebin
--\src
I have a basic Makefile that I stole from a friend's project, but I am unsure how to write them properly.
I suspect this answer could involve telling me how to write my Makefile properly rather than just which directory to plonk some external library code into.
You should really try to avoid project nesting whenever possible. It can lead to all sorts of problems because of how module/application version is structured within Erlang.
In my development environment, I do a few things to simplify dependancies and multiple developed projects. Specifically, I keep most of my projects sourced in a dev directory and create symlinks into an elibs dir that is set in the ERL_LIBS environmental variables.
~/dev/ngerakines-etap
~/dev/jacobvorreuter-log_roller
~/dev/elib/etap -> ~/dev/ngerakines-etap
~/dev/elib/log_roller -> ~/dev/jacobvorreuter-log_roller
For projects that are deployed, I've either had package-rpm or package-apt make targets that create individual packages per project. Applications get boot scripts and init.d scripts for easy start/stop controls but libraries and dependancy projects just get listed as package dependencies.
I use mochiweb-inspired style. To see example of this get your copy of mochiweb:
svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb
and use
path/to/mochiweb/scripts/new_mochiweb.erl new_project_name
to create sample project of the structure (feel free to delete everything inside src afterwards and use it for your project).
It looks like this:
/
/ebin/
/deps/
/src/
/include/
/support/
/support/include.mk
Makefile
start.sh
ebin contains *.beam files
src contains ***.erl files and local *.hrl files
include contains global *.hrl files
deps contains symlinks to root directories of dependencies
Makefile and include.mk takes care of including appropriate paths when project is built.
start.sh takes care of including appropriate paths when project is run.
So using symlinks in deps directory you are able to fine tune the versions of libraries you use for every project. It is advised to use relative paths, so afterwards it is enough to rsync this structure to the production server and run it.
On more global scale I use the following structure:
~/code/erlang/libs/*/
~/code/category/project/*/
~/code/category/project/*/deps/*/
Where every symlink in deps points to the library in ~/code/erlang/libs/ or to another project in the same category.
The simplest way to do this would be to just create a folder named erldir and put the beams you need into it and then in your start script just use the -pa flag to the erlang runtime to point out where it should fetch the beams.
The correct way (at least if you buy into the OTP distribution model) would be to create a release using reltool (http://www.erlang.org/doc/man/reltool.html) or systools (http://www.erlang.org/doc/man/systools.html) which includes both your application and erldis.
Add the external libraries that you need, anywhere you want them, and add them to your ERL_LIBS environment variable. Separate the paths with colon in unix or semicolon in dos.
Erlang will add the "ebin"-named subdirs to its code loading path.
Have your *.app file point out the other applications it depends on.
This is a good halfway-there approach for setting up larger applications.
Another way is put your lib path in ~/.erlang.
code:add_pathz("/Users/brucexin/sources/mochiweb/ebin").
code:add_pathz("/Users/brucexin/sources/webnesia/ebin").
code:add_pathz("./ebin").
code:add_pathz("/Users/brucexin/sources/erlang-history/ebin/2.15.2").