I am currently using an ansible script to deploy a docker-compose file (using the docker_service module), which sets a series of environment variables which are read by the .NET Core service running inside the docker container, like this:
(...)
environment:
- Poller:Username={{ poller_username }}
- Poller:Password={{ poller_password }}
(...)
The variables for poller_username and poller_password are being loaded from an Ansible Vault (which will be moved to a Hashicorp Vault eventually), and are interpolated into the file with no problem.
However, I have come across a scenario where this logic fails: the user has a '$' in the middle of his password. This means that instead of the environment variable being set to 'abc$123' it's instead set to 'abc', causing my application to fail.
Upon writing a debug command, I get the password output to the console correctly. If I do docker exec <container_name> env I get the wrong password.
Is there a Jinja filter I can use to ensure the password is compliant with docker-compose standards? It doesn't seem viable to me to guarantee the password will never have a $.
EDIT: {{ poller_password | replace("$","$$") }} works, but this isn't a very elegant solution to have in, potentially, every variable I use in the docker-compose module.
For this particular scenario, the {{ poller_password | replace("$","$$") }} solution seems to be inevitable. Thankfully, it appears to be the only case that requires this caution.
Had a similar situation was not a $ but some other character, end up using
something: !unsafe "{{ variable }}"
couldn't find a better way.
Related
Is there a way to get all environment variables that a Docker image accepts? Including authentications and all possible ones to make the best out of that image?
For example, I've run a redis:7.0.8 container and I want to use every possible feature this image offers.
First I used docker inspect and saw this:
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"GOSU_VERSION=1.16",
"REDIS_VERSION=7.0.8",
"REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-7.0.8.tar.gz",
"REDIS_DOWNLOAD_SHA=06a339e491306783dcf55b97f15a5dbcbdc01ccbde6dc23027c475cab735e914"
],
I also tried docker exec -it my-container env which just showed me the same thing. I know there are more variables, for example this doesn't include the following:
REDIS_PASSWORD
REDIS_ACLS
REDIS_TLS_CERT_FILE
Absent documentation, this is pretty much impossible.
Let's start by repeating #jonrsharpe's comment:
They accept any env var at all, but they won't respond to all of them.
Consider this Python code, for example:
import os
def get_environ(d, name):
d.get(name, 'absent')
foo = os.environ.get('FOO', 'default_foo')
star_foo = get_environ(os.environ, foo)
print(star_foo)
This fragment looks up an environment variable $FOO. You could probably figure that out, if you knew the main process was in Python and recognized os.environ. But then it passes that value and the standard environment to a helper function, which looks up that environment variable by name. You'd need detailed static analysis to understand this is actually also an environment-variable lookup.
$ ./test.py
absent
$ default_foo=bar ./test.py
bar
$ FOO=BAR BAR=quux ./test.py
quux
$ I=3 ./test.py
absent
(A fair bit of the code I work with accesses environment variables rather haphazardly; it's not just "find the main function" but "find every ENV reference in every file in every library". Some frameworks like Spring Boot make it possible to set hundreds of configuration options via environment variables, and even if it were possible to get every possible setting here, the output would be prohibitive.)
"What environment variables are there" isn't standard container metadata. You'd have to identify the language the main container process runs, and do this sort of analysis on it, including compiled languages. That doesn't seem like a solvable problem.
I've an issue wit a redirect-middleware in traefik V2.
We want to add a trailing-slash to a sublocation and then remove
the path with a PathPrefix-Rule to get correct paths from the docker service. (MkDOCS)
We defined the rule in dynamic_conf.toml for traefik as a general middleware.:
[...]
[http.middlewares.add-trailing-slash.redirectregex]
regex= "(https?://[^/]+/[a-z0-9_]+)$$"
replacement= "$${1}/"
permanent = true
[...]
At the moment this is our label-file included with docker-run:
traefik.enable=true
traefik.http.routers.dockerservice.entryPoints=websecure
traefik.http.routers.dockerservice.rule=PathPrefix(`/dockerservice`)
traefik.http.routers.dockerservice.tls=true
traefik.http.middlewares.dockerservice-strip.stripprefix.prefixes=/dockerservice
traefik.http.routers.dockerservice.middlewares=add-trailing-slash#file,doc-strip
At https://regex101.com/ the rule seems to work fine for eg https://domain.tld/dockerservice
If the service is up and we navigate to https://domain.tld/dockerservice
it redirects to https://domain.tld/${1}/
The Variable is not expanded. Instead we get the 404-not found error (as expected because a service route with this name does not exists in our traefik setup)
So the trailing-slash is added as desired, but the dockerservice-capture is not expanded.
We've also tried this as a #docker rule on the label_file for the docker-run command but the "error" remains.
We also tried this which we found on the web first (as #file in dyanmic_conf or #docker as label-file for docker run):
traefik.http.middlewares.add-trailing-slash.chain.middlewares=strip-prefix-1,strip-prefix-2
traefik.http.middlewares.strip-prefix-1.redirectregex.regex=^(https?://[^/]+/[a-z0-9_]+)$$
traefik.http.middlewares.strip-prefix-1.redirectregex.replacement=$${1}/
traefik.http.middlewares.strip-prefix-1.redirectregex.permanent=true
traefik.http.middlewares.strip-prefix-2.stripprefixregex.regex=/[a-z0-9_]+
We where trying with ${0} and multiple other attempts where made using double quotes, and single quotes or $-signs.
Our toolchain is as follows:
pushing into the git-repo on the master branach
gitlab-runner executes a .sh file with docker build and docker run command
label-file is provided in the git-repo
We would like to have a generic redirect for all services which have this middleware added
to add a trailing slash if only one Path-Element is added and the trailing slash is missng
So
https://domain.tld/dockerservice should redirect to https://domain.tld/dockerservice/
a Request like https://domain.tld/dockerservice/page should not be changed because
of the strip in the mkdocs container only /page is needed.
At this point we tried a lot and we don't know why traefik is not expanding the variable.
Anyone knows what we are doing wrong?
Best wishes
Exa.Byte
I've finally found a solution which suits well for our purpose:
I just used one $ sign in conjunction with two for the regex option.
added in dynamic.toml for traefik itself:
[http.middlewares.add-trailing-slash.redirectRegex]
regex= "(https?://[^/]+/[a-z0-9_]+)$$"
replacement= "${1}/"
permanent = true
lg
exa.byte
I am relatively new to DBT and I have been reading about env_var and I want to use this in a couple of situations and I am having difficultly and looking for some support.
firstly I am trying to use it in my profiles.yml file to replace the user and password so that this can be set when it is invoked. When trying to test this locally (before implementing this on our AWS side) I am failing to find the right syntax and not finding anything useful online.
I have tried variations of:
dbt run --vars '{DBT_USER: my_username, DBT_PASSWORD=my_password}'
but it is not recognizing and giving nothing useful error wise. When running dbt run by itself it does ask for DBT_USER so it is expecting it, but doesn't detail how
I would also like to use it in my dbt_project.yml for the schema but I assume that this will be similar to the above, just a third variable at the end. Is that the case?
Thanks
var and env_var are two separate features of dbt.
You can use var to access a variable you define in your dbt_project.yml file. The --vars command-line option lets you override the values of these vars at runtime. See the docs for var.
You should use env_var to access environment variables that you set outside of dbt for your system, user, or shell session. Typically you would use environment variables to store secrets like your profile's connection credentials.
To access environment variables in your profiles.yml file, you replace the values for username and password with a call to the env_var macro, as they do in the docs for env_var:
profile:
target: prod
outputs:
prod:
type: postgres
host: 127.0.0.1
# IMPORTANT: Make sure to quote the entire Jinja string here
user: "{{ env_var('DBT_USER') }}"
password: "{{ env_var('DBT_PASSWORD') }}"
....
Then BEFORE you issue the dbt_run command, you need to set the DBT_USER and DBT_PASSWORD environment variables for your system, user, or shell session. This will depend on your OS, but there are lots of good instructions on this. To set a var for your shell session (for Unix OSes), that could look like this:
$ export DBT_USER=my_username
$ export DBT_PASSWORD=abc123
$ dbt run
Note that storing passwords in environment variables isn't necessarily more secure than keeping them in your profiles.yml file, since they're stored in plaintext and not protected from being dumped into logs, etc. (You shouldn't be checking profiles.yml into source control). You should consider at least using an environment variable name prefixed by DBT_ENV_SECRET_ so that dbt keeps them out of logs. See the docs for more info
I have seen some similar questions, but none of them appear to solve my problem. I want to add a user to a docker container and in my Dockerfile, I define the username with:
ARG USERNAME="some_user"
Instead, I want the username to be the current user's computer username, as obtained by running the command whoami in the local terminal.
So what I would like to have is something like
ARG USERNAME=$(whoami)
.
This $(whoami) should be obtained from the local system environment, and not from the docker container.
Is there a way to do this for dockerfiles? I have thought of .env and docker-compose solutions but these also require each user to set their own username according to my knowledge.
There is no integrated way to execute arbitrary commands on the host directly outside of a container using just docker build / docker-compose build.
So to execute an arbitrary command to get/generate the required information you'll need to provide a custom script / use another build system to call docker/docker-compose with the respective flags or maybe generate the .env file from a template / interactively.
If you only need the current user name you may want to use the $USER / $LOGNAME environment variables that are set by the system in many default configurations. But since these are just normal environment variables their values may be incorrect / empty / manually changed by the user, see this question.
So I've been looking around for an example of how I can specify environment variables for my Docker container from the AWS EB web interface. Typically in EB you can add environment properties which are available at runtime. I was using these for my previous deployment before I switched to Docker, but it appears as though Docker has some different rules with regards to how the environment properties are handled, is that correct? According to this article [1], ONLY the AWS credentials and PARAM1-PARAM5 will be present in the environment variables, but no custom properties will be present. That's what it sounds like to me, especially considering the containers that do support custom environment properties say it explicitly, like Python shown here [2]. Does anyone have any experience with this software combination? All I need to specify is a single environment variable that tells me whether the application is in "staging" or "production" mode, then all my environment specific configurations are set up by the application itself.
[1] http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html#command-options-docker
[2] http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html#command-options-python
Custom environment variables are supported with the AWS Elastic Beanstalk Docker container. Looks like a miss in the documentation. You can define custom environment variables for your environment and expect that they will be passed along to the docker container.
I've needed to pass environment variable in moment docker run using Elastic Beanstalk, but, is not allowed put this information in Dockerrun.aws.json.
Below the steps to resolve this scenario:
Create a folder .ebextensions
Create a .config file in the folder
Fill the .config file:
option_settings:
-option_name: VARIABLE_NAME value: VARIABLE_VALUE
Zip the folder .ebextensions file along with the Dockerrun.aws.json plus Dockerfile and upload it to Beanstalk
To see the result, inside EC2 instance, execute the command "docker inspect CONTAINER_ID" and will see the environment variable.
At least for me the environment variables that I set in the EB console were not being populated into the Docker container. I found the following link helpful though: https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
I used a slightly different approach where instead of exporting the vars to the shell, I used the ebextension to create a .env file which I then loaded from Python within my container.
The steps would be as follows:
Create a directory called '.ebextensions' in your app root dir
Create a file in this directory called 'load-env-vars.config'
Enter the following contents:
commands:
setvars:
command: /opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "\(.key)=\"\(.value)\""' > /var/app/current/.env
packages:
yum:
jq: []
This will create a .env file in /var/app/current which is where your code should be within the EB instance
Use a package like python-dotenv to load the .env file or something similar if you aren't using Python. Note that this solution should be generic to any language/framework that you're using within your container.
I don't think the docs are a miss as Rohit Banga's answer suggests. Thought I agree that "you can define custom environment variables for your environment and expect that they will be passed along to the docker container".
The Docker container portion of the docs say, "No DOCKER-SPECIFIC configuration options are provided by Elastic Beanstalk" ... which doesn't necessarily mean that no environment variables are passed to the Docker container.
For example, for the Ruby container the Ruby-specific variables that are always passed are ... RAILS_SKIP_MIGRATIONS, RAILS_SKIP_ASSET_COMPILATION, BUNDLE_WITHOUT, RACK_ENV, RAILS_ENV. And so on. For the Ruby container, the assumption is you are running a Ruby app, hence setting some sensible defaults to make sure they are always available.
On the other hand, for the Docker container it seems it's open. You specify whatever variables you want ... they make no assumptions as to what you are running, Rails (Ruby), Django (Python) etc ... because it could be anything. They don't know before hand what you want to run and that makes it difficult to set sensible defaults.