I am just learning Phoenix and Elixir I am confused, what is the best way to handle environment variables for multiple machines and environments? I keep running into different approaches, from using System.get_env, .env files and mentions of Mix env's. I also keep reading about problems compiling env variables at deployment.
Does anyone have an explanation of how Mix variables, system environment variables and possible .env files or .secret files should be used for local development, stage and production servers?
I have been working mostly in Rails and Python recently so that maybe a helpful contextual piece.
Thanks for the help,
Cory
I personally stick with what Phoenix is giving you by default, e.g. using config files for different environments. Since config files are meant for application configuration, e.g. configuring database adapters, they are checked into source control. By default, these are controlled by the MIX_ENV environment variable. If you look at the bottom of your main config/config.exs file, you will notice this:
import_config "#{Mix.env}.exs"
From what I understand from the documentation, Mix.env is just a shorthand for getting the MIX_ENV value.
Phoenix comes with config files for "develop", "prod" and "test" (all in the config directory) which you can modify for your own use. You can also easily add more configurations — if you want to have a "staging"-specific configuration, for example, just set MIX_ENV=staging on the relevant server, and create config/staging.exs.
For sensitive information, like API keys, environment variables are better, since they're not checked into source control and can be easily changed. You can access those env variables from within your config files or from anywhere within your application.
Hope that helps!
Related
I was reading from this article that you can create a config/local_env.yml with environment variables and then use config/application.rb to read it to replace/take priority before the environment variables that you export in your .bashrc. What then is the .env used for then? Does it serve the same purpose as the config/local_env.yml?
All of these methods are used to feed environment variables to your rails application. So, from an app's point of view, it serves the same purpose whether you export it from .env or .bashrc or config/local_env.yml files.
The differences in these methods are really a matter of personal choice among the team members involved in maintaining the app's development and deployment environments. However, here are few things to consider while opting for either of these choices.
.bashrc - Use this if you manually configure deployment servers and really comfortable with linux/unix command line system administration. This configuration file is specific to bash shell (Bourne Again Shell). You need to configure different file if your server uses different shell (for example: .zshrc if it uses Z Shell)
.env - Use this if you want to keep your app centric environment variables within the app itself while maintaining different variations of environment variables for different runtime environments of your rails app. For example: .env.development and .env.test files with specific values of the environment variables for your development and test environments respectively. This gives you more control of your app's environment variables and do not have to rely on the platform (terminal shell) you want to deploy your app.
config/local_env.yml - This is similar to .env approach, which is provided out of the box by rails gem that allows you to configure environment variables for your app in yml format. This method also keeps your app's configuration within the app irrespective of the shell you are using to run your app.
In addition to the previous answer, another downside of using .bashrc is that it is specific to only one user, so if you're e.g. starting your app server as a systemd service then I believe it won't see your variables.
Meanwhile, .env's Github readme says that it is not the most recommended thing to use outside of the development environment, although it is OK for that purpose.
Another two options to consider are:
Rails secrets. The benefit of this approach is that you get to commit it to git since it's encrypted, so when working in a team you will all have access to the same file. The downside is that it is coupled to the RAILS_ENV variable, so you can't use it to set that e.g. to production on your production app (but you can manually pass it every time it's invoked). Another downside is that if you have a staging environment, then apparently Heroku discourages using RAILS_ENV=staging, which you sometimes really need, so if you need it then you can either do it anyway, or you'll need to set the differing variables via a different mechanism - for example my app has a variable which points to the URL of another part of my app, this URL needs to point to its staging variant on staging, and to its production variant on production, so it needs to differ between my production and staging environment.
Using an /etc/environment file - note that systemd services don't have access to it by default so you'd have to add the line EnvironmentFile=/etc/environment. Also if you're running a shell script from a non-login shell (which happens sometimes), they it won't load them either, but the solution is to just include in your script this: set -a; source /etc/environment; set +a. You should be careful not to commit this to git. If you're working in a team and you need to manage this file then it gets a little complicated since it's not committed to git, but maybe there's a way to have it encrypted. (systemd services note: you might optionally use LoadCredential= for sensitive variables such as private keys, so that other processes won't have access to them; if you do this then you can also commit to git /etc/environment which should now contain non-sensitive variables only).
In my opinion both of those are valid and it's fine to pick whatever is easier to do in your app.
When building my own gem, what are the procs and cons to allow a user to configure it via
a) DSL or ruby code
vs
b) *.yml config file
?
For me, they're about the same.
Using a DSL or Ruby code for configuration means that someone using your library doesn't have to conform to your expectations for how those bits of configuration should be stored and managed. YML files may be common, but so are ENV variables.
If you only offer a YML configuration option, then please consider running the YML file through ERB (like Rails) so that folks can reference ENV variables according to their preference.
I have read about the Twelve Factor App's Config - Section III and searched for a way to do it in NodeJS. It seems that most people recommend dotenv to store configs in environment variables.
However, it seems that dotenv contradicts the Twelve-Factor App, as states:
Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific.
The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.
Making sense of this statement, it seems that using dotenv, you will create a config file .env which will then export them as environment variables.
Wouldn't this violate the Twelve-Factor App, since the .env file can get accidentally checked into the code repo?
12factor is not violated until somebody actually commits and pushes the .env ;)
The .env file can also be stored outside the repo itself, since a library or app is still has to read the .env file and push the variables into the environment. Depending on your implementation, this can be as simple as changing the path from ".env" to "../.env".
Using .env files can be a good compromise to allow developers to manage environments easily, but still be compatible with better environment practices during deployment. I might have 30-40 12factor-flavored apps running in a virtual machine, and having to manage each environment separately is daunting without a "shim" like .env.
The OP asked a well-thought-out question.
I would say the dotenv contradict the 12-factor, section 3, for 2 reasons.
By definition, i.e. this paragraph: "Another approach to config is the use of config files which are not checked into revision control, ... still has weaknesses: it’s easy to mistakenly check in a config file to the repo; ... (therefore the 12-factor app uses a different approach as) stores config in environment variables", now you see, just because an .env file could/should be declared inside a .gitignore, does not make dotenv exempt from that "easy to mistakenly check in a config file to the repo" scrutiny.
An app could otherwise be fully complied with 12-factor, section 3, if it read config from and only from env var. But a dotenv feature is to allow the app to automatically pickup ./.env if it is available. In that sense, the .env file - despite its deceptive name - IS a config file, through and through. Again, this falls into the config approach category that the 12-factor explicitly avoids.
That being said, dotenv is still one of the viable option. Other options include: managing env vars in docker layer; or in a unix user's .*rc file; or in web server config; or in /etc/profile (quoted from this another SO post). dotenv offers one of the most universal file format (fwiw, docker env_file is equally straightforward although their specs are different), however dotenv is the only solution that "polutes" the target app's code base with one more dependency.
Bottom line, dotenv is fine, it is also funny that many dotenv implementation inherit the claim that it started from the 12-factor app tenet, but only few admits that its .env approach deviates from 12-factor app.
Most version control systems have ways of ignoring certain files.
Git has the .gitignore
SVN has "Special Ignores"
As a side note, these techniques are similarly used to ignore the node_modules directory to avoid needlessly copying files.
I have a Rails app that I package as a war file for deploying to Tomcat using Warbler. And it works, but the problem is I don't know how to configure the runtime properties like secret_key_base. I use the standard setup of using secrets.yml, with production variables coming from environment variables. But I don't know how to set the variables while still keeping them out of source control.
Ideally I'd still like to be able to deploy the war file automatically, by just dropping it into the webapps/ directory, but I suppose I could edit the server config file? Or is there a better way of handling this?
either do it the same way as you would in a Rails server ... let it read from ENV (of course you will need to make sure Tomcat has the environment variable set).
alternatively you can set it in a web.xml if you're packaging and than do a $servlet_context.getAttribute('foo') in secrets.yml ... or read it from a file location that only the server's tomcat username can access etc.
sky is the limit here - you basically need to decide what fits your deployments the best.
I've enjoyed using Rails on Heroku, and like that I can adjust the configuration property of a Heroku app without having to commit a change to xyz.yml and redeploy.
It would be nice to completely do away with the Yaml config files in my Rails app, and rely as much as possible on storing configuration in ENV. This goes along with the 12-factor config principle.
However, there are some trade-offs in switching from a Yaml-based configuration management to a Heroku/12-factor-based one.
While it's true that a proliferation of deployments (qa, stage, prod, dev, demo, labs) can lead to a proliferation of Yaml files, it's very easy to copy-paste to create a new configuration profile. I don't see a way to 'copy' configuration profiles from one deployment to another in Heroku.
Storing configuration data in the repo means that, in the case of Heroku, deploying and configuring and application are accomplished in a single operation. If I were to move my configuration out of Yaml files and into ENV variables, I would have to configure my application in a separate step after deployment.
Would like to hear from people who have used 12-factor style configuration in their private applications, and how they have managed lots of configuration variables across lots of deployments.
How do you quickly configure a new deployment?
Where do you keep your authoritative source of configuration variables, if not the repo? How do you distribute it among developers?
Thanks!
What I typically use is Yaml using the ENV and provide defaults. For instance, YAML can be ERB'ed happily to include your ENV vars:
foo:
var: ENV["MY_CONFIG"] || "default_value"
You just need to make sure that you load the Yaml with ERB when you read it:
YAML.load(ERB.new(File.read("#{Rails.root}/config/app_config.yml")).result)
By doing this your code works fine in dev, but also allows you to set config vars in the environment as well.
You can accomplish this relatively easy with some simple shell scripts, iterate existing variables via heroku config or heroku release:info v99, and then set heroku config:set k=v --app
But if its a problem/pain/friction perhaps you have too much inside your env var configuration.
A bit of a late answer, but I believe this is what you are looking for.
I developed a gem called settei, allowing you to use YAML files to configure the app. However the gem will serialize the YAML file as one environment variable during deploy. This way one get the best of both worlds: YAML for ease of management/creating derivative environment, and ENV for 12-factor compliance.