I've enjoyed using Rails on Heroku, and like that I can adjust the configuration property of a Heroku app without having to commit a change to xyz.yml and redeploy.
It would be nice to completely do away with the Yaml config files in my Rails app, and rely as much as possible on storing configuration in ENV. This goes along with the 12-factor config principle.
However, there are some trade-offs in switching from a Yaml-based configuration management to a Heroku/12-factor-based one.
While it's true that a proliferation of deployments (qa, stage, prod, dev, demo, labs) can lead to a proliferation of Yaml files, it's very easy to copy-paste to create a new configuration profile. I don't see a way to 'copy' configuration profiles from one deployment to another in Heroku.
Storing configuration data in the repo means that, in the case of Heroku, deploying and configuring and application are accomplished in a single operation. If I were to move my configuration out of Yaml files and into ENV variables, I would have to configure my application in a separate step after deployment.
Would like to hear from people who have used 12-factor style configuration in their private applications, and how they have managed lots of configuration variables across lots of deployments.
How do you quickly configure a new deployment?
Where do you keep your authoritative source of configuration variables, if not the repo? How do you distribute it among developers?
Thanks!
What I typically use is Yaml using the ENV and provide defaults. For instance, YAML can be ERB'ed happily to include your ENV vars:
foo:
var: ENV["MY_CONFIG"] || "default_value"
You just need to make sure that you load the Yaml with ERB when you read it:
YAML.load(ERB.new(File.read("#{Rails.root}/config/app_config.yml")).result)
By doing this your code works fine in dev, but also allows you to set config vars in the environment as well.
You can accomplish this relatively easy with some simple shell scripts, iterate existing variables via heroku config or heroku release:info v99, and then set heroku config:set k=v --app
But if its a problem/pain/friction perhaps you have too much inside your env var configuration.
A bit of a late answer, but I believe this is what you are looking for.
I developed a gem called settei, allowing you to use YAML files to configure the app. However the gem will serialize the YAML file as one environment variable during deploy. This way one get the best of both worlds: YAML for ease of management/creating derivative environment, and ENV for 12-factor compliance.
Related
I have a Rails project PROJECTX, which is hosted on Heroku. For storing production configs and files, I am using a different repository PROJECTX-config. Is it possible to:
clone PROJECTX-config,
remove current config files, and
symlink config files to PROJECTX-config files
Note that this has to be done on Heroku. Also I am aware that Heroku has options to maintain configs using environment variables, but this is not what I am looking for.
Thanks!
No its not possible.
Each dyno gets its own ephemeral filesystem, with a fresh copy of the
most recently deployed code. During the dyno’s lifetime its running
processes can use the filesystem as a temporary scratchpad, but no
files that are written are visible to processes in any other dyno and
any files written will be discarded the moment the dyno is stopped or
restarted. For example, this occurs any time a dyno is replaced due to
application deployment and approximately once a day as part of normal
dyno management.
- https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
Or at least not without a Rube Goldberg machine like setupe where you setup some kind of automation (like a post-commit hook) to merge repo A and repo B and push the result to heroku.
Also I believe that app config should not be present in environment variables, as it is tedious to maintain rather than maintaining a file.
Heroku does not agree here.
The traditional approach for handling such config vars is to put them
under source - in a properties file of some sort. This is an
error-prone process, and is especially complicated for open source
apps which often have to maintain separate (and private) branches with
app-specific configurations.
A better solution is to use environment
variables, and keep the keys out of the code. On a traditional host or
working locally you can set environment vars in your bashrc file. On
Heroku, you use config vars.
- https://devcenter.heroku.com/articles/config-vars
Although you might be overestimating what you actually need to store in ENV vars. You only need to store secrets such as API keys in ENV.
Other non-secret configuration such your settings for various gems can and should be setup in config/initializers.
If you still think using the GUI is that terrible then use YAML files which you parse and use to set the ENV vars:
require 'yaml'
yaml = YAML.load_file(File.join(__dir__, 'conf.yml'))
def create_key(*components)
components.join('_').upcase
end
env_vars = yaml["production"].each_with_object({}) do |(key,value), memo|
key_components = [key]
if value.kind_of? Hash
value.each_pair do |k,v|
memo[create_key(*key_components.dup.push(k))] = v
end
else
memo[create_key(*key_components)] = value
end
end.each do |k,v|
system("heroku config:set #{k}=#{v}")
puts "Setting #{k} = #{v}; #{ $? }"
end
Or you could even store a serialized form (JSON or YAML) in a single env var - there is a total size limit of 32kb though.
I was reading from this article that you can create a config/local_env.yml with environment variables and then use config/application.rb to read it to replace/take priority before the environment variables that you export in your .bashrc. What then is the .env used for then? Does it serve the same purpose as the config/local_env.yml?
All of these methods are used to feed environment variables to your rails application. So, from an app's point of view, it serves the same purpose whether you export it from .env or .bashrc or config/local_env.yml files.
The differences in these methods are really a matter of personal choice among the team members involved in maintaining the app's development and deployment environments. However, here are few things to consider while opting for either of these choices.
.bashrc - Use this if you manually configure deployment servers and really comfortable with linux/unix command line system administration. This configuration file is specific to bash shell (Bourne Again Shell). You need to configure different file if your server uses different shell (for example: .zshrc if it uses Z Shell)
.env - Use this if you want to keep your app centric environment variables within the app itself while maintaining different variations of environment variables for different runtime environments of your rails app. For example: .env.development and .env.test files with specific values of the environment variables for your development and test environments respectively. This gives you more control of your app's environment variables and do not have to rely on the platform (terminal shell) you want to deploy your app.
config/local_env.yml - This is similar to .env approach, which is provided out of the box by rails gem that allows you to configure environment variables for your app in yml format. This method also keeps your app's configuration within the app irrespective of the shell you are using to run your app.
In addition to the previous answer, another downside of using .bashrc is that it is specific to only one user, so if you're e.g. starting your app server as a systemd service then I believe it won't see your variables.
Meanwhile, .env's Github readme says that it is not the most recommended thing to use outside of the development environment, although it is OK for that purpose.
Another two options to consider are:
Rails secrets. The benefit of this approach is that you get to commit it to git since it's encrypted, so when working in a team you will all have access to the same file. The downside is that it is coupled to the RAILS_ENV variable, so you can't use it to set that e.g. to production on your production app (but you can manually pass it every time it's invoked). Another downside is that if you have a staging environment, then apparently Heroku discourages using RAILS_ENV=staging, which you sometimes really need, so if you need it then you can either do it anyway, or you'll need to set the differing variables via a different mechanism - for example my app has a variable which points to the URL of another part of my app, this URL needs to point to its staging variant on staging, and to its production variant on production, so it needs to differ between my production and staging environment.
Using an /etc/environment file - note that systemd services don't have access to it by default so you'd have to add the line EnvironmentFile=/etc/environment. Also if you're running a shell script from a non-login shell (which happens sometimes), they it won't load them either, but the solution is to just include in your script this: set -a; source /etc/environment; set +a. You should be careful not to commit this to git. If you're working in a team and you need to manage this file then it gets a little complicated since it's not committed to git, but maybe there's a way to have it encrypted. (systemd services note: you might optionally use LoadCredential= for sensitive variables such as private keys, so that other processes won't have access to them; if you do this then you can also commit to git /etc/environment which should now contain non-sensitive variables only).
In my opinion both of those are valid and it's fine to pick whatever is easier to do in your app.
I am just learning Phoenix and Elixir I am confused, what is the best way to handle environment variables for multiple machines and environments? I keep running into different approaches, from using System.get_env, .env files and mentions of Mix env's. I also keep reading about problems compiling env variables at deployment.
Does anyone have an explanation of how Mix variables, system environment variables and possible .env files or .secret files should be used for local development, stage and production servers?
I have been working mostly in Rails and Python recently so that maybe a helpful contextual piece.
Thanks for the help,
Cory
I personally stick with what Phoenix is giving you by default, e.g. using config files for different environments. Since config files are meant for application configuration, e.g. configuring database adapters, they are checked into source control. By default, these are controlled by the MIX_ENV environment variable. If you look at the bottom of your main config/config.exs file, you will notice this:
import_config "#{Mix.env}.exs"
From what I understand from the documentation, Mix.env is just a shorthand for getting the MIX_ENV value.
Phoenix comes with config files for "develop", "prod" and "test" (all in the config directory) which you can modify for your own use. You can also easily add more configurations — if you want to have a "staging"-specific configuration, for example, just set MIX_ENV=staging on the relevant server, and create config/staging.exs.
For sensitive information, like API keys, environment variables are better, since they're not checked into source control and can be easily changed. You can access those env variables from within your config files or from anywhere within your application.
Hope that helps!
With the release of the secrets.yml file, I removed my reliance on Figaro and moved all of my keys to secrets.yml and added that file to .gitignore.
But when I tried to push to Heroku, Heroku said they needed that file in my repo in order to deploy the website. which makes sense, but I don't want my keys in git if I can avoid it.
With Figaro, I would run a rake task to deploy the keys to heroku as env variables and keep application.yml in the .gitignore. Obviously, I can't do that any more. So how do I handle this?
Secrets isn't a full solution to the environment variables problem and it's not a direct replacement for something like Figaro. Think of Secrets as an extra interface you're now supposed to use between your app and the broader world of environment variables. That's why you're now supposed to call variables by using Rails.application.secrets.your_variable instead of ENV["your_variable"].
The secrets.yml file itself is that interface and it's not meant to contain actual secrets (it's not well named). You can see this because, even in the examples from the documentation, Secrets imports environment variables for any sensitive values (e.g. the SECRET_KEY_BASE value) and it's automatically checked into source control.
So rather than trying to hack Secrets into some sort of full-flow environment variable management solution, go with the flow:
Pull anything sensitive out of secrets.yml.
Check secrets.yml into source control like they default you to.
For all sensitive values, import them from normal environment variables into secrets ERB (e.g. some_var: <%= ENV["some_var"] %>)
Manage those ENV vars as you normally would, for instance using the Figaro gem.
Send the ENV vars up to Heroku as you normally would, for instance using the Figaro gem's rake task.
The point is, it doesn't matter how you manage your ENV vars -- whether it's manually, using Figaro, a .env file, whatever... secrets.yml is just an interface that translates these ENV vars into your Rails app.
Though it adds an extra step of abstraction and some additional work, there are advantages to using this interface approach.
Whether you believe it's conceptually a good idea or not to use Secrets, it'll save you a LOT of headache to just go with the flow on this one.
PS. If you do choose to hack it, be careful with the heroku_secrets gem. As of this writing, it runs as a before_initialize in the startup sequence so your ENV vars will NOT be available to any config files in your config/environments/ directory (which is where you commonly would put them for things like Amazon S3 keys).
An equivalent for secrets.yml of that Figaro task is provided by the heroku_secrets gem, from https://github.com/alexpeattie/heroku_secrets:
gem 'heroku_secrets', github: 'alexpeattie/heroku_secrets'
This lets you run
rake heroku:secrets RAILS_ENV=production
to make the contents of secrets.yml available to heroku as environment variables.
see this link for heroku settings
if u want to run on local use like this
KEY=xyz OTHER_KEY=123 rails s
The Heroku docs suggest one way of managing config vars: http://devcenter.heroku.com/articles/config-vars
I guess this works well for API keys, but seems clunky for other environment config info.
Are there any recommendations or better ways of doing this?
Thanks
I haven't ever found it clunky - but use heroku config vars for API keys and stuff which I don't want committed into a git repo and the relevant environment .rb file for application config stuff specific to each environment.
You can always https://github.com/fastestforward/heroku_san which lets you set config vars in a yml file which can be run against an app to set the vars without having to do it via the CLI
figaro is another gem which help us to manage this stuff.