Rails secrets.yml VS Dotenv VS Figaro with Capistrano on AWS - ruby-on-rails

There are several posts ans Stack Overflow questions about how to manage API tokens on the web, but I see a lot of people repeat what they read somewhere else, often with contradictions...
How do you deal with API Tokens, secrets and the like ?
Here's what I have read so far using 3 different gems
secrets.yml
Introduced with Rails 4.1, then updated to encrypted secrets around rails 5
When initially released on rails 4, they were (or were not ?) meant to be pushed on repositories. However I often saw examples using environment variables for production
# config/secrets.yml
development:
secret_key_base: super_long_secret_key_for_development
...
production:
secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>
...
And at this point someone asked "Why do we use ENV for production ?". A legit question back then, often answered "We don't want production token to be hard coded in the application" (hence how it is not clear anymore if the secrets should have been committed or not).
Later, with Rails 5, secrets became encrypted with a master key, so the encrypted secrets.yml file could be committed to the repository, but then the problem remained the same with the master key used to read the secrets.
PROs:
Rails convention.
Easy deploy with capistrano-secrets gem and cap [stage] setup (and it only deploys stage secrets nice) or similar gems
YML data structure (array/hash ok) + can use Ruby code via ERB
With encrypted secrets since rails 5, easy to collaborate/share the secrets
CONs:
Need to use Rails.application.secrets.xxx
Many services like AWS still read from ENV to automatically setup their gems/services
Is not the 12 factors way (or is it ?)
Quite new, so not really used yet ?
If using encrypted secrets, need to manage the master key
Bkeepers dotenv
Simply defining a .env file at the root that is read by Rails on startup
Plugins like capistrano-env allow to easily inject environment specific ENV on servers, and secrets can still must be managed using .env.staging, .env.production
PROs
ENV is in 12factor rules
3.5k stars... maybe not for nothing ?
the dotenv approach is now available on almost all other languages (JS, Go, etc)
Recent versions allow reusing variables (API_ROOT=X, SOME_ENDPOINT=${X}/endpoint)
CONs
No Ruby interpolation (unless extra code is added to parse the .env with the ERB templating engine for example)
limited to string-string key/val
Figaro
Some sort of hybrid secrets/ENV. With 12factors/Rails/Heroku in mind, but in the end doesn't seem better than the rest...
From the above and the rest I didn't write, it would seem like secrets.yml would be a great winner if those secrets were put in ENV instead (and tbh I feel lazy about writing Rails.Application.secrets each time).
So, suppose I start a quite new Rails app, and also based on your experience. Which one would you choose ?
(My particular stack uses AWS + Capistrano, no Heroku)

I personally think that the "right" approach depends on your environment.
In my case, I have servers which are managed by IT and I don't have access to the vhost or anything else to easily set environment variables. Because of this, I commit a secrets.yml file which doesn't contain the production stanza, then set up Capistrano to add this file to shared_files. On each server, I add the file.
If I had easy access the the vhost, and I was managing the server vhosts with Puppet, Chef, Ansible, or similar, I would use the environment variable approach like the default secrets.yml file.
Your case with AWS appears to be the latter. Ultimately, either option is fine. There is little downside to committing the secrets.yml file without a production stanza.

First, all three methods can be 12-factor compatible. It is compatible if you pass the config value by ENV variable, instead of copying one file to the server first.
My thoughts are each of these solutions:
Rails secrets
Developers are forced to go 12-factor, either manually set it on production server, or have another file on local machine and then pass it as ENV every time during deploy. (Didn't know about capistrano-secrets, it probably handles this)
(I think what you said in CON #2 and #3 are the opposite to secret.yml solution)
The accessor is also quite long as you mentioned.
dotenv
It does not encourage 12-factor, and was not originally designed for production env anyways. Either you write code to pass its value as ENV to production during deploy (making it 12 factor compatible), or you copy your .env.production file to the production server.
Also it forces you to use the flat key:value configuration. No nesting.
Figaro
Though it uses YAML, it does not allow nested hash.
It is 12 factor compatible, e.g. it includes code to transfer the config to heroku.
My Solution
I wrote a gem, in which settings are stored in gitignored YAML file. Nesting is allowed. When accessing some value, do Setting.dig(:fb,:api).
It provides mechanism for 12-factor app deploy, by serializing the whole YAML file into a string and pass it to production as ENV.
I no longer have to distinguish between secret config and non-secret config. They are in one place and secret by default. I get benefit of 12-factor app while using easy to namespace YAML files.

Related

What is the correct way to use config/credentials in Rails 6? And how do I stop my app from using /tmp/development_secret?

I'm working on a Rails application and I am attempting to organize my secret_key_base and related important secrets (Think API keys, database credentials, etc.)
I am trying to find a way to setup something like the following, under /config
/config
credentials.yml.enc
master.key
/config/credentials
development.yml.enc
development.key
production.yml.enc
production.key
test.yml.enc
test.key
First Question:
Is it that secret_key_base exists in /config/credentials.yml.enc, which is loaded (first?) and then the credentials are loaded for the environment rails is running in? Or should I create a different secret_key_base for each environment?
Second Question:
No matter what I do, when I run in development or test, tmp/development_secret loads first. In addition, whenever I try to access my secrets in development, (Rails.application.secret_key_base) as referenced here: What's the correct way of defining secret_key_base on Rails 6, I run into an issue where I only ever receive nil when looking for secrets I've defined in my development.yml.enc, which I assume is because it's not loading anything in that file, it's going to tmp/development_secret and not finding anything else (Maybe I'm wrong.)
Goals:
Stop tmp/development_secret from being created, and instead access
secrets using the specific .yml.enc file depending on the
environment.
Understand why /config/credentials.yml.enc exists if it doesn't
load in all the environments. If it doesn't, then it isn't clear when it loads.
Why?
Using config/database.yml as an example, I want to store different creds for each environment, but none of them in version control. (I want nobody but a few to have production.) However, I want to access them the exact same way in all of my environments. Not having creds load in production because of an issue with a .yml file will crash my app, so I don't want to load them differently in test/development.
Put together a blog post about this because searching for documentation on this feature is painful. It's a really easy thing because then only the master.key, and production.key would need loaded as ENV variables which is great. (Or possibility just one of them.)
This really should be a simple, out-of-the-box thing, but it's hard to find real documentation on best practices.
I've found the answer, at least the one I'm looking for. You can have your initializer load whatever file you want by overriding the defaults.
config.credentials.content_path = 'config/credentials/development.yml.enc'
config.credentials.key_path = 'config/credentials/development.key'
https://edgeapi.rubyonrails.org/classes/Rails/Application.html

Setting Test environment variables in rails without putting in source code

I'm using Twilio for an app and on production I set the auth token using heroku's CLI. I'm using sms-spec (https://github.com/monfresh/sms-spec) to test my app's Twilio integration locally. I want to set ENV['TWILIO_AUTH_TOKEN'] to my token in the test environment.
I use guard to auto-run my tests whenever I make changes so I don't want to have to manually set the ENV variable each time I run tests. I also don't want to put the token in my source code for security reasons.
Is there a way I can set the ENV variable for my local test environment such that it is permanent and not in my source? I've spent a few hours researching this and can't seem to find a good explanation of how to do this. Any help is much appreciated :)
Two approaches:
Use a gem like Dotenv (link). This is the approach I use in most of my applications for development. Simply include the gem in your gemfile, bundle install and then store any environment variable settings in a top level file called .env. Restart your rails server and ENV will be automatically loaded. Very easy to use and convenient.
If you are flexible on the ENV part, and you are running Rails 4.1+, you can use config/secrets/yml. This is documented very well in the Rails 4.1 release notes, Section 2.2. So, in your case, you would set it up like so:
development:
twilio_auth_token: verysecretstring
Then, in your initializer, instead of referencing ENV['TWILIO_AUTH_TOKEN'], you would use Rails.application.secrets.twilio_auth_token. I haven't tried this myself, but it is on my list as I would rather use native Rails functionality than a separate gem.
Of course, any files which contain your secrets needs to be safeguarded carefully. At a minimum, make sure you include in .gitignore so that your secrets do not find their way into your code respository.

Environment variables for Rails

I am trying a number of different applications into my rails project. For security reasons, I am storing any sensitive keys as environment variables.
This is easy to do with Heroku but on the local environment side I find my windows environment variables starting to pile up. If I happen to have two projects with facebook authentication now I have to name them uniquely on my computer not to get mixed up with each others, which then means I have to rename them in my rails projects, which then means I need to rename them in Heroku... AH
Is there an easier way of doing this such as a configuration file that is added to gitignore, or is that still not quite safe?
What's the best practice for this?
Rails 4.1 comes with secrets.yml, which is where you would put these. Please see this section of the Rails 4.1 release notes for more info.

Where to store sensitive data in public rails app?

My personal rails project uses a few API's for which I store the API keys/secrets in config/environments/production.yml and development.yml as global variables. I now want to push this project to github for others to use, but I don't want them to have those bits of sensitive data. I also don't want this file in .gitignore because it's required for the app to run. I've considered putting them in the DB somewhere, but am hoping to find a better solution.
TLDR: Use environment variables!
I think #Bryce's comment offers an answer, which I'll just flush out. It seems one approach Heroku recommends is to use environment variables to store sensitive information (API key strings, database passwords). So survey your code and see in which you have sensitive data. Then create environment variables (in your .bashrc file for example) that store the sensivite data values. For example for your database:
export MYAPP_DEV_DB_DATABASE=myapp_dev
export MYAPP_DEV_DB_USER=username
export MYAPP_DEV_DB_PW=secret
Now, in your local box, you just refer to the environment variables whenever you need the sensitive data. For example in database.yml :
development:
adapter: mysql2
encoding: utf8
reconnect: false
database: <%= ENV["MYAPP_DEV_DB_DATABASE"] %>
pool: 5
username: <%= ENV["MYAPP_DEV_DB_USER"] %>
password: <%= ENV["MYAPP_DEV_DB_PW"] %>
socket: /var/run/mysqld/mysqld.sock
I think database.yml gets parsed just at the app's initialization or restart so this shouldn't impact performance. So this would solve it for your local development and for making your repository public. Stripped of sensitive data, you can now use the same repository for the public as you do privately. It also solves the problem if you are on a VPS. Just ssh to it and set up the environment variables on your production host as you did in your development box.
Meanwhile, if your production setup involves a hands off deployment where you can't ssh to the production server, like Heroku's does, you need to look at how to remotely set up environment variables. For Heroku this is done with heroku config:add. So, per the same article, if you had S3 integrated into your app and you had the sensitive data coming in from the environment variables:
AWS::S3::Base.establish_connection!(
:access_key_id => ENV['S3_KEY'],
:secret_access_key => ENV['S3_SECRET']
)
Just have Heroku create environment variables for it:
heroku config:add S3_KEY=8N022N81 S3_SECRET=9s83159d3+583493190
Another pro of this solution is that it's language neutral, not just Rails. Works for any app since they can all acquire the environment variables.
How about this...
Create a new project and check it into GitHub with placeholder values in the production.yml and development.yml files.
Update .gitignore to include production.yml and development.yml.
Replace the placeholder values with your secrets.
Now you can check your code into GitHub without compromising your secrets.
And anyone can clone your repo without any extra steps to create missing files (they'll just replace the placeholder values as you did).
Does that meet your goals?
They're probably best put in initializers (config/initializers/api.yaml) though I think what you've got cooked up is fine. Add the actual keys to your .gitignore file and run git rm config/environments/production.yml to remove that sensitive data from your repo. Fair warning, it will remove that file too so back it up first.
Then, just create a config/environments/production.yml.example file next to your actual file with the pertinent details but with the sensitive data left out. When you pull it out to production, just copy the file without the .example and substitute the appropriate data.
Use environment variables.
In Ruby, they're accessible like so:
ENV['S3_SECRET']
Two reasons:
The values will not make it into source control.
"sensitive data" aka passwords tend to change on a per-environment basis anyways. e.g. you should be using different S3 credentials for development vs production.
Is this a best practice?
Yes: http://12factor.net/config
How do I use them locally?
foreman and dotenv are both easy. Or, edit your shell.
How do I use them in production?
Largely, it depends. But for Rails, dotenv is an easy win.
What about platform-as-a-service?
Any PaaS should give you a way to set them. Heroku for example: https://devcenter.heroku.com/articles/config-vars
Doesn't this make it more complicated to set up a new developer for the project?
Perhaps, but it's worth it. You can always check a .env.sample file into source control with some example data in it. Add a note about it to your project's readme.
Rails 4.1 has now a convention for it. You store this stuff in secrets.yml. So you don't end up with some global ENV calls scattered across Your app.
This yaml file is like database.yml erb parsed, so you are still able to use ENV calls here. In that case you can put it under version control, it would then serve just as a documentation which ENV vars has to be used. But you also can exlcude it from version control and store the actual secrets there. In that case you would put some secrets.yml.default or the like into the public repo for documentation purposes.
development:
s3_secret: 'foo'
production:
s3_secret: <%= ENV['S3_SECRET']%>
Than you can access this stuff under
Rails.application.secrets.s3_secret
Its discussed in detail at the beginning of this Episode

How can I manage a different setup for my Ruby on Rails application dependent on production or development?

I am trying to deploy my first rails app and struggling a bit. My plan is to initially host it on a heroku free account to get a feel for live deployments and do some production testing. Eventually I might move it to a VPS.
I use git and do not use Capistrano at the moment.
Heroku primarily uses git, which is fine, but git manages the entire project state not files. So I have issues managing configuration files that are different from production to development, for example captcha keys in the environment.rb or goolge js api keys.
So what I did was to..
1 - Take the environment specific configuration out of the enviornment.rb and put it in the development.rb and production.rb. Created a branch called dev where I do my development and then merge it with master and push master to the production heroku remote.
This all works ok, but wondering if there is a better way to do it.
The other massive problem is I might have to use different gems in dev and in herouku. For example, I use ThinkingSphinix for search in dev, but Heroku I have to use acts_as_solr, which means my "Article.search call in the controller, will have to be Article.find_by_solr in production. This can become messy very fast.
What's the best way to deal with this kind of situation?
Thanks
For non-sensitive keys such as Google's JS API key etc., I found this RailsCasts episode very helpful.
Just created a config file under config/ and store your development settings in there.
# /config/google.yml
development:
google:
js:
key: 123456
test:
google:
js:
key: 345678
production:
google:
js:
key: 567890
Then create an initializer inside config/initializers/ that will parse the yaml and create an object which can be used without worrying about the current environment.
# /config/initializers/google.rb
GOOGLE_CONFIG = YAML.load_file("#{RAILS_ROOT}/config/google.yml")[RAILS_ENV]
The environment variable RAILS_ENV refers to the current environment, so on application startup it picks up the current type, and you can refer to the settings in your code through GOOGLE_CONFIG:
<script type="text/javascript" src="http://www.google.com/jsapi?key=<%= GOOGLE_CONFIG['js']['key'] %>"></script>
For the latter issue, where code itself differs from environment to environment, I believe Capistrano would be a better solution.
For values that you want to keep different between environments, Heroku offers config vars.
As for using one indexing program in production and another in development, that's a bad idea, and will make things way messier than they need to be. Either start using Solr locally, or set up a Thinking Sphinx instance in EC2 yourself, and have your dynos connect to it.
I would suggest that it is very unwise to have different code in development and production.
Your development, test and production environments should be as similar as possible.
In fact, I would go so far as to say the entire point of the different environments is to simply provide an easy system for allowing minor configuration changes between setups. Different databases, different API parameters, different aching options, but the core system MUST be the same.
The the issue you will face is doubling your development effort. You still have to write the code. So in the search example you provide above, you will have to develop and test twice - once for the Solr production system and once for you local Sphinx, then you need to be able to switch and test between the two approaches in all of your environments to ensure test coverage and functionality.

Resources