What should be removed from public source control in Ruby on Rails? - ruby-on-rails

I've been searching the web, and I can't find any good/recent examples of what to exclude from a new public rails app. I'm looking to open source my app on GitHub and am wondering what types of data should be removed from source control.
From what I can tell, there should be a config/config.yml file that has private information. I've been looking through the other files, and it looks like config/database.yml, config/intializers/secret_token.rb and config/initializers/session_store.rb should also be excluded?
Is it best practice to exclude all of these files separately? Or is there a way to have the info all defined in config/config.yml and be called in each of those files? Additionally, what files and data should be kept private and hidden? Is that all of them?
I'm just wondering what approach I should take and what is the best practice. Thanks for any help!

I've been looking into this recently as well; I wanted to keep sensitive information hidden throughout the process of pushing open source code to Github, then automatically pushed to Travis CI for testing, then from Travis being automatically deployed to Heroku. Here are all the details of what I've found so far looking at various StackOverflow Q&As, blogs etc, which will hopefully serve as a reference for you, even if only for config inside the Rails app (omit any {{ ... }} you see)
Disclaimer: I'm by no means an expert here, so please keep in mind there are likely better ways to do this than what I'm trying. I'd love to be able to learn some new tricks in this Q&A thread.
Inside the Rails App
I currently use the Figaro gem to hide sensitive information in ENV environment variables. In my (.gitignored) config/application.yml, I keep the following information:
# App keys
SECRET_TOKEN: # your rake secret generated token
development:
DB_NAME: # your dev db name here
DB_USER: # your dev db username here
DB_PASSWORD: # your dev db password here
test:
DB_NAME: # your test db name here
DB_USER: # your test db username here
DB_PASSWORD: # your test db password here
production:
DB_NAME: # your prod db name here
DB_USER: # your prod db username here
DB_PASSWORD: # your prod db password here
# Third Party keys that you will reference in their relevant files
THIRD_PARTY_API_OR_LICENSE_KEY: # list of whatever api/license keys you use
(DB_NAME, DB_USER, and DB_PASSWORD will be used dynamically depending on what environment your app is running in).
An empty version of the above file (config/application.example.yml) gets pushed to Github with some instructions on how to fill it in.
The files that are pushed to Github and reference these variables look like this:
config/database.yml
(Postgresql is used here, but you should be able to change the settings for whatever database you use)
postgresql: &postgresql
adapter: postgresql
database: <%= ENV['DB_NAME'] %>
username: <%= ENV['DB_USER'] %>
password: <%= ENV['DB_PASSWORD'] %>
min_messages: ERROR
defaults: &defaults
pool: 5
timeout: 5000
host: localhost
<<: *<%= ENV['DB'] || "postgresql" %>
development:
<<: *defaults
test:
<<: *defaults
production:
<<: *defaults
config/initializers/secret_token.rb
if Rails.env.production? && ENV['SECRET_TOKEN'].blank?
raise 'SECRET_TOKEN environment variable must be set!'
end
YourApp::Application.config.secret_token =
ENV['SECRET_TOKEN'] || {{WHATEVER_SECRET_TOKEN_RAILS_GENERATED_BY_DEFAULT}}
(Plus, whatever files would be referencing THIRD_PARTY_API_OR_LICENSE_KEY-type keys.)
Testing on Travis CI
Create encrypted travis variables using the Travis gem. The Heroku API key and Heroku Git URL are needed if you deploy direct to Heroku from a Travis worker (see this StackOverflow Q&A for details), otherwise you can omit them if you just use it for testing:
$ gem install travis
$ travis encrypt your_username/your_repo HEROKU_API_KEY={{YOUR_HEROKU_API_KEY}}
$ travis encrypt HEROKU_GIT_URL={{YOUR_HEROKU_GIT_URL}} # eg git#heroku.com:your_app.git
$ travis encrypt DB_NAME={{YOUR_DB_NAME_UNDER_TEST}} # eg your_app_test
$ travis encrypt DB_USER={{YOUR_DB_USER_UNDER_TEST}}
$ travis encrypt DB_PASSWORD={{YOUR_DB_PASSWORD_UNDER_TEST}}
(Plus, encrypt any other keys you may need during testing, if any...)
Then add them to .travis.yml
(once again Postgresql-focused, but you should be able to change the settings for whatever database you use)
env:
global:
- secure: {{YOUR_ENCRYPTED_HEROKU_API_KEY}}
- secure: {{YOUR_ENCRYPTED_HEROKU_GIT_URL}}
- secure: {{YOUR_ENCRYPTED_DB_NAME}}
- secure: {{YOUR_ENCRYPTED_DB_USER}}
- secure: {{YOUR_ENCRYPTED_DB_PASSWORD}}
matrix:
- DB: postgresql
before_script:
- psql -c "create database $DB_NAME;" -U $DB_USER
- RAILS_ENV=test bundle exec rake db:migrate
script:
- bundle exec rspec spec/
after_success:
- gem install heroku
- git remote add heroku $HEROKU_GIT_URL
# ... see link above for the rest of the config content
Multiple variables marked with the same name of secure are fine; they'll show up in the config as HEROKU_API_KEY=[secure] HEROKU_GIT_URL=[secure] etc.
Deployment to Heroku
Use the Figaro's Heroku rake task to automatically set the environment variables that Heroku needs to see in production:
$ rake figaro:heroku
Or, set them manually:
$ heroku config:set SECRET_TOKEN={{YOUR_SECRET_TOKEN}}
$ heroku config:set DB_NAME={{YOUR_DB_NAME_UNDER_PRODUCTION}} # eg your_app_production
$ heroku config:set DB_USER={{YOUR_DB_USER_UNDER_PRODUCTION}}
$ heroku config:set DB_PASSWORD={{YOUR_DB_PASSWORD_UNDER_PRODUCTION}}
$ heroku config:set THIRD_PARTY_API_OR_LICENSE_KEY={{YOUR_THIRD_PARTY_API_OR_LICENSE_KEY}}
Then, attempt a deployment.
That's all I have for now. Not sure at the moment if I should be hiding more info or if I'm not hiding it well enough, but it's a work in progress.

You'll get different opinions. IMHO, it's best-practice to include these files, but omit the secret content from them. Document what you're doing so developers who are new to your project know what they need to fill in.
Phusion has a good blog post about how to handle the Rails session secret, and the tradeoffs you can make to include or exclude information:
http://blog.phusion.nl/2013/01/04/securing-the-rails-session-secret/#.URYPXekTMak
My favorite way to document these is using a "rake setup" task. You can have the task print what the developer needs to do-- in other words, you don't need to automate it all (though that's nice if you're able to do it).
If you want to get fancy, have your files read the secret settings from a shared/ directory, which also enables deployment symlinking. This is described in the Phusion blog too. This is how I build apps that need to be deployed frequently.

Related

Rails: Paypal configuration file and figaro environment variables

I´m using the gem 'paypal-sdk-adaptivepayments' to integrate Paypal in my Rails app. The configuration file is paypal.yml:
development:
# Credentials for Classic APIs
username: ENV["PAYPAL_CLASSIC_USERNAME_DEV"]
password: ENV["PAYPAL_CLASSIC_PASSWORD_DEV"]
signature: ENV["PAYPAL_CLASSIC_SIGNATURE_DEV"]
app_id: ENV["PAYPAL_CLASSIC_APP_ID_DEV"]
http_timeout: 30
# Mode can be 'live' or 'sandbox'
mode: sandbox
test:
<<: *default
production:
<<: *default
#mode: live
Cause this information is secret I want to use another gem called 'Figaro' that externalize this variables. I used this for another configuration files in my app but it doesn´t works with 'paypal.yml'. I know that doesn´t works because when I put the real information in the paypal.yml file it works.
development:
# Credentials for Classic APIs
username: *******#yahoo.com
password: *******
signature: ******
app_id: ******
http_timeout: 30
# Mode can be 'live' or 'sandbox'
mode: sandbox
test:
<<: *default
production:
<<: *default
#mode: live
Has anyone used Figaro with this file? Is there any other option to "secretize" this information in Rails?
Thanks in advance!
Okay, what you should do is replace your yaml config file with a config.rb file:
#config/initializers/paypal.rb
PayPal::SDK.configure(
username: ENV["PAYPAL_CLASSIC_USERNAME_DEV"],
password: ENV["PAYPAL_CLASSIC_PASSWORD_DEV"],
signature: ENV["PAYPAL_CLASSIC_SIGNATURE_DEV"],
app_id: ENV["PAYPAL_CLASSIC_APP_ID_DEV"],
http_timeout: 30
)
Then define these variables in a yaml file such as application.yml:
#config/application.yml
PAYPAL_CLASSIC_USERNAME_DEV: yourpaypalusername
PAYPAL_CLASSIC_PASSWORD_DEV: yourpaypalpassword
PAYPAL_CLASSIC_SIGNATURE_DEV: yourpaypaysignature
PAYPAL_CLASSIC_APP_ID_DEV: yourappid
Lastly assuming you are using git for your version control, go into your gitignore and tell it to ignore the yaml file with your secret info:
#.git_ignore
/config/application.yml
This will properly keep your environment variables out of your code base when you push it to github or other git repo.
Edit: To use different environment keys you merely specify the keys for different environments in your yaml file:
#application.yml
production:
PAYPAL_CLASSIC_USERNAME_DEV: yourpaypalusername
PAYPAL_CLASSIC_PASSWORD_DEV: yourpaypalpassword
PAYPAL_CLASSIC_SIGNATURE_DEV: yourpaypaysignature
PAYPAL_CLASSIC_APP_ID_DEV: yourappid
That being said, I highly suspect what you are doing is a violation of 12factor best practices. This yaml file is not a part of your application - when you push your code up to a server like Heroku, this file is not supposed to go with it. The yaml file with your keys should only ever exist on your local machine. It should not exist in Github, nor on your production server. To read more about this, check out this link: http://12factor.net/config
So how do you tell a server like Heroku what these config variables are? This will vary based on what service provider you use, but is typically done via the command line tools or through an admin management page. e.g., for Heroku you would run:
heroku config:set PAYPAL_CLASSIC_USERNAME_DEV=putyourusernamehere

AWS::S3::Errors::InvalidAccessKeyId

So, I'm running rake db:drop db:create db:migrate db:seed and for some reason I'm getting the following error:
AWS::S3::Errors::InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
Bare in mind, most of the migrations run however, I seem to be getting the error directly after this migration:
20140606122523 CreateActiveAdminComments: migrated (0.8930s) ===============
I have all of the updated keys in the appropriate files as follows:
config/aws.yml:
defaults: &defaults
access_key_id: '*key here*'
secret_access_key: '*key here*'
bucket_name: '*bucket name*'
development:
<<: *defaults
test:
<<: *defaults
production:
<<: *defaults
initializers/aws.rb
AWS.config(YAML.load_file(Rails.root.join("config/aws.yml"))[Rails.env])
What am I doing wrong? I've double and triple checked the keys, and have even tried generating new ones.. Still getting the same error.
Please help!
Migration
After a discussion in the comments, the issue was that the migration referenced Paperclip, which used S3 settings. To fix the migration, the OP had to remove the Paperclip S3 settings from the config/application.rb, allowing the migration to proceed.
--
ENV Vars
The ENV vars are slightly different
It's strange how you'd try and set them in the application.rb - by their nature, environment (ENV) variables are meant to reside in the OS - making them accessible by your app & other parts of the system. It's just another security feature to help keep data both secure & modular
The problem you have is setting ENV vars in development is about putting the variables on the machine, and it's also the case that Rake won't load them unless they are available at OS-level
The solution you need is two-fold:
--
secrets.yml
Rails 4.1.0 introduced secrets.yml to help fix this:
Rails 4.1 generates a new secrets.yml file in the config folder. By
default, this file contains the application's secret_key_base, but it
could also be used to store other secrets such as access keys for
external APIs.
#config/secrets.yml
If you upgrade to Rails 4.1, you'll get a `secrets.yml` file which will look like this:
# Be sure to restart your server when you modify this file.
# Your secret key is used for verifying the integrity of signed cookies.
# If you change this key, all old signed cookies will become invalid!
# Make sure the secret is at least 30 characters and all random,
# no regular words or you'll be exposed to dictionary attacks.
# You can use `rake secret` to generate a secure secret key.
# Make sure the secrets in this file are kept private
# if you're sharing your code publicly.
development:
secret_key_base: [a salt]
# Do not keep production secrets in the repository,
# instead read values from the environment.
production:
secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>
You'll be able to put your API keys in there, and then reference using Rails.application.secrets.some_api_key
--
figaro
An alternative, which is just as effective, is to use the figaro gem. This is kind of redundant with secrets.yml but anyway, it creates an application.yml which you can then reference using the direct ENV["setting"] var
Either way should solve your issue!

Securely providing the database password in a Rails app

As you know, you MUST provide the correct database name, username, and password for the database in the config/database.yml file, or your Rails app will refuse to work.
In the default setup, your password is in plain text in the config/database.yml file. If your app is on a free GitHub repository, then your password is public information. This is not a viable option for a serious app. (It's OK for a tutorial exercise, provided that you don't use this password for anything else.)
I have a solution that has worked for me so far, but I'm wondering if there is something better. You can see my deployed example at https://github.com/jhsu802701/bsf .
What I do is set up the config/database.yml file to provide the username and password for the development environment programatically. For the development environment, I add commands to the config/database.yml script to acquire the development environment username (which is my regular username for the Debian Linux setup I use) and a blank password. (I give my username Postgres superuser privileges.) For the production environment, I add a command in the deployment script that acquires the username and password from files elsewhere on my account and writes this information to the config/database.yml file.
Is there a better solution?
Is there a Ruby gem that covers this? If not, I'm thinking of creating one.
The way that heroku does it, and a vast majority of other rails shops are with ENV variables
Export two variables to your environment,
export POSTGRES_USERNAME='username'
export POSTGRES_PASSWORD='password'
then in your database.yml file you can do
username: <%= ENV['POSTGRES_USERNAME'] %>
password: <%= ENV['POSTGRES_PASSWORD'] %>
This is how I make it work:
On terminal/cmd:
heroku config:set YOUR_DATABASE_PASSWORD=passywordy
Then, in /config/database.yml file;
production:
<<: *default
password: <%= ENV['YOUR_DATABASE_PASSWORD'] %>
(this password area is automatically generated when I used rails new my_app -d postgresql)
On other than heroku export you variables to system environment (linux) by typing in bash
export KEY=value
Then you can call it in Rails by ENV['KEY']
e.g.
in bash:
export CMS_DATABASE_PASSWORD=MySecurePassword
in secrets.yml:
password: <%= ENV['CMS_DATABASE_PASSWORD'] %>
Setting the environment variables as described in existing posts above, will only persist the environment variables for the duration of the current shell session.
To set the environment variables permanently, the export instruction(s) should be added to your shell config file. (Then run source ~/.bashrc to apply the changes to your current session)
TL;DR: If you're using BASH, add the export instruction(s) to ~/.bashrc.
While the above should suffice (if using BASH on most popular Linux distros), confidently identifying which config file to update for your shell can be quite tricky. The following post explains the reasons why and provides guidance on which config file to edit.
https://unix.stackexchange.com/questions/117467/how-to-permanently-set-environmental-variables

When trying to configure Rails Ruby or jRuby app the secret key is visible in the YAML file. Is there a gem that will provide this at runtime?

There are lots of secrets that apps have that need to be secured in production when running in public cloud and PaaS environments. Common one is database.yml entries for mysql user and password, but there can be others. Your Google App secret, facebook app secret,... the list goes on. There are no clear way of securing these essentially configuration parameters. You DO NOT want to put these in a file as there is no guarantee who will have access to it.
In Heroku you can specify things via environment variables. In Cloudbees (a Java PaaS) you can specify these as Java System properties. Both Heroku and Cloudbees have a commandline utility for uploading this configuration parameters but there is no support for making this work both in development and production easily.
Question is how do you configure your parameters so that you can develop easily in development but not have the production secret be available in development
Ideally there would be a gem that will work in ruby and jruby environment and PaaS that will allow me to specify me secrets in a YML file that has development settings in development but actual production secrets pulled in from ENV or java.lang.System.getProperty.
##
# file: config/secure_config.yml
development:
db:
user_id: 'dev_mysql_user'
password: 'my_dev_pwd'
google:
app_id: 'xxxxx' # this is the secret for the dev app so it can be visible
app_secret: 'xxxxx'
# ...
production:
db:
user_id: <%= get_secure_config %>
password: <%= get_secure_config %>
google:
app_id: <%= get_secure_config %>
app_secret: <%= get_secure_config %>
Where the get_secure_config helper gets the value from ENV or java.lang.System.getProperty in case of Ruby or jRuby. The finally you can use them in your app as needed. For example in database.yml or in the devise code to authenticate using google.
# config/database.yml
# ...
production:
adapter: mysql2
username: <%= SecureConfig.db.user_id %>
password: <%= SecureConfig.db.password %>
And then for extra coolness the gem should also give me an executable that allows me to push the config to my PaaS
~/work/myproject> bundle exec secure_config -push_to_heroku
or
~/work/myproject> bundle exec secure_config -push_to_cloudbees
Check out Figaro. It's almost exactly what you're after.
You place your sensitive data in a git-ignored YAML file, which are then made available to the app in ENV. It also provides a rake task for configuring your Heroku instance with the variables.
As Daniel Wright suggested above Figaro is great! It does every thing I need for ruby on rails. I needed the same support on JRuby on rails and properties via JVM system properties for Cloudbees PaaS service as well. I have forked Figaro and made these extensions and sent a pull request to laserlemon/Figaro. In the mean time you can pull it using git directive in your gem file.
gem 'figaro', '0.4.2', :git => "git://github.com/RedMicaInc/figaro.git"
Main differences are documented below
How does it work?
It works really well.
There are a few similar solutions out there, and a lot of homegrown attempts. Most namespace your configuration under a Config (or similar) namespace. That's fine, but there's already a place to describe the application environment… ENV!
ENV is a collection of simple string key/value pairs and it works just great for application configuration.
These configuration parameters are also then made available as properties of FigaroSettings object. So if you had a property called MY_PROP you can use it in your code or configuration files using FigaroSettings.my_prop or FigaroSettings.MY_PROP
For JRUBY based applications properties stored in JVM system properties are also available in a similar manner. For instance if you had a property called MY_JAVA_PROP it is accessible as FigaroSettings.MY_JAVA_PROP. Java properties are case sensitive.
As an added bonus, this is exactly how apps on Heroku or Cloudbees are configured. So if you configure your Rails app using ENV, you're already set to deploy to Heroku. For Cloudbees you can use ENV or JVM properties similarly using FigaroSettings.<property>
How does it work with Cloudbees?
Cloudbees provides application configuration in a similar manner.
Typically, to configure your application parameters accessible via JVM system properties, you would do the following from the command line using the cloudbees sdk:
cloudbees config:set -a <my_app> PUSHER_APP_ID=8926
cloudbees config:set -a <my_app> PUSHER_KEY=0463644d89a340ff1132
cloudbees config:set -a <my_app> PUSHER_SECRET=0eadfd9847769f94367b
But Figaro provides a rake task to do just that! Just run:
rake figaro:cloudbees
Optionally, you can pass in the name of the Cloudbees app:
rake figaro:cloudbees[my-awesome-app]
If you just want to see the commands used you can run rake figaro:heroku_test

Rails 3 - rake tasks to work with multiple database connections

Let's say I have s different database connections...
Right now for the purposes of this example let's only be concerned with development. When I run 'rake db:setup' it only creates development because from the rake task perspective the other connection is another environment. I could pass in the other connection as the RAILS_ENV to create the database. However, the problem comes in with how we've defined our database connections. I'd rather not undo this as it makes management and deployment much easier. What we've done is this...
In our database.yml we've added the following code.
databases_file = File.join(Rails.root.to_s, "config", "databases", "#{Rails.env.to_s}.yml")
IO.read(databases_file) if File.exist?(databases_file)
Then in a databases sub-folder under config we've created the different environment yml files.
So in development.yml we have...
development:
... details ...
logging:
... details ...
Now if I set RAILS_ENV to 'logging' it will never load the file since the file is called development.yml. Thus it will never find the logging connection.
If I consolidate all these settings back into the database.yml file then we're back to not checking in the database.yml into the git repo and having to manually create it on the development machines for new hires and such. We'd also have to change our deploy process to start putting the file in place on the lower environments. (Our production process already supports this)
You could use environment variables to manage these. You could have a single database.yml with things defined such as:
development:
database: ENV['DEVELOPMENT_DATABASE']
And then set the environment variables in your .bashrc/.zshrc.
We have multiple databases and we use branches heavily... so I came up with this solution to create multiple databases and tie their names to the branch being worked on...
# config/database.yml
<%=
databases_file = File.join(Rails.root.to_s, "config", "databases", "#{Rails.env.to_s}.yml")
if Rails.env.development? || Rails.env.test?
branch = ENV['BRANCH'] || `git branch --no-color 2> /dev/null`.chomp.split("\n").grep(/^[*]/).first[/(\S+)$/,1].sub("-", "_")
puts "Using databases for #{branch}"
IO.read(databases_file).gsub!("<branch>", branch) if File.exist?(databases_file)
else
IO.read(databases_file) if File.exist?(databases_file)
end
%>
And then you create a database file for each environment. In the development.yml and test.yml you use the token to let this script look for what it needs to in order to do replacement. So an example file would look like this.
# config/databases/development.yml
development:
adapter: mysql2
host: 127.0.0.1
database: <branch>_dev
username: user
password: pass
versions:
adapter: mysql2
host: 127.0.0.1
database: <branch>_versions
username: user
password: pass
reporting:
adapter: mysql2
host: 127.0.0.1
database: <branch>_reporting
username: user
password: pass
ods:
adapter: mysql2
host: 127.0.0.1
database: <branch>_ods
username: user
password: pass
I then extended the rake tasks to handle multiple databases
This is where most of the work went in. Therefore, I'm going to leave it out of the answer and make you work for it! No actually, it's just a big large mess that I'm not proud of and would love time to fix it, but haven't found any. I don't want to lead anyone down the wrong path but if you have questions just get me a message and I'll be happy to help.

Resources