Is database.yml the right place to read the AWS keys from bashrc? database.yml sounds like a place only for database configs. Is there a more appropriate place where the AWS configs from bashrc could be read inside my Rails app?
Rails 5.2 onwards
Rails 5.2 has introduced the concept of encrypted credentials. Basically, from Rails 5.2 onwards, there is an encrypted credentials file that is generated on initializing the app in config/credentials.yml.enc. This file is encrypted, and hence, can be pushed to your source control tool. There is also a master.key file which is generated while initializing the app, which can be used to decrypt the credentials file, and make changes to it.
So, credentials for AWS could be added to it as:
aws:
access_key_id: 123
secret_access_key: 345
These keys could be accessed in your app as Rails.application.credentials.aws[:secret_access_key]. Other sensitive config, like credentials to other external services that are being used, can also be added to this config. Check out this blog by Marcelo Casiraghi for more details.
Pre Rails 5.2
There was no concept of a credentials system prior to Rails 5.2. There are a couple of ways in which you could try to come up with a solution to store your configuration.
A. You could create a YAML file for defining your config from scratch.
Create a file called my_config.yml and place it in the config folder. Path: config/my_config.yml
Add whatever configuration is required to the file, in YAML format (s described for AWS above)
Make changes in application.rb to load this file during initialization as follows:
APP_CONFIG = YAML.load(ERB.new(File.new(File.expand_path('../my_config.yml', __FILE__)).read).result)[Rails.env] rescue {}
Using this approach, you will then be able to use APP_CONFIG['aws']['access_key_id'] for the AWS configuration defined above. For this use case, it is strongly recommended to have separate configuration files for development and production environments. The production file should probably not be checked in to version control for security.
B. Another approach would be to use some gems for managing configurations like railsconfig/config
NOTE: To answer the bit about storing this configuration in database.yml, it is strongly recommended to not do so. database.yml is a configuration file for storing configuration related to databases. Separation of concerns really helps while scaling any application, and hence, it is recommended to place such configurations in a separate file, which can be independently maintained, without any reliance on the database config.
Absolutely. The standard place to configure things like AWS would be inside config/initializers. You can create a file in there called aws.rb.
app/
bin/
config/
|__ initializers/
|__ aws.rb
and inside this file you can configure your AWS setup using the environment variables from your bashr
Aws.config.update({
credentials: Aws::Credentials.new('your_access_key_id', 'your_secret_access_key')
})
Files inside this directory are executed on app start, so this configuration will be executed right when your app starts, before it starts handling requests.
It may also be useful to note that the AWS SDK for Ruby will automatically search for specific environment variables to configure itself with. If that's what you're using, and if you have the following environment variables set up in your bashrc
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
then you won't need any additional code in your Rails app to configure AWS. Check out more details here.
Related
It says in the docs they recommend to use separate keys for development and production I have my site keys stored in .env
I have my site keys stored in .env
dotenv is only supposed to be used for development. It's a convenience for developers to be able to set environment variables. However, because it is a file on disk it loses the security advantages and configuration convenience of environment variables. It should not be checked into version control, and it should not be deployed to production.
For production, put your secrets in environment variables. Most cloud production environments include convenient interfaces to set environment variables for your deployments.
Alternatively, put your secrets in Rails encrypted credentials.
Just use different .env file on your server
Or create .env.production file. If you use dotenv gem, it has higher priority
https://github.com/bkeepers/dotenv#what-other-env-files-can-i-use
Of course both files must be git ignored
Or use rails credentials. In new rails (probably since 6.0) you can use different credentials in different environments
When installing Figaro gem, an application.yml is automatically created. And inside this file I planned on storing login credentials for SendGrid.
But by default in the rails application, there is another secrets.yml file, with the secret_key_base.
I'm little confused on their relationship.
My question: Is it necessary to have both? Can I combine them? Should both be added to gitignore ?
You can leave it there as it is, just put the new variables to the application.yml and make sure you gitignore both files since you need those only for localhost. For production you have to put the keys to a different place based on the service. I'm using heroku and have to save production variables from terminal.
My personal rails project uses a few API's for which I store the API keys/secrets in config/environments/production.yml and development.yml as global variables. I now want to push this project to github for others to use, but I don't want them to have those bits of sensitive data. I also don't want this file in .gitignore because it's required for the app to run. I've considered putting them in the DB somewhere, but am hoping to find a better solution.
TLDR: Use environment variables!
I think #Bryce's comment offers an answer, which I'll just flush out. It seems one approach Heroku recommends is to use environment variables to store sensitive information (API key strings, database passwords). So survey your code and see in which you have sensitive data. Then create environment variables (in your .bashrc file for example) that store the sensivite data values. For example for your database:
export MYAPP_DEV_DB_DATABASE=myapp_dev
export MYAPP_DEV_DB_USER=username
export MYAPP_DEV_DB_PW=secret
Now, in your local box, you just refer to the environment variables whenever you need the sensitive data. For example in database.yml :
development:
adapter: mysql2
encoding: utf8
reconnect: false
database: <%= ENV["MYAPP_DEV_DB_DATABASE"] %>
pool: 5
username: <%= ENV["MYAPP_DEV_DB_USER"] %>
password: <%= ENV["MYAPP_DEV_DB_PW"] %>
socket: /var/run/mysqld/mysqld.sock
I think database.yml gets parsed just at the app's initialization or restart so this shouldn't impact performance. So this would solve it for your local development and for making your repository public. Stripped of sensitive data, you can now use the same repository for the public as you do privately. It also solves the problem if you are on a VPS. Just ssh to it and set up the environment variables on your production host as you did in your development box.
Meanwhile, if your production setup involves a hands off deployment where you can't ssh to the production server, like Heroku's does, you need to look at how to remotely set up environment variables. For Heroku this is done with heroku config:add. So, per the same article, if you had S3 integrated into your app and you had the sensitive data coming in from the environment variables:
AWS::S3::Base.establish_connection!(
:access_key_id => ENV['S3_KEY'],
:secret_access_key => ENV['S3_SECRET']
)
Just have Heroku create environment variables for it:
heroku config:add S3_KEY=8N022N81 S3_SECRET=9s83159d3+583493190
Another pro of this solution is that it's language neutral, not just Rails. Works for any app since they can all acquire the environment variables.
How about this...
Create a new project and check it into GitHub with placeholder values in the production.yml and development.yml files.
Update .gitignore to include production.yml and development.yml.
Replace the placeholder values with your secrets.
Now you can check your code into GitHub without compromising your secrets.
And anyone can clone your repo without any extra steps to create missing files (they'll just replace the placeholder values as you did).
Does that meet your goals?
They're probably best put in initializers (config/initializers/api.yaml) though I think what you've got cooked up is fine. Add the actual keys to your .gitignore file and run git rm config/environments/production.yml to remove that sensitive data from your repo. Fair warning, it will remove that file too so back it up first.
Then, just create a config/environments/production.yml.example file next to your actual file with the pertinent details but with the sensitive data left out. When you pull it out to production, just copy the file without the .example and substitute the appropriate data.
Use environment variables.
In Ruby, they're accessible like so:
ENV['S3_SECRET']
Two reasons:
The values will not make it into source control.
"sensitive data" aka passwords tend to change on a per-environment basis anyways. e.g. you should be using different S3 credentials for development vs production.
Is this a best practice?
Yes: http://12factor.net/config
How do I use them locally?
foreman and dotenv are both easy. Or, edit your shell.
How do I use them in production?
Largely, it depends. But for Rails, dotenv is an easy win.
What about platform-as-a-service?
Any PaaS should give you a way to set them. Heroku for example: https://devcenter.heroku.com/articles/config-vars
Doesn't this make it more complicated to set up a new developer for the project?
Perhaps, but it's worth it. You can always check a .env.sample file into source control with some example data in it. Add a note about it to your project's readme.
Rails 4.1 has now a convention for it. You store this stuff in secrets.yml. So you don't end up with some global ENV calls scattered across Your app.
This yaml file is like database.yml erb parsed, so you are still able to use ENV calls here. In that case you can put it under version control, it would then serve just as a documentation which ENV vars has to be used. But you also can exlcude it from version control and store the actual secrets there. In that case you would put some secrets.yml.default or the like into the public repo for documentation purposes.
development:
s3_secret: 'foo'
production:
s3_secret: <%= ENV['S3_SECRET']%>
Than you can access this stuff under
Rails.application.secrets.s3_secret
Its discussed in detail at the beginning of this Episode
I am using the figaro gem and have created an application.yml file with all of my variables as per the documentation. This application.yml file is located in a shared folder (I'm using capistrano) and is symlinked to config/application.yml within the current live app directory, however I can only access the variables in the rails console and not the app. My credentials are listed as follows (real details omitted):
Note: I have tried removing the "" speech marks and also prefixing this list with production: with each line having 2 spaces, not tabbed, and it doesn't solve anything. The permissions on the file are exactly the same, 777, as the databse.yml file which was implemented in the same way.
application.yml
FFMPEG_LOCATION: "/path/to/ffmpeg"
EMAIL_USERNAME: "me#gmail.com"
EMAIL_PASSWORD: "password"
S3_BUCKET: "my_bucket"
AWS_SECRET_KEY_ID: "my_secret_key"
AWS_ACCESS_KEY_ID: "my_access_key"
I can access these variables in the production console =>
Loading production environment (Rails 3.2.14)
irb(main):001:0> ENV["S3_BUCKET"]
=> "my-s3-bucket-name"
However they don't return anything in the app itself. I set my linux box up following Ryan's excellent Pro railscast episode http://railscasts.com/episodes/335-deploying-to-a-vps
How can I get these variables accessible in the app itself?
If anyone needs more code just shout.
EDIT
I removed the figaro gem implemented the yaml config shown in the following railscasts tutorial: http://railscasts.com/episodes/85-yaml-configuration-revised. I think this is effectively what the figaro gem was doing however instead of using ENV variables, the tutorial uses CONFIG[:variables] which seem to work great.
Per Comment:
Nginx runs as its own user, so the environment variables need to live in it's space. As a user when you log in and run console, you're accessing a different set of environment variables than the nginx user accesses.
You can do this if you choose by adding them to the nginx config in the main context. But it's probably easier to go with straight yaml and add your secret tokens to your yaml file.
I've enjoyed using Rails on Heroku, and like that I can adjust the configuration property of a Heroku app without having to commit a change to xyz.yml and redeploy.
It would be nice to completely do away with the Yaml config files in my Rails app, and rely as much as possible on storing configuration in ENV. This goes along with the 12-factor config principle.
However, there are some trade-offs in switching from a Yaml-based configuration management to a Heroku/12-factor-based one.
While it's true that a proliferation of deployments (qa, stage, prod, dev, demo, labs) can lead to a proliferation of Yaml files, it's very easy to copy-paste to create a new configuration profile. I don't see a way to 'copy' configuration profiles from one deployment to another in Heroku.
Storing configuration data in the repo means that, in the case of Heroku, deploying and configuring and application are accomplished in a single operation. If I were to move my configuration out of Yaml files and into ENV variables, I would have to configure my application in a separate step after deployment.
Would like to hear from people who have used 12-factor style configuration in their private applications, and how they have managed lots of configuration variables across lots of deployments.
How do you quickly configure a new deployment?
Where do you keep your authoritative source of configuration variables, if not the repo? How do you distribute it among developers?
Thanks!
What I typically use is Yaml using the ENV and provide defaults. For instance, YAML can be ERB'ed happily to include your ENV vars:
foo:
var: ENV["MY_CONFIG"] || "default_value"
You just need to make sure that you load the Yaml with ERB when you read it:
YAML.load(ERB.new(File.read("#{Rails.root}/config/app_config.yml")).result)
By doing this your code works fine in dev, but also allows you to set config vars in the environment as well.
You can accomplish this relatively easy with some simple shell scripts, iterate existing variables via heroku config or heroku release:info v99, and then set heroku config:set k=v --app
But if its a problem/pain/friction perhaps you have too much inside your env var configuration.
A bit of a late answer, but I believe this is what you are looking for.
I developed a gem called settei, allowing you to use YAML files to configure the app. However the gem will serialize the YAML file as one environment variable during deploy. This way one get the best of both worlds: YAML for ease of management/creating derivative environment, and ENV for 12-factor compliance.