My personal rails project uses a few API's for which I store the API keys/secrets in config/environments/production.yml and development.yml as global variables. I now want to push this project to github for others to use, but I don't want them to have those bits of sensitive data. I also don't want this file in .gitignore because it's required for the app to run. I've considered putting them in the DB somewhere, but am hoping to find a better solution.
TLDR: Use environment variables!
I think #Bryce's comment offers an answer, which I'll just flush out. It seems one approach Heroku recommends is to use environment variables to store sensitive information (API key strings, database passwords). So survey your code and see in which you have sensitive data. Then create environment variables (in your .bashrc file for example) that store the sensivite data values. For example for your database:
export MYAPP_DEV_DB_DATABASE=myapp_dev
export MYAPP_DEV_DB_USER=username
export MYAPP_DEV_DB_PW=secret
Now, in your local box, you just refer to the environment variables whenever you need the sensitive data. For example in database.yml :
development:
adapter: mysql2
encoding: utf8
reconnect: false
database: <%= ENV["MYAPP_DEV_DB_DATABASE"] %>
pool: 5
username: <%= ENV["MYAPP_DEV_DB_USER"] %>
password: <%= ENV["MYAPP_DEV_DB_PW"] %>
socket: /var/run/mysqld/mysqld.sock
I think database.yml gets parsed just at the app's initialization or restart so this shouldn't impact performance. So this would solve it for your local development and for making your repository public. Stripped of sensitive data, you can now use the same repository for the public as you do privately. It also solves the problem if you are on a VPS. Just ssh to it and set up the environment variables on your production host as you did in your development box.
Meanwhile, if your production setup involves a hands off deployment where you can't ssh to the production server, like Heroku's does, you need to look at how to remotely set up environment variables. For Heroku this is done with heroku config:add. So, per the same article, if you had S3 integrated into your app and you had the sensitive data coming in from the environment variables:
AWS::S3::Base.establish_connection!(
:access_key_id => ENV['S3_KEY'],
:secret_access_key => ENV['S3_SECRET']
)
Just have Heroku create environment variables for it:
heroku config:add S3_KEY=8N022N81 S3_SECRET=9s83159d3+583493190
Another pro of this solution is that it's language neutral, not just Rails. Works for any app since they can all acquire the environment variables.
How about this...
Create a new project and check it into GitHub with placeholder values in the production.yml and development.yml files.
Update .gitignore to include production.yml and development.yml.
Replace the placeholder values with your secrets.
Now you can check your code into GitHub without compromising your secrets.
And anyone can clone your repo without any extra steps to create missing files (they'll just replace the placeholder values as you did).
Does that meet your goals?
They're probably best put in initializers (config/initializers/api.yaml) though I think what you've got cooked up is fine. Add the actual keys to your .gitignore file and run git rm config/environments/production.yml to remove that sensitive data from your repo. Fair warning, it will remove that file too so back it up first.
Then, just create a config/environments/production.yml.example file next to your actual file with the pertinent details but with the sensitive data left out. When you pull it out to production, just copy the file without the .example and substitute the appropriate data.
Use environment variables.
In Ruby, they're accessible like so:
ENV['S3_SECRET']
Two reasons:
The values will not make it into source control.
"sensitive data" aka passwords tend to change on a per-environment basis anyways. e.g. you should be using different S3 credentials for development vs production.
Is this a best practice?
Yes: http://12factor.net/config
How do I use them locally?
foreman and dotenv are both easy. Or, edit your shell.
How do I use them in production?
Largely, it depends. But for Rails, dotenv is an easy win.
What about platform-as-a-service?
Any PaaS should give you a way to set them. Heroku for example: https://devcenter.heroku.com/articles/config-vars
Doesn't this make it more complicated to set up a new developer for the project?
Perhaps, but it's worth it. You can always check a .env.sample file into source control with some example data in it. Add a note about it to your project's readme.
Rails 4.1 has now a convention for it. You store this stuff in secrets.yml. So you don't end up with some global ENV calls scattered across Your app.
This yaml file is like database.yml erb parsed, so you are still able to use ENV calls here. In that case you can put it under version control, it would then serve just as a documentation which ENV vars has to be used. But you also can exlcude it from version control and store the actual secrets there. In that case you would put some secrets.yml.default or the like into the public repo for documentation purposes.
development:
s3_secret: 'foo'
production:
s3_secret: <%= ENV['S3_SECRET']%>
Than you can access this stuff under
Rails.application.secrets.s3_secret
Its discussed in detail at the beginning of this Episode
Related
Is database.yml the right place to read the AWS keys from bashrc? database.yml sounds like a place only for database configs. Is there a more appropriate place where the AWS configs from bashrc could be read inside my Rails app?
Rails 5.2 onwards
Rails 5.2 has introduced the concept of encrypted credentials. Basically, from Rails 5.2 onwards, there is an encrypted credentials file that is generated on initializing the app in config/credentials.yml.enc. This file is encrypted, and hence, can be pushed to your source control tool. There is also a master.key file which is generated while initializing the app, which can be used to decrypt the credentials file, and make changes to it.
So, credentials for AWS could be added to it as:
aws:
access_key_id: 123
secret_access_key: 345
These keys could be accessed in your app as Rails.application.credentials.aws[:secret_access_key]. Other sensitive config, like credentials to other external services that are being used, can also be added to this config. Check out this blog by Marcelo Casiraghi for more details.
Pre Rails 5.2
There was no concept of a credentials system prior to Rails 5.2. There are a couple of ways in which you could try to come up with a solution to store your configuration.
A. You could create a YAML file for defining your config from scratch.
Create a file called my_config.yml and place it in the config folder. Path: config/my_config.yml
Add whatever configuration is required to the file, in YAML format (s described for AWS above)
Make changes in application.rb to load this file during initialization as follows:
APP_CONFIG = YAML.load(ERB.new(File.new(File.expand_path('../my_config.yml', __FILE__)).read).result)[Rails.env] rescue {}
Using this approach, you will then be able to use APP_CONFIG['aws']['access_key_id'] for the AWS configuration defined above. For this use case, it is strongly recommended to have separate configuration files for development and production environments. The production file should probably not be checked in to version control for security.
B. Another approach would be to use some gems for managing configurations like railsconfig/config
NOTE: To answer the bit about storing this configuration in database.yml, it is strongly recommended to not do so. database.yml is a configuration file for storing configuration related to databases. Separation of concerns really helps while scaling any application, and hence, it is recommended to place such configurations in a separate file, which can be independently maintained, without any reliance on the database config.
Absolutely. The standard place to configure things like AWS would be inside config/initializers. You can create a file in there called aws.rb.
app/
bin/
config/
|__ initializers/
|__ aws.rb
and inside this file you can configure your AWS setup using the environment variables from your bashr
Aws.config.update({
credentials: Aws::Credentials.new('your_access_key_id', 'your_secret_access_key')
})
Files inside this directory are executed on app start, so this configuration will be executed right when your app starts, before it starts handling requests.
It may also be useful to note that the AWS SDK for Ruby will automatically search for specific environment variables to configure itself with. If that's what you're using, and if you have the following environment variables set up in your bashrc
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
then you won't need any additional code in your Rails app to configure AWS. Check out more details here.
I'm working in a team in a Rails project.
Each of us have a local database for development purpose. We have a problem: Everyone have different configuration for the local database. When someone make a commit without reset the /config/database.yml the other members of the team can't use their database because the access is not configured.
Can I have a local configuration not commited? To each one can works without problem and without the need of re-set the file every time? Sometime like the local_settings.py in Django
You can configure the database in Rails via both the config/database.yml file and the DATABASE_URL env var. They are merged but settings from DATABASE_URL take precedence.
A good solution is to check in the config/database.yml with a basic configuration thats just enough to support postgres.app or whatever is the most common solution.
Developers that need other settings can set the DATABASE_URL env var to customize the database configuration. Gems like figaro and dotenv make this easier.
If you have ever wondered this how Heroku manages to use the correct DB no matter what you throw into database.yml and the DATABASE_URL ENV var is how you should be configuring your production DB.
Why should I use ENV vars and not more database_*.yaml files?
The twelve-factor app stores config in environment variables (often
shortened to env vars or env). Env vars are easy to change between
deploys without changing any code; unlike config files, there is
little chance of them being checked into the code repo accidentally;
and unlike custom config files, or other config mechanisms such as
Java System Properties, they are a language- and OS-agnostic standard.
https://12factor.net/config
Add config/database.yml in to the .gitignore file at root path of your rails-app.
Copy config/database.yml with the values you need for production into config/database_example.yml.
Now you can modify your local database and in production you copy config/database_expample.yml to config/database.yml
If the config file is ignored by git, everyone can change it locally without getting tracked by git.
EDIT:
HERE YOU SEE HOW YOU CAN REMOVE FILE FROM TRACKING!!!
Ignore files that have already been committed to a Git repository
I work with Rails 4 and Ruby 2.1 and sorry but I am working on Windows
I have read a lot about this topic "Missing secret token, secret key base" but actually I do not understang anything.
I do not use Heroku, Git, Puma, Passenger or everything else I've read. I just thought I could instead of running rails s as usual run rails s -e production and see what is the version of my web application in production.
But I have the error "Missing secret_token and secret_key_base for production environment, set these values in config/secrets.yml"
I read about solutions using openSSL, export SECRET_KEY_BASE=<the long string> but I do understand the solutions.
I thought it was a problem related to the system of connection by password I settled thanks to Rails tutorial of Micheal Hartl. So disabled SSL connection. But nothing change.
This is my config/secrets.yml :
production:
secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>
Can someone explain how to concretely solve this issue ?
GDMN I am sorry that everyone gave you such poor explanations and instructions. Ok onto it then, shall we......
First everyone is right you no longer need "secret_token", you do however need "secret_key_base". What this does is it secures your visitors connection and keeps your system and app more secure. That is a simple explanation but all you need to worry about at the beginners level.
Second the ENV stands for "Environment Variable" they are used in all operating systems and they refer to variables on the OS level that hold information that you do not want to be accessible t someone trying to gain access to your site. For instance in Ruby On Rails if you HARD CODE the secrety_token_base string/hash then I hacker can gain access by using your security_token against you. I have seen this happen and it is not pretty, if the individual is skilled enough then they can gain access to even your root/admin account.
Now on to setting it all up. I only know the linux way and I know you are looking for the windows method but this should at least give you an understanding to seek out the information relevant to your operating system.
First thing you would need to do is generate your secret_token_base by running
bundle exec rake secret
To my knowlege this is the way you do it in all Operating Systems. After you run the above command the console will return a string and you would need to copy it. Once copied you would run the following command:
export SECRET_KEY_BASE=WhatYouJustCopied
Then we would check to make sure the Environment Variable SECRET_KEY_BASE is set by running:
env | grep -E "SECRET_TOKEN|SECRET_KEY_BASE"
If you do not have SECRET_TOKEN set you will only get the KEY_BASE.
If you want to learn more in depth information please visit this link it may be a little dated but most of it is still relevant and conceptually it is the same.
I wish you luck on your new found ROR Adventure! It is fun once you get the hang of it!
From your command prompt, run:
bundle exec rake secret
It will generate a long string of characters. Copy this string and paste it into config/secrets.yml as follows:
production:
secret_key_base: <paste the string here>
Note: only do this if you are not using a public repository. This key should not be accessible to anyone else. An alternate, and more secure way of doing this is using an environment variable. See this: http://daniel.fone.net.nz/blog/2013/05/20/a-better-way-to-manage-the-rails-secret-token/
So, if you look inside the secrets.yml file, you will see where your secret_key_base is set for each of your environments. When you look at the production setting, it wants an env variable to initialize your secret_key_base. Normally, in production, you would want your app server to fetch the value from a general place in case you need to spin multiple servers up, you wouldn't have to hard-code your secret_key_base everywhere since that is not a secure way of setting that variable.
Basically, you will have to have that env variable set up on the machine that will run your rails app in production. There are so many different ways to set this.
What I have set up to initialize my ENV variables for production is have a separate yml file that is constructed like this
# config/env_provider.yml
production:
SECRET_KEY_BASE: "KEY GOES HERE"
other_production_variables: #...etc
Then my separate servers will be told where to find this file before initializing the variables (This is not checked into version control). After the file is in place it will know to initialize the variables from the following code in environment.rb before your app initializes
#config/environment.rb
YAML.load_file("#{::Rails.root}/config/env_provider.yml")[::Rails.env].each {|k,v| ENV[k] = v }
# This is before Rails.application.initialize!
The thing with this set up is to make sure that you do not have this file accessible to everyone to see, only allow your application servers to use it. Anyway, this is how I deal with ENV variables and deploy them to production. I hope this helps you.
With the release of the secrets.yml file, I removed my reliance on Figaro and moved all of my keys to secrets.yml and added that file to .gitignore.
But when I tried to push to Heroku, Heroku said they needed that file in my repo in order to deploy the website. which makes sense, but I don't want my keys in git if I can avoid it.
With Figaro, I would run a rake task to deploy the keys to heroku as env variables and keep application.yml in the .gitignore. Obviously, I can't do that any more. So how do I handle this?
Secrets isn't a full solution to the environment variables problem and it's not a direct replacement for something like Figaro. Think of Secrets as an extra interface you're now supposed to use between your app and the broader world of environment variables. That's why you're now supposed to call variables by using Rails.application.secrets.your_variable instead of ENV["your_variable"].
The secrets.yml file itself is that interface and it's not meant to contain actual secrets (it's not well named). You can see this because, even in the examples from the documentation, Secrets imports environment variables for any sensitive values (e.g. the SECRET_KEY_BASE value) and it's automatically checked into source control.
So rather than trying to hack Secrets into some sort of full-flow environment variable management solution, go with the flow:
Pull anything sensitive out of secrets.yml.
Check secrets.yml into source control like they default you to.
For all sensitive values, import them from normal environment variables into secrets ERB (e.g. some_var: <%= ENV["some_var"] %>)
Manage those ENV vars as you normally would, for instance using the Figaro gem.
Send the ENV vars up to Heroku as you normally would, for instance using the Figaro gem's rake task.
The point is, it doesn't matter how you manage your ENV vars -- whether it's manually, using Figaro, a .env file, whatever... secrets.yml is just an interface that translates these ENV vars into your Rails app.
Though it adds an extra step of abstraction and some additional work, there are advantages to using this interface approach.
Whether you believe it's conceptually a good idea or not to use Secrets, it'll save you a LOT of headache to just go with the flow on this one.
PS. If you do choose to hack it, be careful with the heroku_secrets gem. As of this writing, it runs as a before_initialize in the startup sequence so your ENV vars will NOT be available to any config files in your config/environments/ directory (which is where you commonly would put them for things like Amazon S3 keys).
An equivalent for secrets.yml of that Figaro task is provided by the heroku_secrets gem, from https://github.com/alexpeattie/heroku_secrets:
gem 'heroku_secrets', github: 'alexpeattie/heroku_secrets'
This lets you run
rake heroku:secrets RAILS_ENV=production
to make the contents of secrets.yml available to heroku as environment variables.
see this link for heroku settings
if u want to run on local use like this
KEY=xyz OTHER_KEY=123 rails s
We have inherited a Rails project hosted on Heroku. The problem is that I don't understand how to get a copy of the database.yml or other various config files that have been ignored in the .gitignore file.
Sure I can look through the history but it seems like a hip shot when comparing it to what should be on the production server. Right now we are having to make changes to the staging environment which makes things a bit arduous and less efficient than having a local dev environment so we can really get under the hood. Obviously, having an exact copy of the config is a good place to start.
With other hosts it's easy to get access to all the files on the server using good ol' ftp.
This might be more easily addressed with some sort of git procedure that I am less familiar with.
Heroku stores config variables to the ENV environment
heroku config will display these list of variables
heroku config:get DATABASE_URL --app YOUR_APP
Will return the database url that Heroku as assigned to your app, using this string one can deduce the parameters necessary to connect to your heroku applications database, it follows this format
username:password#database_ip:port/database_name
This should provide you with all the values you'll need for the heroku side database.yml, its a file that is created for you each time you deploy and there is nothing it that can't be gotten from the above URL.
gitignoring the database.yml file is good practice
It's not something you can do - entries added to .gitignore means they've been excluded from source control. They've never made it into Git.
As for database.yml you can just create the file in app\config\database.yml with local settings, it should look something like;
development:
adapter: postgresql
host: 127.0.0.1
username: somelocaluser
database: somelocaldb
It's safe to say though, that if the file is not in Git then it's not on Heroku since that's the only way you can get files to Heroku. Any config is likely to be in environment variables - heroku config will show you that.