In my app I have some code that changes when running locally versus in development on the remote server, such as app_domain in application.yml, etc. Besides having to manually change them each time I run locally versus deploy to remote machine, is there a better way of managing this?
EDIT - for yml
It depends a bit on how you are loading the application.yml but generally when you load yml files ruby parses it all into a hash for you. Thus you can set something up like the database.yml and structure all the yml under your different environments:
development:
domain: localhost
production:
domain: mydomain.com
test:
domain: foo
and then when you load your yml file you want to grab the settings for the specific environment you are in, like this:
MY_CONFIGS = YAML.load_file("[path to my yaml file]/application.yml")[RAILS_ENV]
I like Matthew's answer unless you have some secrets in those files (passwords).
If you have passwords you don't want in source, what I tell my UrbanDeploy customers when they have this type of scenario is to use a script (or we have a built-in script) that effectively does a token replace based on environment. Where the replacements are coming from the deployment system, or a magic file somewhere on the deployment target.
Either way, treat an environment as a first order thing that has data about it that needs to be managed.
Using kwateeSDCM you can customize any file at deployment time on a server by server basis. This way you only need one templatized application.xml and parametrize it depending on the target server.
Related
I'm working on a Rails app with a few collaborators and we decided to begin using separate database.yml files for some time until we can a configuration that works for all of us.
After adding database.yml to the .gitignore file and pushing a version without it, I realized that this would likely prevent the Heroku app from running.
My confusion is that the deployment was successful and the database.yml file was not needed. Why is this? Is our old database.yml file cached?
This is actually the expected behavior. For more details see: https://devcenter.heroku.com/articles/rails-database-connection-behavior
Which boils down to (for Rails 4.1+):
While the default connection information will be pulled from
DATABASE_URL, any additional configuration options in your
config/database.yml will be merged in.
Heroku will always use DATABASE_URL and merge the rest from database.yml to the config contained in that url.
Ah yes the old db config developer war.
Heroku actually uses the solution to this issue - Rails merges the database configuration from database.yml with a hash created from parsing ENV["DATABASE_URL"]. The ENV var takes precedence over the file based configuration.
When you first push a Rails app, Heroku automatically attaches a Postgres addon and sets ENV["DATABASE_URL"] and presto your app magically connects to the database.
Even if you add complete nonsense settings like setting the database name in database.yml the ENV var still wins.
How can this solve our developer war?
Do the opposite of what you are currently doing. Strip everything except the bare minimum required to run the application out of database.yml and check it back into version control.
Developers can use direnv or one of the many tools available to set ENV[DATABASE_URL] to customize the settings while database.yml should be left untouched unless you actually need to tweak the db.
Is database.yml the right place to read the AWS keys from bashrc? database.yml sounds like a place only for database configs. Is there a more appropriate place where the AWS configs from bashrc could be read inside my Rails app?
Rails 5.2 onwards
Rails 5.2 has introduced the concept of encrypted credentials. Basically, from Rails 5.2 onwards, there is an encrypted credentials file that is generated on initializing the app in config/credentials.yml.enc. This file is encrypted, and hence, can be pushed to your source control tool. There is also a master.key file which is generated while initializing the app, which can be used to decrypt the credentials file, and make changes to it.
So, credentials for AWS could be added to it as:
aws:
access_key_id: 123
secret_access_key: 345
These keys could be accessed in your app as Rails.application.credentials.aws[:secret_access_key]. Other sensitive config, like credentials to other external services that are being used, can also be added to this config. Check out this blog by Marcelo Casiraghi for more details.
Pre Rails 5.2
There was no concept of a credentials system prior to Rails 5.2. There are a couple of ways in which you could try to come up with a solution to store your configuration.
A. You could create a YAML file for defining your config from scratch.
Create a file called my_config.yml and place it in the config folder. Path: config/my_config.yml
Add whatever configuration is required to the file, in YAML format (s described for AWS above)
Make changes in application.rb to load this file during initialization as follows:
APP_CONFIG = YAML.load(ERB.new(File.new(File.expand_path('../my_config.yml', __FILE__)).read).result)[Rails.env] rescue {}
Using this approach, you will then be able to use APP_CONFIG['aws']['access_key_id'] for the AWS configuration defined above. For this use case, it is strongly recommended to have separate configuration files for development and production environments. The production file should probably not be checked in to version control for security.
B. Another approach would be to use some gems for managing configurations like railsconfig/config
NOTE: To answer the bit about storing this configuration in database.yml, it is strongly recommended to not do so. database.yml is a configuration file for storing configuration related to databases. Separation of concerns really helps while scaling any application, and hence, it is recommended to place such configurations in a separate file, which can be independently maintained, without any reliance on the database config.
Absolutely. The standard place to configure things like AWS would be inside config/initializers. You can create a file in there called aws.rb.
app/
bin/
config/
|__ initializers/
|__ aws.rb
and inside this file you can configure your AWS setup using the environment variables from your bashr
Aws.config.update({
credentials: Aws::Credentials.new('your_access_key_id', 'your_secret_access_key')
})
Files inside this directory are executed on app start, so this configuration will be executed right when your app starts, before it starts handling requests.
It may also be useful to note that the AWS SDK for Ruby will automatically search for specific environment variables to configure itself with. If that's what you're using, and if you have the following environment variables set up in your bashrc
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
then you won't need any additional code in your Rails app to configure AWS. Check out more details here.
My personal rails project uses a few API's for which I store the API keys/secrets in config/environments/production.yml and development.yml as global variables. I now want to push this project to github for others to use, but I don't want them to have those bits of sensitive data. I also don't want this file in .gitignore because it's required for the app to run. I've considered putting them in the DB somewhere, but am hoping to find a better solution.
TLDR: Use environment variables!
I think #Bryce's comment offers an answer, which I'll just flush out. It seems one approach Heroku recommends is to use environment variables to store sensitive information (API key strings, database passwords). So survey your code and see in which you have sensitive data. Then create environment variables (in your .bashrc file for example) that store the sensivite data values. For example for your database:
export MYAPP_DEV_DB_DATABASE=myapp_dev
export MYAPP_DEV_DB_USER=username
export MYAPP_DEV_DB_PW=secret
Now, in your local box, you just refer to the environment variables whenever you need the sensitive data. For example in database.yml :
development:
adapter: mysql2
encoding: utf8
reconnect: false
database: <%= ENV["MYAPP_DEV_DB_DATABASE"] %>
pool: 5
username: <%= ENV["MYAPP_DEV_DB_USER"] %>
password: <%= ENV["MYAPP_DEV_DB_PW"] %>
socket: /var/run/mysqld/mysqld.sock
I think database.yml gets parsed just at the app's initialization or restart so this shouldn't impact performance. So this would solve it for your local development and for making your repository public. Stripped of sensitive data, you can now use the same repository for the public as you do privately. It also solves the problem if you are on a VPS. Just ssh to it and set up the environment variables on your production host as you did in your development box.
Meanwhile, if your production setup involves a hands off deployment where you can't ssh to the production server, like Heroku's does, you need to look at how to remotely set up environment variables. For Heroku this is done with heroku config:add. So, per the same article, if you had S3 integrated into your app and you had the sensitive data coming in from the environment variables:
AWS::S3::Base.establish_connection!(
:access_key_id => ENV['S3_KEY'],
:secret_access_key => ENV['S3_SECRET']
)
Just have Heroku create environment variables for it:
heroku config:add S3_KEY=8N022N81 S3_SECRET=9s83159d3+583493190
Another pro of this solution is that it's language neutral, not just Rails. Works for any app since they can all acquire the environment variables.
How about this...
Create a new project and check it into GitHub with placeholder values in the production.yml and development.yml files.
Update .gitignore to include production.yml and development.yml.
Replace the placeholder values with your secrets.
Now you can check your code into GitHub without compromising your secrets.
And anyone can clone your repo without any extra steps to create missing files (they'll just replace the placeholder values as you did).
Does that meet your goals?
They're probably best put in initializers (config/initializers/api.yaml) though I think what you've got cooked up is fine. Add the actual keys to your .gitignore file and run git rm config/environments/production.yml to remove that sensitive data from your repo. Fair warning, it will remove that file too so back it up first.
Then, just create a config/environments/production.yml.example file next to your actual file with the pertinent details but with the sensitive data left out. When you pull it out to production, just copy the file without the .example and substitute the appropriate data.
Use environment variables.
In Ruby, they're accessible like so:
ENV['S3_SECRET']
Two reasons:
The values will not make it into source control.
"sensitive data" aka passwords tend to change on a per-environment basis anyways. e.g. you should be using different S3 credentials for development vs production.
Is this a best practice?
Yes: http://12factor.net/config
How do I use them locally?
foreman and dotenv are both easy. Or, edit your shell.
How do I use them in production?
Largely, it depends. But for Rails, dotenv is an easy win.
What about platform-as-a-service?
Any PaaS should give you a way to set them. Heroku for example: https://devcenter.heroku.com/articles/config-vars
Doesn't this make it more complicated to set up a new developer for the project?
Perhaps, but it's worth it. You can always check a .env.sample file into source control with some example data in it. Add a note about it to your project's readme.
Rails 4.1 has now a convention for it. You store this stuff in secrets.yml. So you don't end up with some global ENV calls scattered across Your app.
This yaml file is like database.yml erb parsed, so you are still able to use ENV calls here. In that case you can put it under version control, it would then serve just as a documentation which ENV vars has to be used. But you also can exlcude it from version control and store the actual secrets there. In that case you would put some secrets.yml.default or the like into the public repo for documentation purposes.
development:
s3_secret: 'foo'
production:
s3_secret: <%= ENV['S3_SECRET']%>
Than you can access this stuff under
Rails.application.secrets.s3_secret
Its discussed in detail at the beginning of this Episode
I have a Rails app that I package as a war file for deploying to Tomcat using Warbler. And it works, but the problem is I don't know how to configure the runtime properties like secret_key_base. I use the standard setup of using secrets.yml, with production variables coming from environment variables. But I don't know how to set the variables while still keeping them out of source control.
Ideally I'd still like to be able to deploy the war file automatically, by just dropping it into the webapps/ directory, but I suppose I could edit the server config file? Or is there a better way of handling this?
either do it the same way as you would in a Rails server ... let it read from ENV (of course you will need to make sure Tomcat has the environment variable set).
alternatively you can set it in a web.xml if you're packaging and than do a $servlet_context.getAttribute('foo') in secrets.yml ... or read it from a file location that only the server's tomcat username can access etc.
sky is the limit here - you basically need to decide what fits your deployments the best.
I want to run the same rails app several times with a few configuration differences on the same server. Each app must have its own:
database
ports
cookie_store key(not sure if needed)
secret_key_base
Let's say I want to run the same code multiple times to service different cities:
newyork.myapp.com and boston.myapp.com. I wonder what would be the best way to store and use the different configurations.
Use environments:
Add a file to config/environments, one for each site you want to host. Name it something along the lines of 'production_[city]', replacing [city] with the city name. Copy the production.rb file contents into each.
in config/database.yml find the 'production' block of yml and duplicate it once for each site you want to host. Rename the root node of each block to production_[city], matching the filenames above. For example:
production_ny:
adapter: mysql2
username: my_user
password: my_pa$$w0rd%&*#
database: production_ny
This takes care of the database settings per app.
Assuming Rails 4, your secret key base will be in config/secrets.yml under an environment node, as per config/database.yml so just add an entry per site:
production_ny:
secret_key_base: xxxxxxxx1111111122222223333333344444444...
All sites will need an end-point. Using a different domain for each would give you separate cookies and sessions for free. Or you could go the subdomain route:
http://guides.rubyonrails.org/configuring.html#deploy-to-a-subdirectory-relative-url-root
tldr:
In your config/environments/production_ny.rb:
config.relative_url_root = "/ny"
for example. Then set up your webserver to handle subdirectories. You may need to add a path to cookies in order to scope them to the virtual directory. Just use:
Rails.configuration.relative_url_root
Ports, again, will have to be set up at the webserver level. (Apache, nginx, etc.)
To see if it all works, try this in the command-line:
RAILS_ENV=production_ny bundle exec rails s
This should start a development style webserver for you to access, but use the production_ny environment.
You will need to create and set up your database as normal - create, migrate, seed.
The final step is setting the RAILS_ENV environment variable to production_[city] per app using your web server. The steps to do this will depend on your technology choice.