I have a Rails project PROJECTX, which is hosted on Heroku. For storing production configs and files, I am using a different repository PROJECTX-config. Is it possible to:
clone PROJECTX-config,
remove current config files, and
symlink config files to PROJECTX-config files
Note that this has to be done on Heroku. Also I am aware that Heroku has options to maintain configs using environment variables, but this is not what I am looking for.
Thanks!
No its not possible.
Each dyno gets its own ephemeral filesystem, with a fresh copy of the
most recently deployed code. During the dyno’s lifetime its running
processes can use the filesystem as a temporary scratchpad, but no
files that are written are visible to processes in any other dyno and
any files written will be discarded the moment the dyno is stopped or
restarted. For example, this occurs any time a dyno is replaced due to
application deployment and approximately once a day as part of normal
dyno management.
- https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
Or at least not without a Rube Goldberg machine like setupe where you setup some kind of automation (like a post-commit hook) to merge repo A and repo B and push the result to heroku.
Also I believe that app config should not be present in environment variables, as it is tedious to maintain rather than maintaining a file.
Heroku does not agree here.
The traditional approach for handling such config vars is to put them
under source - in a properties file of some sort. This is an
error-prone process, and is especially complicated for open source
apps which often have to maintain separate (and private) branches with
app-specific configurations.
A better solution is to use environment
variables, and keep the keys out of the code. On a traditional host or
working locally you can set environment vars in your bashrc file. On
Heroku, you use config vars.
- https://devcenter.heroku.com/articles/config-vars
Although you might be overestimating what you actually need to store in ENV vars. You only need to store secrets such as API keys in ENV.
Other non-secret configuration such your settings for various gems can and should be setup in config/initializers.
If you still think using the GUI is that terrible then use YAML files which you parse and use to set the ENV vars:
require 'yaml'
yaml = YAML.load_file(File.join(__dir__, 'conf.yml'))
def create_key(*components)
components.join('_').upcase
end
env_vars = yaml["production"].each_with_object({}) do |(key,value), memo|
key_components = [key]
if value.kind_of? Hash
value.each_pair do |k,v|
memo[create_key(*key_components.dup.push(k))] = v
end
else
memo[create_key(*key_components)] = value
end
end.each do |k,v|
system("heroku config:set #{k}=#{v}")
puts "Setting #{k} = #{v}; #{ $? }"
end
Or you could even store a serialized form (JSON or YAML) in a single env var - there is a total size limit of 32kb though.
Related
TL; DR
How to use production.yml.enc and staging.yml.enc credential files in two production- and staging- Rails apps, while the app has only the regular development, test and production environments?
I'm using Heroku and refer to it in this question. Yet this is not specific to this vendor.
In detail
An application is often deployed multiple times. An instance serves as the production, while another is the staging app, expected to be put in production. Rails facilitates this pattern, since creating new environments is easy.
However, Heroku suggests not to do this, with good reasons. For example, one can be tempted to put some if Rails.env.production? here and there, paving the way for some "but it works in staging!?" on Friday evening. It is better to have a single production environment, with different sets of parameters to actually differentiate the stages (eg. a different AWS S3 bucket name, a different API key, etc.). To achieve this, Heroku's advice is to rely on environment variables.
Since Rails 5.2 and later in Rails 6, credentials are conveniently handled via encrypted Yaml files in config/credentials. This is typically where one would like to store all these variables that change from an environment to another, instead of using messy environment variables. This mechanism can be used in Heroku thanks to a single RAILS_MASTER_KEY environment variable that contains the key used to decrypt the credentials file.
But these pieces do not fit well. How can we have a single production environment, whereas credentials files are per-environment?
This can be implemented with a staging credentials file and key, and a switch in application.rb based on an environment variable:
Create and populate a staging credentials file and key with EDITOR=vi rails credentials:edit --environment staging.
In application.rb, add:
# Introduce the environment variable RAILS_CREDENTIALS_ENVIRONMENT to specify a custom
# environment name of which to use the credentials file. Specifically aimed to use in staging,
# where the environment is set to "production", but we need to use the "staging" environment variables.
# This environment variable is also used to select the corresponding key file or -if that does not exist-
# the corresponding environment variable.
if ENV['RAILS_CREDENTIALS_ENVIRONMENT'].present?
new_filename_no_ext = Rails.root.join 'config', 'credentials', ENV['RAILS_CREDENTIALS_ENVIRONMENT']
config.credentials.content_path = "#{new_filename_no_ext}.yml.enc"
if File.exist? "#{new_filename_no_ext}.key"
config.credentials.key_path = "#{new_filename_no_ext}.key"
end
end
Set an environment variable RAILS_CREDENTIALS_ENVIRONMENT to production or staging, depending on the case. For example, Heroku does this with:
heroku config:set -a theapp-staging RAILS_CREDENTIALS_ENVIRONMENT=staging
If on staging or produciton you want to store your key in an environment variable instead of a key file, then simply assign the key to the RAILS_MASTER_KEY environment variable. As documented, this takes precendence over keys stored in files. Note that on your development machine you wouldn't want to have a RAILS_MASTER_KEY set, otherwise the credential files for all environments still get the same key and are thus accessible by everyone that needs to have access to (e.g.) only the development credentials.
Have several files one by each environment in your source code repository o local server disk, works but you will have several files with hardcoded values.
But, if you expect several teams with several requirements on your single app, you will need several development and test environments in order to keep an independent teams which are fully responsible for their Services:
Development
Release / Deployment
Ops (not platform/system administration)
An approach to management this is : externalize your configurations on platforms called: Configurations Manager
This platforms, must have the following features:
key-vaue pair creation by app. Like heroku web dashboard
expose http endpoint to get this variables from remote apps
security
Your rails app must get the variables at the startup stage or instantaneous if your language support hot reload variables.
Here some Configurations Managers:
Zookeeper : http://www.therore.net/java/2015/05/03/distributed-configuration-with-zookeeper-curator-and-spring-cloud-config.html
Consul : https://www.consul.io
Spring Cloud : https://www.baeldung.com/spring-cloud-configuration
Confignet: https://github.com/software-architect-tools/confignet
Whit this approach you will have a platform to management several apps for several environments like heroku web variables creation but more sophisticated.
I was reading from this article that you can create a config/local_env.yml with environment variables and then use config/application.rb to read it to replace/take priority before the environment variables that you export in your .bashrc. What then is the .env used for then? Does it serve the same purpose as the config/local_env.yml?
All of these methods are used to feed environment variables to your rails application. So, from an app's point of view, it serves the same purpose whether you export it from .env or .bashrc or config/local_env.yml files.
The differences in these methods are really a matter of personal choice among the team members involved in maintaining the app's development and deployment environments. However, here are few things to consider while opting for either of these choices.
.bashrc - Use this if you manually configure deployment servers and really comfortable with linux/unix command line system administration. This configuration file is specific to bash shell (Bourne Again Shell). You need to configure different file if your server uses different shell (for example: .zshrc if it uses Z Shell)
.env - Use this if you want to keep your app centric environment variables within the app itself while maintaining different variations of environment variables for different runtime environments of your rails app. For example: .env.development and .env.test files with specific values of the environment variables for your development and test environments respectively. This gives you more control of your app's environment variables and do not have to rely on the platform (terminal shell) you want to deploy your app.
config/local_env.yml - This is similar to .env approach, which is provided out of the box by rails gem that allows you to configure environment variables for your app in yml format. This method also keeps your app's configuration within the app irrespective of the shell you are using to run your app.
In addition to the previous answer, another downside of using .bashrc is that it is specific to only one user, so if you're e.g. starting your app server as a systemd service then I believe it won't see your variables.
Meanwhile, .env's Github readme says that it is not the most recommended thing to use outside of the development environment, although it is OK for that purpose.
Another two options to consider are:
Rails secrets. The benefit of this approach is that you get to commit it to git since it's encrypted, so when working in a team you will all have access to the same file. The downside is that it is coupled to the RAILS_ENV variable, so you can't use it to set that e.g. to production on your production app (but you can manually pass it every time it's invoked). Another downside is that if you have a staging environment, then apparently Heroku discourages using RAILS_ENV=staging, which you sometimes really need, so if you need it then you can either do it anyway, or you'll need to set the differing variables via a different mechanism - for example my app has a variable which points to the URL of another part of my app, this URL needs to point to its staging variant on staging, and to its production variant on production, so it needs to differ between my production and staging environment.
Using an /etc/environment file - note that systemd services don't have access to it by default so you'd have to add the line EnvironmentFile=/etc/environment. Also if you're running a shell script from a non-login shell (which happens sometimes), they it won't load them either, but the solution is to just include in your script this: set -a; source /etc/environment; set +a. You should be careful not to commit this to git. If you're working in a team and you need to manage this file then it gets a little complicated since it's not committed to git, but maybe there's a way to have it encrypted. (systemd services note: you might optionally use LoadCredential= for sensitive variables such as private keys, so that other processes won't have access to them; if you do this then you can also commit to git /etc/environment which should now contain non-sensitive variables only).
In my opinion both of those are valid and it's fine to pick whatever is easier to do in your app.
I'm working in a team in a Rails project.
Each of us have a local database for development purpose. We have a problem: Everyone have different configuration for the local database. When someone make a commit without reset the /config/database.yml the other members of the team can't use their database because the access is not configured.
Can I have a local configuration not commited? To each one can works without problem and without the need of re-set the file every time? Sometime like the local_settings.py in Django
You can configure the database in Rails via both the config/database.yml file and the DATABASE_URL env var. They are merged but settings from DATABASE_URL take precedence.
A good solution is to check in the config/database.yml with a basic configuration thats just enough to support postgres.app or whatever is the most common solution.
Developers that need other settings can set the DATABASE_URL env var to customize the database configuration. Gems like figaro and dotenv make this easier.
If you have ever wondered this how Heroku manages to use the correct DB no matter what you throw into database.yml and the DATABASE_URL ENV var is how you should be configuring your production DB.
Why should I use ENV vars and not more database_*.yaml files?
The twelve-factor app stores config in environment variables (often
shortened to env vars or env). Env vars are easy to change between
deploys without changing any code; unlike config files, there is
little chance of them being checked into the code repo accidentally;
and unlike custom config files, or other config mechanisms such as
Java System Properties, they are a language- and OS-agnostic standard.
https://12factor.net/config
Add config/database.yml in to the .gitignore file at root path of your rails-app.
Copy config/database.yml with the values you need for production into config/database_example.yml.
Now you can modify your local database and in production you copy config/database_expample.yml to config/database.yml
If the config file is ignored by git, everyone can change it locally without getting tracked by git.
EDIT:
HERE YOU SEE HOW YOU CAN REMOVE FILE FROM TRACKING!!!
Ignore files that have already been committed to a Git repository
I've enjoyed using Rails on Heroku, and like that I can adjust the configuration property of a Heroku app without having to commit a change to xyz.yml and redeploy.
It would be nice to completely do away with the Yaml config files in my Rails app, and rely as much as possible on storing configuration in ENV. This goes along with the 12-factor config principle.
However, there are some trade-offs in switching from a Yaml-based configuration management to a Heroku/12-factor-based one.
While it's true that a proliferation of deployments (qa, stage, prod, dev, demo, labs) can lead to a proliferation of Yaml files, it's very easy to copy-paste to create a new configuration profile. I don't see a way to 'copy' configuration profiles from one deployment to another in Heroku.
Storing configuration data in the repo means that, in the case of Heroku, deploying and configuring and application are accomplished in a single operation. If I were to move my configuration out of Yaml files and into ENV variables, I would have to configure my application in a separate step after deployment.
Would like to hear from people who have used 12-factor style configuration in their private applications, and how they have managed lots of configuration variables across lots of deployments.
How do you quickly configure a new deployment?
Where do you keep your authoritative source of configuration variables, if not the repo? How do you distribute it among developers?
Thanks!
What I typically use is Yaml using the ENV and provide defaults. For instance, YAML can be ERB'ed happily to include your ENV vars:
foo:
var: ENV["MY_CONFIG"] || "default_value"
You just need to make sure that you load the Yaml with ERB when you read it:
YAML.load(ERB.new(File.read("#{Rails.root}/config/app_config.yml")).result)
By doing this your code works fine in dev, but also allows you to set config vars in the environment as well.
You can accomplish this relatively easy with some simple shell scripts, iterate existing variables via heroku config or heroku release:info v99, and then set heroku config:set k=v --app
But if its a problem/pain/friction perhaps you have too much inside your env var configuration.
A bit of a late answer, but I believe this is what you are looking for.
I developed a gem called settei, allowing you to use YAML files to configure the app. However the gem will serialize the YAML file as one environment variable during deploy. This way one get the best of both worlds: YAML for ease of management/creating derivative environment, and ENV for 12-factor compliance.
I have a number of Rails apps hosted on GitHub. They are all currently private, and I often will deploy them from their GitHub repository. I'd like to be able to make some of them open source, just like the ones you can find on http://opensourcerails.com.
My question is: How can I make these repositories public without giving away super secret credentials?
For example, I can look in /config/initializers/cookie_verification_secret.rb and see the cookie secret for nearly every one of them. I don't understand how this is acceptable. Are these users all changing these values in their deploy environments somehow?
Some users even expose their AWS secret and key! Others will instead set their AWS secret to something like:
ENV['aws-secret']
although I'm not sure at what point they're setting that value.
So, what are the best practices for open sourcing your Rails app without compromising your app's security.
I recently went through this with one of my own apps. My solution was to store anything secret in a git-ignored YAML config file, and then to access that file using a simple class in the initializers directory. The config file is stored in the 'shared' folder for the Capistrano deployment and copied to config at each deploy.
Config store: http://github.com/tsigo/jugglf/blob/master/config/initializers/juggernaut.rb
Example usage: https://github.com/tsigo/jugglf/blob/6b91baae72fbe4b1f7efa2759bb472541546f7cf/config/initializers/session_store.rb
You may also want to remove from source control all history of the file that used these secret values. Here's a guide for doing this in Git that I used: http://help.github.com/removing-sensitive-data/
If you're using foreman, put an .env file in the root of your app. (foreman docs)
.env will have
AWS_SECRET=xxx
AWS_ACCESS=yyy
Then when you need to use the keys, insert:
ENV['AWS_SECRET']
ENV['AWS_ACCESS']
Though it's important that you don't commit this .env to your version control. So if you're using git, add the .env to your .gitignore.
Bonus round! - Heroku
If deploying to Heroku, these environment variables need to be configured in the Heroku environment, too. There are two options:
Manually add the keys through the heroku config:add command
Use the heroku-config gem to synchronize your local environment variables, both ways.
Not storing any secret value at all. At any point in the history of a Git repo.
Those values should be stored elsewhere, leaving only template config files versioned, along with a script able:
to read the right values from the external repo
and build the final config file complete (with the secret values in it)
By keeping the tow set of data separate (sources on one side, secret values on the other), you can then open source the sources repo without comprising any secrets.
I actually took a hint from your question, using ENV.
I had three different secret values that I didn't want made available. They're the app's secret token of course, and Twitter's consumer key and secret. In my secret token initializer:
KinTwit::Application.config.secret_token = ENV['SECRET_TOKEN']
Twitter.consumer_key = ENV['CONSUMER_KEY']
Twitter.consumer_secret = ENV['CONSUMER_SECRET']
I'm hosting my project on Heroku, so I added these as configuration variables to Heroku.
[03:07:48] [william#enterprise ~/dev/rwc/kintwit]$ heroku config:add CONSUMER_KEY=ub3rs3cr3tk3y
Adding config vars and restarting app... done, v7
CONSUMER_KEY => ub3rs3cr3tk3y
[03:08:40] [william#enterprise ~/dev/rwc/kintwit]$ heroku config:add CONSUMER_SECRET=ub3rs3cr3tk3y
Adding config vars and restarting app... done, v8
CONSUMER_SECRET => ub3rs3cr3tk3y
[03:08:57] [william#enterprise ~/dev/rwc/kintwit]$ heroku config:add SECRET_TOKEN=ub3rs3cr3tk3y
Adding config vars and restarting app... done, v9
SECRET_TOKEN => ub3rs3cr3tk3y
Now, the values are ready on my next push. But, what if you aren't using Heroku? I'm obviously not an expert on every single rails deployment (jeesh, not even a Heroku pro), but an example of this would be doing a db:migrate for testing.
$ RAILS_ENV=test rake db:migrate
The KEY=value pair before the command sets the environment variable, so running this command, echo ENV['RAILS_ENV'] would print test. So however this is set up in your environment is how you would do it. But, the environment variables aren't in your code, so that's the trick.
[EDIT - The following method has the annoyance of having to switch to the Production branch to run "rails server" in order to include necessary cookies. Thus, making edits while the server is difficult... and I'm still looking for a good solution]
After further investigation, I think the solution I was looking for was to exclude anything that stored a secret value from my Git repo's master branch (just as #VonC said). But instead of then reading from those files in a separate repo, I simply create a new "production" branch and add them to that.
This way they're excluded from Master and I can push that to Github or some other public repo just fine. When I'm ready to deploy, I can checkout the Production branch and merge Master into it and deploy Production.
I need to be able to do this because Heroku and other hosts require a single git repo to be pushed to their servers.
More information here:
http://groups.google.com/group/heroku/browse_thread/thread/d7b1aecb42696568/26d5249204c70574