I'd like to introduce per-machine configuration to my rails application. It's currently deployed with Capistrano, and I need to introduce at least one machine-specific variable -- a hostname, so that performance characteristics can be properly tracked. In the future, there are likely going to be further local differences. The code is deployed to a 3 node load balanced cluster from the same git repo.
What is the best way to manage machine specific config?
You can set environment variables in Capistrano config file, and fetch it in your rails app.
config/deploy/.rb
%w[
node1.example.com
node2.example.com
node3.example.com
].each do |host|
server host, roles: %w[app], default_env: {hostname: host}
end
Then you can access the hostname with ENV['HOSTNAME'].
Related
I'm working on a hobbyist project and have reached the point where either of my next two most important features require sending email, and this is a place that all the guides I find online seem to gloss over a lot of seemingly important details.
My development environment is Windows + RubyMine.
My production environment will be Heroku. (And probably SendGrid, maybe Mailgun, still evaluating). I don't have real domain name yet, still evaluating/work-shopping a couple of options there. It would be nice if the solution can work with a herokuapp domain name in the interim before I get the actual domain.
This will likely be an open source project, if it matters. Ie there may be more than one dev, and not all devs are under a single domain name.
This feels like the development default for mail configuration should probably be gmail, with each dev configuring their own authenticated/2-factor/apppkey via environment variables. While prod has the configuration for the chosen service provider/with authentication details via env variables.
Is this a reasonable approach? It feels common enough to me that I would have expected some of the great guides to address it, but all seem to assume you'll use the same implementation in dev and prod, and that you already know your prod delivery system & domain name.
the development default for mail configuration should probably be gmail
The dev default should be something like the letter_opener gem. Do not send mails from your dev setup.
prod has the configuration for the chosen service via env variables
Definitely true. ENV variables and also rails secrets, and yaml configuration files. My personal preference is to have yaml files (both for non-secret configs, read by Rails.application.config_for and the secrets file) with interpolated values from ENV:
# config/action_mailer.yml
production:
smtp_server: some-value
smtp_password: <%= Rails.application.secrets.smtp_password %>
# config/secrets.yml
production:
smtp_password: <%= ENV['SMTP_PASSWORD'] %>
this way you can abstract out where the config comes from. For instance, you can push a completely replaced secrets file in production, instead of injecting anything in ENV, and the application won't need to know the difference.
I am having a very difficult time trying to launch a sample Rails 6 application to Elastic Beanstalk. For context, I am following these instructions
ADD RDS to Ruby Application
ADD an RDS to Beanstalk
I have followed these instructions to a tee and am still unable to connect to the rds database that I have provisioned. I keep receiving the following error:
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Whenever I try to run RAILS_ENV=production rails db:migrate or any other rake task, I keep getting that error.
On my AWS console, under Configuration and Software, I have the following environment variables:
Also in my database.yml file I have the rds configured variables listed as such.
production:
adapter: postgresql
database: <%= ENV['RDS_DB_NAME'] %>
username: <%= ENV['RDS_USERNAME'] %>
password: <%= ENV['RDS_PASSWORD'] %>
host: <%= ENV['RDS_HOSTNAME'] %>
port: <%= ENV['RDS_PORT'] %>
I have mapped my values as instructed in the documentation and am certain that they are correct.
Finally, I have sshed into my beanstalk provisioned ec2 instance and have executed the following command:
psql -U username -p 5432 -h examplehost.rds.amazonaws.com -d ebdb
provided the password and am able to connect. I am really at my wits end, I've spent too much time trying to diagnose this and am running out of ideas. I don't know where to look too next for ideas on how to trouble shoot this. I've read so many stack overflow questions and blogs that my head is spinning. If anyone has any ideas on how to resolve this, I would greatly appreciate it.
---Update----
I have created a new environment variable on the elastic beanstalk console.
ENV['DATABASE_URL'] = postgres://YourUserName:YourPassword#YourHostname:5432/YourDatabaseName
I made the necessary configurations, uploaded my .zip file and the connection to the database failed.
---- UPDATE-----
printenv does not show the varialbes provided by beanstalk, however this command does sudo /opt/elasticbeanstalk/bin/get-config environment.
My first advice is that, in my opinion, it is a much better option to create an Amazon RDS on their own, and not tied to Beanstalk.
As the AWS documentation indicates (emphasis mine):
AWS Elastic Beanstalk provides support for running Amazon Relational Database Service (Amazon RDS) instances in your Elastic Beanstalk environment. To learn about that, see Adding a database to your Elastic Beanstalk environment. This works great for development and testing environments. However, it isn't ideal for a production environment because it ties the lifecycle of the database instance to the lifecycle of your application's environment.
And:
To decouple your database instance from your environment, you can run a database instance in Amazon RDS and configure your application to connect to it on launch. This enables you to connect multiple environments to a database, terminate an environment without affecting the database, and perform seamless updates with blue-green deployments.
In my opinion, even for testing or development, it is always advisable to configure a small database instance and give your application the ability of define the most appropriate mechanism for connecting to your database.
The only downside is that you will probably need to configure a VPC, although it should not be actually a problem and, in ay case, it is worth value.
If for any reason you need to use the Beanstalk provisioned RDS database perhaps you have some workarounds to your problem (it should be a workaround because your configuration looks fine - please, only, verify that the database configuration is defined for the right Beanstalk environment).
For instance, one thing you can try is to store the database connection configuration in a S3 bucket, as also suggested in the AWS documentation. The idea is basically create some configuration file with the necessary connectivity information, store it in S3, and read that configuration in your application, i.e., process that file, in order to initialize your database.
But maybe you can try another approach.
Please, consider this SO question, and the answer from Jon McAuliffe and others. As indicated, Beanstalk will provide your application with environment variables, but maybe this variables will not be exposed as shell variables, they will be exposed to your application in different ways depending on the runtime the application needs to be executed on.
In the case of Ruby, you are accessing these variables in the correct way but, for any reason, your program is not having access to that information.
This probably also explains why printenv does not print any if your variables but the get-config script does.
But maybe you can take advantage of the fact that get-config provides you the right information and, either define this variables in your ENV by executing the get-config script for every RDS* key, perhaps in your environment.rb - please, be aware that I programmed in Ruby when I was a student but there is a long time since that, do the task in the file you consider appropriate - or using .ebextensions and a custom configuration file. You can find several examples here.
For instance, consider the following (copy and paste, with minor modifications of this example configuration):
commands:
01_update_env:
command: "/tmp/update_environment_variables.sh"
files:
"/tmp/update_environment_variables.sh":
mode: "000755"
content : |
#!/bin/bash
RDS_HOSTNAME=$(/opt/elasticbeanstalk/bin/get-config environment -k RDS_HOSTNAME)
if [ -z "$RDS_HOSTNAME" ]; then
echo "Could not determine RDS hostname"
exit 1
fi
echo "RDS hostname $RDS_HOSTNAME..."
# Just export the variable at OS level, or make it visible to
# the rails env in some other way
export RDS_HOSTNAME=$RDS_HOSTNAME
# Process the rest of the variables...
# Probably we should create a list and iterate through it
A similar approach could be the one exposed in this stackoverrun question, but restricted to the container that Beanstalk will use to encapsulate your app. AFAIK, the container should receive as env variables the different RDS* ones corresponding to the database configuration.
Dan, be aware that I have not tested these solutions, they are only ideas: please, be careful with that, I do not want to cause any damage to your system.
I found an answer for this problem with a mysql server that might still help you. Basically, even though I followed all your steps, could see my envars using sudo /opt/elasticbeanstalk/bin/get-config environment and could connect directly to my database with the mysql command, I was still getting the following error:
Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) (Mysql2::Error::ConnectionError)
The solution turned out to be the fact that Elastic Beanstalk was not connecting my envars to my bundle exec rails console command in the eb ssh instance access. I solved the issue by prepending all of the required envars explicitly to any rails commands I ran from within the eb ssh instance access. So for example, in order to run rails console, I had to run the following:
RAILS_MASTER_KEY=xxxxxxx RAILS_ENV=production RDS_HOSTNAME=xxxxxxx RDS_PASSWORD=xxxxxxx RDS_USERNAME=xxxxxxx RDS_DB_NAME=xxxxxxx AWS_REGION=xxxxxxx AWS_BUCKET=xxxxxxx bundle exec rails c
Replace the xxxxxxxs above with the values from the corresponding variables in your EB > Configuration > Software tab, and you should be able to connect to the remote database and run migrations, rake tasks and other database-reliant functions.
For Linux2 instances I was having the same issue and just noticed that the env variables I set in the config just didn't exist for su that I had set myself to -- if I remain the default login after eb ssh env prints everything I expected
edit: sorry -- env printing of variables on linux 2 instance enabled by
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
so what I did was find where those env variables were being exported for default user shell, which was /etc/profile.d/sh.local as noted in the above aws knowledge center link and just source that file when I needed to start the rails console as su
TL; DR
How to use production.yml.enc and staging.yml.enc credential files in two production- and staging- Rails apps, while the app has only the regular development, test and production environments?
I'm using Heroku and refer to it in this question. Yet this is not specific to this vendor.
In detail
An application is often deployed multiple times. An instance serves as the production, while another is the staging app, expected to be put in production. Rails facilitates this pattern, since creating new environments is easy.
However, Heroku suggests not to do this, with good reasons. For example, one can be tempted to put some if Rails.env.production? here and there, paving the way for some "but it works in staging!?" on Friday evening. It is better to have a single production environment, with different sets of parameters to actually differentiate the stages (eg. a different AWS S3 bucket name, a different API key, etc.). To achieve this, Heroku's advice is to rely on environment variables.
Since Rails 5.2 and later in Rails 6, credentials are conveniently handled via encrypted Yaml files in config/credentials. This is typically where one would like to store all these variables that change from an environment to another, instead of using messy environment variables. This mechanism can be used in Heroku thanks to a single RAILS_MASTER_KEY environment variable that contains the key used to decrypt the credentials file.
But these pieces do not fit well. How can we have a single production environment, whereas credentials files are per-environment?
This can be implemented with a staging credentials file and key, and a switch in application.rb based on an environment variable:
Create and populate a staging credentials file and key with EDITOR=vi rails credentials:edit --environment staging.
In application.rb, add:
# Introduce the environment variable RAILS_CREDENTIALS_ENVIRONMENT to specify a custom
# environment name of which to use the credentials file. Specifically aimed to use in staging,
# where the environment is set to "production", but we need to use the "staging" environment variables.
# This environment variable is also used to select the corresponding key file or -if that does not exist-
# the corresponding environment variable.
if ENV['RAILS_CREDENTIALS_ENVIRONMENT'].present?
new_filename_no_ext = Rails.root.join 'config', 'credentials', ENV['RAILS_CREDENTIALS_ENVIRONMENT']
config.credentials.content_path = "#{new_filename_no_ext}.yml.enc"
if File.exist? "#{new_filename_no_ext}.key"
config.credentials.key_path = "#{new_filename_no_ext}.key"
end
end
Set an environment variable RAILS_CREDENTIALS_ENVIRONMENT to production or staging, depending on the case. For example, Heroku does this with:
heroku config:set -a theapp-staging RAILS_CREDENTIALS_ENVIRONMENT=staging
If on staging or produciton you want to store your key in an environment variable instead of a key file, then simply assign the key to the RAILS_MASTER_KEY environment variable. As documented, this takes precendence over keys stored in files. Note that on your development machine you wouldn't want to have a RAILS_MASTER_KEY set, otherwise the credential files for all environments still get the same key and are thus accessible by everyone that needs to have access to (e.g.) only the development credentials.
Have several files one by each environment in your source code repository o local server disk, works but you will have several files with hardcoded values.
But, if you expect several teams with several requirements on your single app, you will need several development and test environments in order to keep an independent teams which are fully responsible for their Services:
Development
Release / Deployment
Ops (not platform/system administration)
An approach to management this is : externalize your configurations on platforms called: Configurations Manager
This platforms, must have the following features:
key-vaue pair creation by app. Like heroku web dashboard
expose http endpoint to get this variables from remote apps
security
Your rails app must get the variables at the startup stage or instantaneous if your language support hot reload variables.
Here some Configurations Managers:
Zookeeper : http://www.therore.net/java/2015/05/03/distributed-configuration-with-zookeeper-curator-and-spring-cloud-config.html
Consul : https://www.consul.io
Spring Cloud : https://www.baeldung.com/spring-cloud-configuration
Confignet: https://github.com/software-architect-tools/confignet
Whit this approach you will have a platform to management several apps for several environments like heroku web variables creation but more sophisticated.
I have a Rails project PROJECTX, which is hosted on Heroku. For storing production configs and files, I am using a different repository PROJECTX-config. Is it possible to:
clone PROJECTX-config,
remove current config files, and
symlink config files to PROJECTX-config files
Note that this has to be done on Heroku. Also I am aware that Heroku has options to maintain configs using environment variables, but this is not what I am looking for.
Thanks!
No its not possible.
Each dyno gets its own ephemeral filesystem, with a fresh copy of the
most recently deployed code. During the dyno’s lifetime its running
processes can use the filesystem as a temporary scratchpad, but no
files that are written are visible to processes in any other dyno and
any files written will be discarded the moment the dyno is stopped or
restarted. For example, this occurs any time a dyno is replaced due to
application deployment and approximately once a day as part of normal
dyno management.
- https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
Or at least not without a Rube Goldberg machine like setupe where you setup some kind of automation (like a post-commit hook) to merge repo A and repo B and push the result to heroku.
Also I believe that app config should not be present in environment variables, as it is tedious to maintain rather than maintaining a file.
Heroku does not agree here.
The traditional approach for handling such config vars is to put them
under source - in a properties file of some sort. This is an
error-prone process, and is especially complicated for open source
apps which often have to maintain separate (and private) branches with
app-specific configurations.
A better solution is to use environment
variables, and keep the keys out of the code. On a traditional host or
working locally you can set environment vars in your bashrc file. On
Heroku, you use config vars.
- https://devcenter.heroku.com/articles/config-vars
Although you might be overestimating what you actually need to store in ENV vars. You only need to store secrets such as API keys in ENV.
Other non-secret configuration such your settings for various gems can and should be setup in config/initializers.
If you still think using the GUI is that terrible then use YAML files which you parse and use to set the ENV vars:
require 'yaml'
yaml = YAML.load_file(File.join(__dir__, 'conf.yml'))
def create_key(*components)
components.join('_').upcase
end
env_vars = yaml["production"].each_with_object({}) do |(key,value), memo|
key_components = [key]
if value.kind_of? Hash
value.each_pair do |k,v|
memo[create_key(*key_components.dup.push(k))] = v
end
else
memo[create_key(*key_components)] = value
end
end.each do |k,v|
system("heroku config:set #{k}=#{v}")
puts "Setting #{k} = #{v}; #{ $? }"
end
Or you could even store a serialized form (JSON or YAML) in a single env var - there is a total size limit of 32kb though.
a) I'm a ruby in rails beginner developer, and I use windows 7 machine as developement environment...
b) With VirtualBox I just installed, inside the Windows 7 "host", a Linux ubuntu sever "guest", just to run the rails DEVELOPMENT environment ALSO in the linux machine.
c) To do that I configured a virtualbox SHARED FOLDER:
let say I have this shared folder on the host machine (window):
c:\rails\esamiAnatomia
and mounted it on the linux embedded server:
/home/solyaris/host/esamianatomia
d) In this exptended "developement environment" I would like to edit source files with my preferred visual editor on windows (sublime text) and run rails server on linux.
The problem concern database.yml configuration file:
/home/solyaris/host/esamianatomia/config/database.yml
because on Windows I have a database (postgresql) responding port 5433, with specific username/password
but in linux database respond to port 5432, etc.
Questions:
1) It's that "arcgitectural solution ok ? (I mean: developing/editing from a windows 7 host, but running rails server of the linux guest server);
2) There is a way to change/configure database.yml on the fly (I mean: using two different database.yml files: one for the linux machine and anotherone for the window machine) ?
thanks a lot
giorgio
You can technically accomplish 2 if you're not afraid to play around with the guts of Rails. As with any solution that has you accessing internal rails components this may stop working at any time, but fortunately this part of the API is not likely to change often, if ever. Still, use this at your own risk.
Here's how I do it on my projects. First modify your application as follows:
# config/application.rb:
# After require 'rails/all'
require_relative 'db_override'
Then create this new file:
# config/db_override.rb:
case Socket.gethostname
when 'host1'
$db_config = 'config/host1_database.yml'
when 'host2'
$db_config = 'config/host2_database.yml'
else
$db_config = nil # Use the default config/database.yml
end
if $db_config
class DBConfigSelect < Rails::Railtie
initializer "db_config_select", before: "active_record.initialize_database" do
puts "Using custom DB configuration: #{$db_config}"
# Get the existing path configuration
cur_paths = Rails.application.config.paths['config/database'].instance_variable_get :#paths
# Override the default config sources
cur_paths.shift
cur_paths.push $db_config
end
end
end
What you are describing is pretty much the setup that Vagrant is offering, so yeah, you are doing fine, everyone else is also doing it but they didn't set it up themselves (and by that probably get some really nice addons as well, you should take a look at Vagrant).
For your second question: no. Not on the fly. Rails loads the database.yml end then connects to the database with that. When you change it while your Rails server is running, the changes won't get noticed. What you can do however is setup a new environment for your two different machines. Then you can switch between the different environments and depending on the environment, one or the other database gets accessed.