I will host my RoR API-App in the Google App-Engine.
Everything works so far, but I have to store usernames, passwords and keys (e.g. Database user/password) in plain text in the app.yaml. this is just stupid, so I will never be able to push this to my git repo! Usually I store stuff like this in an env variable and use them in my application.
But I did not find a way to set or access env variables.
Is there a way or an alternative to do so?
I did it!
For local development, I just set my env as usual.
If the mode is Production I load them from the Google Datastore all key values pairs and set them as an env variable.
I do this in an Initializer, to do so just create a file in YourApp/config/initializers/ and put the code in it! Just Create new Entities copy the Kind name in the code and set your project id. As your App is hosted in Google it should have access to the datastore (You need to set the right in the IAM-Manager)
require "google/cloud/datastore"
# Load the enviroment variables from the google datastore!
if Rails.env == "production"
data_store = Google::Cloud::Datastore.new(
project_id: 'YOUR_PROJECT_ID'
)
query = data_store.query "YOUR_KIND_NAME"
results = data_store.run query
puts "Set custom env variables!"
# Set each result as an env variable
results[0].properties.to_h.each do |key, value|
ENV[key]= value
end
end
As mentioned in Best practices for managing credentials, you may use an environment variable pointing to credentials outside of the application's source code, such as Cloud Key Management Service. I also recommend to take a look at Secret management with Cloud KMS documentation which explains solutions when you choose a secret management.
Related
I have a project configured to use Rails encrypted secrets. Everything works fine until I try to access a secret within the production.rb environment file.
I found that if I try to access something like Rails.application.secrets.smtp_user_name within the configure block it wipes out all of the encrypted secrets (I'm only left with things in secrets.yml ... which I'm not using). Example:
Loading production environment (Rails 5.1.2)
irb(main):001:0> Rails.application.secrets
=> {:secret_key_base=>nil, :secret_token=>nil}
If I remove the attempt to access the secrets it works correctly:
irb(main):001:0> Rails.application.secrets
=> {:secret_key_base=>"...", :smtp_user_name=>"...", :smtp_password=>"...", :secret_token=>nil}
I'm currently working around it by using two configure blocks in production.rb as follows:
# This is hacky, it needs to come before the second configure block where
# the encrypted secrets are used.
Rails.application.configure do
config.read_encrypted_secrets = true
end
Rails.application.configure do
... stuff that uses Rails.application.secrets, like ActionMailer
end
Anybody else faced this and possibly have a more correct way to work around it?
It makes sense why this is happening (Rails doesn't know to load the encrypted secrets because we haven't told it to you), but I'm thinking there must be a better way to deal with it.
Update
This nailed me again 9 months later. To be clear, if you reference Rails.application.secrets BEFORE calling config.read_encrypted_secrets = true you will cache empty secrets and not be able to access any of the values in secrets.yml.enc!
In my case I had tried to configure Paperclip S3 credentials in application.rb while my config.read_encrypted_secrets = true was set in production.rb. Result was devise.rb blowing up trying to read a secret for the key base, all because in application.rb I had effectively cached nil secrets.
here is bug report related to your issue:
https://github.com/rails/rails/issues/30362#issuecomment-326821656
In general even if you have all things set up properly you need check also order how your application loads secrets. If your application ask first for Rails.application.secrets and then set proper flag... Rails.application.secrets will cache version without secrets... and Secrets from secret.yml.enc will not be merge.
I am deploying my Rails app to GAE, whose codes are stored in github.
Obviously, I need to hide my secret key and database password.
In Heroku, I can set them in environment variables very easily and nicely using Heroku GUI, so it won't appear in any source code or database.
What about GAE?
I cannot set them in app.yaml because:
.gitignore is not an option: Even I hide app.yaml file or alternative json file by .gitignore, I have to save it in my local computer. It means that Only I can deploy, and I have to do backup by myself. This is terrible.
Someone says that I can store secret values in database. But I want to hide database password too.
Any idea?
The most secure way to store this info is using project metadata. On a Flexible/ManagedVM environment you can access the metadata via a simple http request.
From the google blog post:
With Compute Engine, Container Engine, and Managed VMs, there is a magic URL you can CURL to get metadata.
ManagedVMs are the old name for what is now called 'AppEngine Flexible Environment'. Since you say you are using Ruby on App Engine you must be using Flexible/ManagedVMs. Therefore you should be able to use these 'magic URLs'.
So to get an application secret called mysecret in Ruby you might do:
Net::HTTP.get(
URI.parse('http://metadata.google.internal/computeMetadata/v1/project/attributes/mysecret'))
(For #joshlf) Here's how to access project metadata on AppEngine Standard Environment in Python:
# Note that the code will not work on dev_appserver,
# you will need to switch to some other mechanism
# for configuration in that environment
# Specifically the project_id will resolve to something
# compute engine API will treat as invalid
from google.appengine.api import app_identity
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
compute = discovery.build(
'compute', 'v1', credentials=GoogleCredentials.get_application_default())
def get_project_metadata(metadata_key):
project_id = app_identity.get_application_id()
project = compute.projects().get(project=project_id).execute()
for entry in project['commonInstanceMetadata']['items']:
if entry['key'] == metadata_key:
return entry['value']
return None
get_project_metadata('my_key')
I addressed this problem in an answer to a similar question. Essentially, you can create a credentials.yaml file alongside your app.yaml and import it in app.yaml. This will allow you to specify your credentials as ENV variables while retaining the ability to ignore the file in git. The includes: tag allows you to import an array of files in your app.yaml.
Example app.yaml:
runtime: go
api_version: go1
env_variables:
FIST_VAR: myFirstVar
includes:
- credentials.yaml
credentials.yaml:
env_variables:
SECOND_VAR: mySecondVar
API_KEY: key-123
I'm trying to display SendGrid email statistics inside my Rails application.
So far, I've been able to display them in API's return format (e.g. [{"delivered"=>9, "unsubscribes"=>0, "repeat_bounces"=>2,....), but that's only if I put the plaintext username and password into my controller's private method because I cannot seem to use 'SENDGRID_USERNAME' and the 'SENDGRID_PASSWORD' environment variables. What's a guy to do?
In summation:
NewslettersController:
before_action :set_client, only: :index
private
def set_client
#client = SendGridWebApi::Client.new('the plaintext username', 'the plaintext password')
end
Newsletters Index View
<p><%= #client.stats.get %></p>
Is there another place where I can set #client where my view can get to it and my plaintext password can be hidden from the outside world? preferably a place I can put in secrets.yml?
#client = SendGridWebApi::Client.new('SENDGRID_USERNAME', 'SENDGRID_PASSWORD')
Returns an incorrect username and password, even though I got the plaintext username and password I'm using for testing from those variables, so I know that they're set to the correct ones.
Or, is there a way to use environment variables in either my controller or my view?
I'm able to set the variables in the Rails Console and then use them, but since they don't get saved I can't access them afterwords in my actual application.
It is better to add your secrets to system environments variables by using
export SENDGRID_USERNAME=my-super-secret-username
export SENDGRID_PASSWORD=my-super-secret-password
Or alternatively add environment variables to bashrc file. Open bashrc file in your favorite text editor i.e. nano, vim, vi etc
sudo nano ~/.bashrc
and than add/save environment variable. After that you need to touch bashrc file using source command
source ~/.bashrc
In controller, access environment variable using ENV hash like
def set_client
#client = SendGridWebApi::Client.new(ENV['SENDGRID_USERNAME'], ENV['SENDGRID_PASSWORD'])
end
In case you want to control your secrets from within your rails application, You can create a new initializer file and define environment variables there like
# config/initializer/environment_variables.rb
ENV['SENDGRID_USERNAME']='my-super-secret-username'
ENV['SENDGRID_PASSWORD']='my-super-secret-password'
Keeping env in initilizer is not a good approach as your secrets are still present in your code in plain text format and may be exposed. You must add this file to .gitignore file.
Although my solution is not secrets.yml, this works fine.
# config/initializers/my-supersecret-vars.rb
ENV['SENDGRID_USERNAME'] = 'my_super_secret_username'
ENV['SENDGRID_PASSWORD'] = 'my_super_secret_password'
Just remember to add it on your .gitignore
Am I crazy, or is it a bad idea to keep my SMTP username and password for ActionMailer in the actual (development/production) config file? It seems like I should store it an encrypted place, or at the very minimum, exclude it from my Mercurial pushes.
Right now, I'm just removing the password from my source file before performing a push, but there's got to be a smarter way than the one I'm using. :)
Perhaps I should store it in my database as another user (which is already stored with encrypted passwords) and fetch it programatically?
Use an application configuration file that is not stored in your repository for storing sensitive information. Here is how I've done it:
Add an app_config.yml in your config directory. Its contents would look like this:
smtp_password: kl240jvfslkr32rKgjlk
some_other_password: 34hg9r0j0g402jg
and_so_on: lkn$#gJkjgsFLK4gaj
Add a preinitializer.rb in your config directory with the following contents:
require 'yaml'
APP_CONFIG = YAML.load(File.read(RAILS_ROOT + "/config/app_config.yml"))
Substitute your passwords for values in the APP_CONFIG variable, like so:
smtp_password = kl240jvfslkr32rKgjlk # old version
smtp_password = APP_CONFIG['smtp_password'] # new version
Make sure you don't include app_config.yml in your repository, though you may want to create an example file that is checked in, just to show a sample of what should be in it. When you deploy your application, make sure that app_config.yml is stored on the server. If you're using a standard Capistrano deployment, put the file in the shared folder and update your deployment task to create a symlink to it in the current release's directory.
Jimmy's answer is perfect (+1), I would also note that Github has recommended .gitignore files for every language and the Rails one is here Note that it includes config/*.yml so that no config/yml file is in the respository to begin with. Probably a good move.
Use Capistrano to ask for these things upon deploy:setup the same way you should be doing for your database stuff:
task :my_silly_task do
sendgrid_password = Capistrano::CLI.password_prompt("Sendgrid password: ")
require 'yaml'
spec = {... whatever yaml you need -- probably what Jimmy said...}
run "mkdir -p #{shared_path}/config"
put(spec.to_yaml, "#{shared_path}/config/mailer_config.yml")
end
I need to store app specific configuration in rails. But it has to be:
reachable in any file (model, view, helpers and controllers
environment specified (or not), that means each environment can overwrite the configs specified in environment.rb
I've tried to use environment.rb and put something like
USE_USER_APP = true
that worked to me but when trying to overwrite it in a specific environment it wont work because production.rb, for instance, seems to be inside the Rails:Initializer.run block.
So, anyone?
Look at Configatron: http://github.com/markbates/configatron/tree/master
I have yet to use it, but he's actively developing it now, and looks quite nice.
I was helping a friend set up the solution mentioned by Ricardo yesterday. We hacked it a bit by loading the YAML file with something similar to this (going from memory here):
require 'ostruct'
require 'yaml'
require 'erb'
#config = OpenStruct.new(YAML.load_file("#{RAILS_ROOT}/config/config.yml"))
config = OpenStruct.new(YAML.load(ERB.new(File.read("#{RAILS_ROOT}/config/config.yml")).result))
env_config = config.send(RAILS_ENV)
config.common.update(env_config) unless env_config.nil?
::AppConfig = OpenStruct.new(config.common)
This allowed him to embed Ruby code in the config, like in Rhtml:
development:
path_to_something: <%= RAILS_ROOT %>/config/something.yml
The most basic thing to do is to set a class variable from your environment.rb. I've done this for Google Analytics. Essentially I want a different key depending on which environment I'm in so development or staging don't skew the metrics.
This is how I did it.
In lib/analytics/google_analytics.rb:
module Analytics
class GoogleAnalytics
##account_id = nil
cattr_accessor :account_id
end
end
And then in environment.rb or in environments/production.rb or any of the other environment files:
Analytics::GoogleAnalytics.account_id = "xxxxxxxxx"
Then anywhere you ned to reference, say the default layout with the Google Analytics JavaScript, it you just call Analytics::GoogleAnalytics.account_id.
I found a good way here
Use environment variables. Heroku uses this. Remember that if you keep configuration in the codebase, anyone with access to the code has access to any secret configuration (aws api keys, gateway api keys, etc).
daemontool's envdir is a good tool for setting configuration, I'm pretty sure that's what Heroku uses to give application their environment variables.
I have used Rails Settings Cached.
It is very simple to use, keeps your configuration values cached and allows you to change them dynamically.