I am deploying my Rails app to GAE, whose codes are stored in github.
Obviously, I need to hide my secret key and database password.
In Heroku, I can set them in environment variables very easily and nicely using Heroku GUI, so it won't appear in any source code or database.
What about GAE?
I cannot set them in app.yaml because:
.gitignore is not an option: Even I hide app.yaml file or alternative json file by .gitignore, I have to save it in my local computer. It means that Only I can deploy, and I have to do backup by myself. This is terrible.
Someone says that I can store secret values in database. But I want to hide database password too.
Any idea?
The most secure way to store this info is using project metadata. On a Flexible/ManagedVM environment you can access the metadata via a simple http request.
From the google blog post:
With Compute Engine, Container Engine, and Managed VMs, there is a magic URL you can CURL to get metadata.
ManagedVMs are the old name for what is now called 'AppEngine Flexible Environment'. Since you say you are using Ruby on App Engine you must be using Flexible/ManagedVMs. Therefore you should be able to use these 'magic URLs'.
So to get an application secret called mysecret in Ruby you might do:
Net::HTTP.get(
URI.parse('http://metadata.google.internal/computeMetadata/v1/project/attributes/mysecret'))
(For #joshlf) Here's how to access project metadata on AppEngine Standard Environment in Python:
# Note that the code will not work on dev_appserver,
# you will need to switch to some other mechanism
# for configuration in that environment
# Specifically the project_id will resolve to something
# compute engine API will treat as invalid
from google.appengine.api import app_identity
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
compute = discovery.build(
'compute', 'v1', credentials=GoogleCredentials.get_application_default())
def get_project_metadata(metadata_key):
project_id = app_identity.get_application_id()
project = compute.projects().get(project=project_id).execute()
for entry in project['commonInstanceMetadata']['items']:
if entry['key'] == metadata_key:
return entry['value']
return None
get_project_metadata('my_key')
I addressed this problem in an answer to a similar question. Essentially, you can create a credentials.yaml file alongside your app.yaml and import it in app.yaml. This will allow you to specify your credentials as ENV variables while retaining the ability to ignore the file in git. The includes: tag allows you to import an array of files in your app.yaml.
Example app.yaml:
runtime: go
api_version: go1
env_variables:
FIST_VAR: myFirstVar
includes:
- credentials.yaml
credentials.yaml:
env_variables:
SECOND_VAR: mySecondVar
API_KEY: key-123
Related
I have a dbt project and I'm using ad_reporting model.
all the sources are working good, except than google_ads.
and that's because I don't have google_ads schema, instead I have google_ads_us schema.
I tried to change it in main dbt_project.yml file:
vars:
google_ads_schema: google_ads_us
but still when I'm running this command:
dbt run --select ad_reporting
I keep get this error:
Database Error in model stg_google_ads (models\stg_google_ads.sql)
SQL compilation error:
Object 'DATABASE.HISTORY_GOOGLE_ADS.GOOGLE_ADS__URL_AD_ADAPTER' does not exist or not
authorized.
compiled SQL at target\run\ad_reporting\models\stg_google_ads.sql
the HISTORY_ prefix is from profiles.yml file (SCHEMA).
Anyone know where else I need to change the schema?
For the package to find the specified vars it has defined, you will need to do four things, all specified in the docs:
1. Make sure that ad_reporting for google_ads is enabled -->
# inside your dbt_project.yml
vars:
ad_reporting__google_ads_enabled: true
2. Also, make sure that you let dbt know where to look for the google_ads raw data:
# inside your dbt_project.yml
vars:
google_ads_database: your_database_name # in our case, `raw`
google_ads_schema: your_schema_name # in our case, `google_ads`
3. Define the schemas where dbt will build the models for both google_ads and google_ads_source -->
# inside your dbt_project.yml
models:
google_ads:
+schema: my_new_schema_name # leave blank for just the target_schema
google_ads_source:
+schema: my_new_schema_name # leave blank for just the target_schema
Since, as according to the docs:
"By default this package will build the Google Ads staging models within a schema titled (<target_schema> + _stg_google_ads) and the Google Ads final models with a schema titled (<target_schema> + _google_ads) in your target database."
4. Specify which API you're going to use (Google Adwords or Google Ads):
# inside your dbt_project.yml
vars:
api_source: google_ads # adwords by default and is case sensitive!
I will host my RoR API-App in the Google App-Engine.
Everything works so far, but I have to store usernames, passwords and keys (e.g. Database user/password) in plain text in the app.yaml. this is just stupid, so I will never be able to push this to my git repo! Usually I store stuff like this in an env variable and use them in my application.
But I did not find a way to set or access env variables.
Is there a way or an alternative to do so?
I did it!
For local development, I just set my env as usual.
If the mode is Production I load them from the Google Datastore all key values pairs and set them as an env variable.
I do this in an Initializer, to do so just create a file in YourApp/config/initializers/ and put the code in it! Just Create new Entities copy the Kind name in the code and set your project id. As your App is hosted in Google it should have access to the datastore (You need to set the right in the IAM-Manager)
require "google/cloud/datastore"
# Load the enviroment variables from the google datastore!
if Rails.env == "production"
data_store = Google::Cloud::Datastore.new(
project_id: 'YOUR_PROJECT_ID'
)
query = data_store.query "YOUR_KIND_NAME"
results = data_store.run query
puts "Set custom env variables!"
# Set each result as an env variable
results[0].properties.to_h.each do |key, value|
ENV[key]= value
end
end
As mentioned in Best practices for managing credentials, you may use an environment variable pointing to credentials outside of the application's source code, such as Cloud Key Management Service. I also recommend to take a look at Secret management with Cloud KMS documentation which explains solutions when you choose a secret management.
For the Rails 3 application I'm writing, I am considering reading some of the configuration data from XML, YAML or JSON files on the local filesystem.
The point is: where should I put those files? Is there any default location in Rails apps where to store this kind of content?
As a side note, my app is deployed on Heroku.
What I always do is:
If the file is a general configuration file: I create a YAML file in the directory /config with one upper class key per environment
If I have a file for each environment (big project): I create one YAML per environment and store them in /config/environments/
Then I create an initializer where I load the YAML, I symbolize the keys of the config hash and assign it to a constant like APP_CONFIG
I will usually adopt this method :
a config/config.yml
development:
another_key: "test"
app_name: "My App"
test:
another_key: "test"
production:
prova: "ciao"
then create a ostruct in a initializer
#config/initializer/load_config.rb
require 'ostruct'
config = OpenStruct.new(YAML.load_file("#{RAILS_ROOT}/config/config.yml"))
::AppSetting = OpenStruct.new(config.send(RAILS_ENV))
No DB table, per environment setup and you could retrive info in a simple way
AppSetting.another_key
AppSetting.app_name
here a reference
have a nice day!
You can also include it in a model so you can call Settings.var_name from anywhere in your app and it will parse the file for the right environment.
With settingslogic gem:
class Settings < Settingslogic
source "#{Rails.root}/config/settings.yml"
namespace Rails.env
end
Rails creates a config directory by default, containing a lot of configuration info for your application, including the database and environment information. I think that's a logical first place to consider.
A second choice would be the app directory, which contains all the models, views and controllers for the application, but I think of that directory as containing executable code and its templates, so I'd go with the config directory, personally.
Am I crazy, or is it a bad idea to keep my SMTP username and password for ActionMailer in the actual (development/production) config file? It seems like I should store it an encrypted place, or at the very minimum, exclude it from my Mercurial pushes.
Right now, I'm just removing the password from my source file before performing a push, but there's got to be a smarter way than the one I'm using. :)
Perhaps I should store it in my database as another user (which is already stored with encrypted passwords) and fetch it programatically?
Use an application configuration file that is not stored in your repository for storing sensitive information. Here is how I've done it:
Add an app_config.yml in your config directory. Its contents would look like this:
smtp_password: kl240jvfslkr32rKgjlk
some_other_password: 34hg9r0j0g402jg
and_so_on: lkn$#gJkjgsFLK4gaj
Add a preinitializer.rb in your config directory with the following contents:
require 'yaml'
APP_CONFIG = YAML.load(File.read(RAILS_ROOT + "/config/app_config.yml"))
Substitute your passwords for values in the APP_CONFIG variable, like so:
smtp_password = kl240jvfslkr32rKgjlk # old version
smtp_password = APP_CONFIG['smtp_password'] # new version
Make sure you don't include app_config.yml in your repository, though you may want to create an example file that is checked in, just to show a sample of what should be in it. When you deploy your application, make sure that app_config.yml is stored on the server. If you're using a standard Capistrano deployment, put the file in the shared folder and update your deployment task to create a symlink to it in the current release's directory.
Jimmy's answer is perfect (+1), I would also note that Github has recommended .gitignore files for every language and the Rails one is here Note that it includes config/*.yml so that no config/yml file is in the respository to begin with. Probably a good move.
Use Capistrano to ask for these things upon deploy:setup the same way you should be doing for your database stuff:
task :my_silly_task do
sendgrid_password = Capistrano::CLI.password_prompt("Sendgrid password: ")
require 'yaml'
spec = {... whatever yaml you need -- probably what Jimmy said...}
run "mkdir -p #{shared_path}/config"
put(spec.to_yaml, "#{shared_path}/config/mailer_config.yml")
end
I need to store app specific configuration in rails. But it has to be:
reachable in any file (model, view, helpers and controllers
environment specified (or not), that means each environment can overwrite the configs specified in environment.rb
I've tried to use environment.rb and put something like
USE_USER_APP = true
that worked to me but when trying to overwrite it in a specific environment it wont work because production.rb, for instance, seems to be inside the Rails:Initializer.run block.
So, anyone?
Look at Configatron: http://github.com/markbates/configatron/tree/master
I have yet to use it, but he's actively developing it now, and looks quite nice.
I was helping a friend set up the solution mentioned by Ricardo yesterday. We hacked it a bit by loading the YAML file with something similar to this (going from memory here):
require 'ostruct'
require 'yaml'
require 'erb'
#config = OpenStruct.new(YAML.load_file("#{RAILS_ROOT}/config/config.yml"))
config = OpenStruct.new(YAML.load(ERB.new(File.read("#{RAILS_ROOT}/config/config.yml")).result))
env_config = config.send(RAILS_ENV)
config.common.update(env_config) unless env_config.nil?
::AppConfig = OpenStruct.new(config.common)
This allowed him to embed Ruby code in the config, like in Rhtml:
development:
path_to_something: <%= RAILS_ROOT %>/config/something.yml
The most basic thing to do is to set a class variable from your environment.rb. I've done this for Google Analytics. Essentially I want a different key depending on which environment I'm in so development or staging don't skew the metrics.
This is how I did it.
In lib/analytics/google_analytics.rb:
module Analytics
class GoogleAnalytics
##account_id = nil
cattr_accessor :account_id
end
end
And then in environment.rb or in environments/production.rb or any of the other environment files:
Analytics::GoogleAnalytics.account_id = "xxxxxxxxx"
Then anywhere you ned to reference, say the default layout with the Google Analytics JavaScript, it you just call Analytics::GoogleAnalytics.account_id.
I found a good way here
Use environment variables. Heroku uses this. Remember that if you keep configuration in the codebase, anyone with access to the code has access to any secret configuration (aws api keys, gateway api keys, etc).
daemontool's envdir is a good tool for setting configuration, I'm pretty sure that's what Heroku uses to give application their environment variables.
I have used Rails Settings Cached.
It is very simple to use, keeps your configuration values cached and allows you to change them dynamically.