I have a dbt project and I'm using ad_reporting model.
all the sources are working good, except than google_ads.
and that's because I don't have google_ads schema, instead I have google_ads_us schema.
I tried to change it in main dbt_project.yml file:
vars:
google_ads_schema: google_ads_us
but still when I'm running this command:
dbt run --select ad_reporting
I keep get this error:
Database Error in model stg_google_ads (models\stg_google_ads.sql)
SQL compilation error:
Object 'DATABASE.HISTORY_GOOGLE_ADS.GOOGLE_ADS__URL_AD_ADAPTER' does not exist or not
authorized.
compiled SQL at target\run\ad_reporting\models\stg_google_ads.sql
the HISTORY_ prefix is from profiles.yml file (SCHEMA).
Anyone know where else I need to change the schema?
For the package to find the specified vars it has defined, you will need to do four things, all specified in the docs:
1. Make sure that ad_reporting for google_ads is enabled -->
# inside your dbt_project.yml
vars:
ad_reporting__google_ads_enabled: true
2. Also, make sure that you let dbt know where to look for the google_ads raw data:
# inside your dbt_project.yml
vars:
google_ads_database: your_database_name # in our case, `raw`
google_ads_schema: your_schema_name # in our case, `google_ads`
3. Define the schemas where dbt will build the models for both google_ads and google_ads_source -->
# inside your dbt_project.yml
models:
google_ads:
+schema: my_new_schema_name # leave blank for just the target_schema
google_ads_source:
+schema: my_new_schema_name # leave blank for just the target_schema
Since, as according to the docs:
"By default this package will build the Google Ads staging models within a schema titled (<target_schema> + _stg_google_ads) and the Google Ads final models with a schema titled (<target_schema> + _google_ads) in your target database."
4. Specify which API you're going to use (Google Adwords or Google Ads):
# inside your dbt_project.yml
vars:
api_source: google_ads # adwords by default and is case sensitive!
Related
I will host my RoR API-App in the Google App-Engine.
Everything works so far, but I have to store usernames, passwords and keys (e.g. Database user/password) in plain text in the app.yaml. this is just stupid, so I will never be able to push this to my git repo! Usually I store stuff like this in an env variable and use them in my application.
But I did not find a way to set or access env variables.
Is there a way or an alternative to do so?
I did it!
For local development, I just set my env as usual.
If the mode is Production I load them from the Google Datastore all key values pairs and set them as an env variable.
I do this in an Initializer, to do so just create a file in YourApp/config/initializers/ and put the code in it! Just Create new Entities copy the Kind name in the code and set your project id. As your App is hosted in Google it should have access to the datastore (You need to set the right in the IAM-Manager)
require "google/cloud/datastore"
# Load the enviroment variables from the google datastore!
if Rails.env == "production"
data_store = Google::Cloud::Datastore.new(
project_id: 'YOUR_PROJECT_ID'
)
query = data_store.query "YOUR_KIND_NAME"
results = data_store.run query
puts "Set custom env variables!"
# Set each result as an env variable
results[0].properties.to_h.each do |key, value|
ENV[key]= value
end
end
As mentioned in Best practices for managing credentials, you may use an environment variable pointing to credentials outside of the application's source code, such as Cloud Key Management Service. I also recommend to take a look at Secret management with Cloud KMS documentation which explains solutions when you choose a secret management.
In a Ruby on Rails application, where would the most logical place be to put a "file flag."
I am attempting to externalize configuration and allow the presence of a file to be the deciding factor on whether or not something shows on the webapp.
Right now, I have a file here:
lib/
deployment_enabled
Model deployment.rb
class Deployment...
...
def deployment_enabled?
Dir["#{Rails.root}/lib/deployment_enabled"].any?
end
end
Now this works of course, but i'm not sure this follows the MVC paradigms, since the lib directory should consist of scripts. I could put it in config, but again - not sure it belongs there as rails uses this for rails specific configuration, not necessarily the webapp.
I could of course put this in our database, but that require a new table to be created, and that seems unnecessary.
Where's the most logical place to put this flag file? Does Rails have a directory that's created during the generation to put these sort of files?
I suggest using the Rails tmp directory for this purpose. Then:
File.exist?("#{Rails.root}/tmp/deployment_enabled")
Phusion Passenger use this kind of mechanism too.
https://www.phusionpassenger.com/library/walkthroughs/basics/ruby/reloading_code.html#tmp-always_restart-txt
I recommend that you follow the Twelve-Factor App guidelines and keep your code separate from your configuration. In this case you are really talking about a simple boolean value, and the presence of the file is just the mechanism you use to define the value. This should be done instead through an environment variable.
Something like:
DEPLOYMENT_ENABLED=1 RAILS_ENV=production rails server
You would then use an initializer in Rails to detect the value:
# config/initializers/deployment.rb
foo if ENV['DEPLOYMENT_ENABLED']
The value can still be modified at runtime, e.g., ENV['DEPLOYMENT_ENABLED'] = 0.
I am deploying my Rails app to GAE, whose codes are stored in github.
Obviously, I need to hide my secret key and database password.
In Heroku, I can set them in environment variables very easily and nicely using Heroku GUI, so it won't appear in any source code or database.
What about GAE?
I cannot set them in app.yaml because:
.gitignore is not an option: Even I hide app.yaml file or alternative json file by .gitignore, I have to save it in my local computer. It means that Only I can deploy, and I have to do backup by myself. This is terrible.
Someone says that I can store secret values in database. But I want to hide database password too.
Any idea?
The most secure way to store this info is using project metadata. On a Flexible/ManagedVM environment you can access the metadata via a simple http request.
From the google blog post:
With Compute Engine, Container Engine, and Managed VMs, there is a magic URL you can CURL to get metadata.
ManagedVMs are the old name for what is now called 'AppEngine Flexible Environment'. Since you say you are using Ruby on App Engine you must be using Flexible/ManagedVMs. Therefore you should be able to use these 'magic URLs'.
So to get an application secret called mysecret in Ruby you might do:
Net::HTTP.get(
URI.parse('http://metadata.google.internal/computeMetadata/v1/project/attributes/mysecret'))
(For #joshlf) Here's how to access project metadata on AppEngine Standard Environment in Python:
# Note that the code will not work on dev_appserver,
# you will need to switch to some other mechanism
# for configuration in that environment
# Specifically the project_id will resolve to something
# compute engine API will treat as invalid
from google.appengine.api import app_identity
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
compute = discovery.build(
'compute', 'v1', credentials=GoogleCredentials.get_application_default())
def get_project_metadata(metadata_key):
project_id = app_identity.get_application_id()
project = compute.projects().get(project=project_id).execute()
for entry in project['commonInstanceMetadata']['items']:
if entry['key'] == metadata_key:
return entry['value']
return None
get_project_metadata('my_key')
I addressed this problem in an answer to a similar question. Essentially, you can create a credentials.yaml file alongside your app.yaml and import it in app.yaml. This will allow you to specify your credentials as ENV variables while retaining the ability to ignore the file in git. The includes: tag allows you to import an array of files in your app.yaml.
Example app.yaml:
runtime: go
api_version: go1
env_variables:
FIST_VAR: myFirstVar
includes:
- credentials.yaml
credentials.yaml:
env_variables:
SECOND_VAR: mySecondVar
API_KEY: key-123
I am trying to use an ECS Ops-Works layer to manage some automation, but I cannot figure out how to set up those instances to set up the ecs.config to contain my private Docker repository credentials, as one would do manually if managing ec2 jobs directly. I think I need to somehow use some custom Chef to override the setup recipe to load my template rather than the default template for that file, but I am new to Chef so how to do this is unclear.
So to restate the problem, you want to modify this template in the opsworks_ecs::setup recipe:
template "ecs.config" do
path "/etc/ecs/ecs.config"
source "ecs.config.erb"
owner "root"
group "root"
mode 0644
end
I don't know how you are 'calling' this but I'll assume for now that you're either putting this recipe directly in your run_list and/or calling it explicitly with include_recipe "opsworks_ecs::setup"
In that case, write a wrapper cookbook. If you work for "Acme, Org" it might be something like acme_opsworks_ecs::setup.
acme_opsworks_ecs/metadata.rb should at least have:
name 'acme_opsworks_ecs'
version '0.0.1'
depends 'opsworks_ecs'
acme_opsworks_ecs/recipes/setup.rb should look like:
include_recipe "opsworks_ecs::setup"
resources(template: "ecs.config").cookbook(cookbook_name)
acme_opsworks_ecs/templates/default/ecs.config.erb is also required
/* add your own template content to this file -- copy theirs and edit */
That should allow you to fix it. What you're doing is using Chef's two-pass parser so that the opsworks recipe defines the template resource, then you re-open it and edit it, before it actually runs. Now, wherever you have referenced opsworks_ecs::setup in your run_list or include_recipe calls, replace that recipe with acme_opsworks_ecs::setup.
If you don't directly call opsworks_ecs::setup, then wrap the opsworks recipe(s) that you do call instead following the same pattern.
If you google "chef-rewind" you can find more information about this kind of pattern of using chef. Note that the syntax that I used is built-in to chef though and does not require a custom gem install or chef_rewind resource/definition to use, so it will be substantially simpler to use the syntax in this answer.
For the Rails 3 application I'm writing, I am considering reading some of the configuration data from XML, YAML or JSON files on the local filesystem.
The point is: where should I put those files? Is there any default location in Rails apps where to store this kind of content?
As a side note, my app is deployed on Heroku.
What I always do is:
If the file is a general configuration file: I create a YAML file in the directory /config with one upper class key per environment
If I have a file for each environment (big project): I create one YAML per environment and store them in /config/environments/
Then I create an initializer where I load the YAML, I symbolize the keys of the config hash and assign it to a constant like APP_CONFIG
I will usually adopt this method :
a config/config.yml
development:
another_key: "test"
app_name: "My App"
test:
another_key: "test"
production:
prova: "ciao"
then create a ostruct in a initializer
#config/initializer/load_config.rb
require 'ostruct'
config = OpenStruct.new(YAML.load_file("#{RAILS_ROOT}/config/config.yml"))
::AppSetting = OpenStruct.new(config.send(RAILS_ENV))
No DB table, per environment setup and you could retrive info in a simple way
AppSetting.another_key
AppSetting.app_name
here a reference
have a nice day!
You can also include it in a model so you can call Settings.var_name from anywhere in your app and it will parse the file for the right environment.
With settingslogic gem:
class Settings < Settingslogic
source "#{Rails.root}/config/settings.yml"
namespace Rails.env
end
Rails creates a config directory by default, containing a lot of configuration info for your application, including the database and environment information. I think that's a logical first place to consider.
A second choice would be the app directory, which contains all the models, views and controllers for the application, but I think of that directory as containing executable code and its templates, so I'd go with the config directory, personally.