Fog-google doesn't find credentials - ruby-on-rails

I have a Rails 5 application with Carrierwave. I would like to use fog-google gem but I cannot set it up because fog cannot retrieve the credentials.
I created a .fog file in my application root populated this way:
default:
google_project: XXXX-website-cdn
google_client_email: XXXX#XXXX-website-cdn.iam.gserviceaccount.com
google_json_key_location: google-storage-cdn.json
I then tried to run pry as mentioned in the guidelines, but it doesn't get the credentials.
[3] pry(main)> connection = Fog::Compute::Google.new
ArgumentError: Missing required arguments: google_project
from /Users/ab/.rvm/gems/ruby-2.3.1/gems/fog-core-1.43.0/lib/fog/core/service.rb:244:in `validate_options'
Infact:
[4] pry(main)> Fog.credentials
=> {}
Where do I tell fog to get credentials from the .fog file?
I don't know if it might be useful to know that I'm using Figaro gem to manage my secrets.

QUICK SOLUTION
Put the .fog file in the root of the server (or your computer), not the one of the app.
This is pretty bad, but it's the first I found while quickly looking to solve the problem.
RIGHT SOLUTION
If you use google_json_key_location: google-storage-cdn.json Rails will look into / folder of the current server (your computer if you are working locally). In order to look into the application folder you need to use a Rails helper.
Rails.root.join( 'google-storage-cdn.json' )
# return /path/to/your/app/google-storage-cdn.json

I was looking for a while for a solution on how to avoid putting this .fog file into my home directory as it makes completely no sense. Up to the point of writing this comment, the official github documentation isn't updated. However, there is an open issue on fog-google github repo that demonstrates how to acheive it.
config/initializers/carrierwave.rb
CarrierWave.configure do |config|
config.fog_provider = 'fog/google'
config.fog_credentials = {
provider: 'Google',
google_project: Rails.application.secrets.google_cloud_storage_project_name,
google_json_key_string: Rails.application.secrets.google_cloud_storage_credential_content
# can optionally use google_json_key_location if using an actual file;
# however, I am using **Heroku** where you can't store physical files unless you
# check them into the repo (and you don't want to do that with service account credentials!)
}
config.fog_directory = Rails.application.secrets.google_cloud_storage_bucket_name
end
config/secrets.yml
development:
google_cloud_storage_project_name: your-project-name
google_cloud_storage_credential_content: '{
"type": "service_account",
"project_id": "your-project-name",
"private_key_id": "REDACTED",
"private_key": "-----BEGIN PRIVATE KEY-----REDACTED-----END PRIVATE KEY-----\n",
"client_email": "REDACTED#your-project-name.iam.gserviceaccount.com",
"client_id": "REDACTED",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/REDACTED%40your-project-name.iam.gserviceaccount.com"
}'
google_cloud_storage_bucket_name: your-bucket-name
All credit goes to the poster of the solution cireficc

Related

ENV variables when doing Rails commands (migration, others)

Using Carrierwave and fog and everything working fine with AWS but when I try and do a migration and some other rails commands I get:
lib/fog/core/service.rb:244:in validate_options: Missing required arguments: aws_access_key_id, aws_secret_access_key (ArgumentError)
This also happens with the Rails console. I think for some reason rails is not able to access my ENV variables for some reason? But it works when running as part of a Rails server...
Any thoughts on why? aws key defined as below:
CarrierWave.configure do |config|
config.fog_provider = 'fog/aws'
config.fog_credentials = {
provider: 'AWS',
aws_access_key_id: ENV['AWS_ACCESS_KEY'],
aws_secret_access_key: ENV['AWS_SECRET'],
region: 'eu-west-2'
}
config.fog_directory = 'images' # bucket name
config.cache_dir = "#{Rails.root}/tmp/uploads" # To let CarrierWave work on heroku
end
Not an answer to the above question but as OP has asked again for any advice..
Stop using ENV variables in development. Create a secrets.yml file, and you'll be able to access these values in your project. Make sure you add this to your .gitignore file as committing this is obviously not a good idea.
A very brief, succinct runthrough of how to use secrets:
https://richonrails.com/articles/the-rails-4-1-secrets-yml-file

Authenticating with service account (email & key) in fog-google

I keep getting Missing required arguments: google_storage_access_key_id, google_storage_secret_access_key. I understand that I am supposed to put my credential "in a /.fog" file, but I don't quite understand how that's supposed to work in the context of a Rails app. Can someone elaborate on how to configure this? I have tried passing the settings in an initializer (as suggested here), but they don't seem to get recognized in the validate_options method.
config/initializers/fog.rb
GoogleStorage = Fog::Storage.new(
provider: 'Google',
google_project: 'xxxxxxxxxxxxx',
google_client_email: 'xxxxxxxxxxxxx-yyyyyyyyyyyyy#developer.gserviceaccount.com',
google_key_location: 'private/google-cloud-service-key.p12'
)
Turns out this is not currently possible with the fog-google gem. See this Github issue. I will update this answer when the gem is updated to handle this authentication strategy.
Use the Figaro Gem instead to handle any ENV vars you want to store and user throughout the app securely.
Add Figaro to your Gemfile and bundle install:
gem "figaro"
Figaro installation is easy:
$ bundle exec figaro install
This creates a commented config/application.yml file and adds it to
your .gitignore. Add your own configuration to this file and you're
done!
Example application.yml File
# config/application.yml
GOOGLE_ID: "ID"
GOOGLE_KEY: "KEY"
THEN config/initializers/fog.rb
GoogleStorage = Fog::Storage.new(
provider: 'Google',
google_project: 'xxxxxxxxxxxxx',
google_client_email: 'xxxxxxxxxxxxx-yyyyyyyyyyyyy#developer.gserviceaccount.com',
google_key_location: Rails.root.join('private','google-cloud-service-key.p12'),
google_storage_secret_access_key_id: ENV["GOOGLE_ID"],
google_storage_secret_access_key: ENV["GOOGLE_KEY"]
)

Sitemap_generator fails to upload

I've followed the instructions on a couple of pages for getting a sitemap to generate and be uploaded to my S3 Bucket. The sitemap is generating, but not uploading.
I'm using carrierwave for the upload, which is working fine for image uploads.
The key file seems to be config/sitemap.rb. Here's mine:
require 'rubygems'
require 'sitemap_generator'
# Set the host name for URL creation
SitemapGenerator::Sitemap.default_host = "https://www.driverhunt.com"
# pick a place safe to write the files
SitemapGenerator::Sitemap.public_path = 'tmp/'
# store on S3 using #Fog# Carrierwave
SitemapGenerator::Sitemap.adapter = SitemapGenerator::WaveAdapter.new
# SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new
# This is a different problem to the one in the question, but using this second adaptor gives the error: "...lib/fog/storage.rb:27:in `new': is not a recognized storage provider (ArgumentError)"
# inform the map cross-linking where to find the other maps
SitemapGenerator::Sitemap.sitemaps_host = "http://#{ENV['S3_BUCKET']}.s3.amazonaws.com/"
# pick a namespace within your bucket to organize your maps
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'
SitemapGenerator::Sitemap.create do
add '/home', :changefreq => 'daily', :priority => 0.9
# add '/contact_us', :changefreq => 'weekly'
end
# SitemapGenerator::Sitemap.ping_search_engines # Not needed if you use the rake tasks
What's going on? How do I debug a carrierwave upload?
I will answer the question as your comment for the S3Adapter brought me to this topic while I was googling the not recognized provider. If you turn back on the comment using the S3Adapter and do the following you will get it working.
If you do not specify any fog ENV VARS for the fog-aws gem you will get the error:
ArgumentError: is not a recognized provider
by using as an adapter the SitemapGenerator::S3Adapter.new
The setup you have got above is perfectly fine, just use the S3Adapter.new instead of the WaveAdapter!
The error you are getting (and I was getting as well) is due to the fact that SitemapGenerator::S3Adapter uses fog-aws and in order to make it run by default you should have the following ENV VARS:
ENV['AWS_ACCESS_KEY_ID'] = XXX
ENV['AWS_SECRET_ACCESS_KEY'] = XXX
ENV['FOG_PROVIDER'] = AWS
ENV['FOG_DIRECTORY'] = your-bucket-name
ENV['FOG_REGION'] = your-bucket-region (ex: us-west-2)
If you are missing even one of the following you will get the error:
ArgumentError: is not a recognized provider
Alternativelly, if you want to avoid using ENV VARS for some reason you should specify the values when you initialize your adapter as follows:
SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new(fog_provider: 'AWS',
aws_access_key_id: 'your-access-key-id',
aws_secret_access_key: 'your-access-key',
fog_directory: 'your-bucket',
fog_region: 'your-aws-region')
However using just the above ENV VARS you will be fine and get your sitemap up and running. This setup was tested with sitemap_generator version: 5.1.0
For your question:
The Image uploading works as it does not require the exact same configuration as the WaveAdapter. I am guessing that your carrierwave.rb file is missing the following:
config.cache_dir = "#{Rails.root}/tmp/"
config.permissions = 0666
The complete configuration for the carrierwave initializer can be found here:
Generate Sitemaps on read only filesystems like Heroku (check if you are missing something or use the other adapter)
However, I believe that your problem has to do with missing ENV VARS from the production environment.

How to grant permissions to AWS user for Hartl's rails tutorial

I don't know how (or where also) to grant read and write permission to the user from AWS so users can post pictures on sample_app in production enviroment. This is final task in 11th chapter, it isn't covered by tutorial and I can't find solution anywhere.
This is carrier_wave.rb file:
if Rails.env.production?
CarrierWave.configure do |config|
config.fog_credentials = {
# Configuration for Amazon S3
:provider => 'AWS',
:aws_access_key_id => ENV['lalala'],
:aws_secret_access_key => ENV['oloalle']
}
config.fog_directory = ENV['name of bucket']
end
end
This is procedure from tutorial:
1) create AWS IAM User and record access and secret key - done
2) create S3 bucket - done
3) grant read and write permission to the user created in the previous step - how???
4) I then run this three commands:
$ heroku config:set S3_ACCESS_KEY=lalala
$ heroku config:set S3_SECRET_KEY=oloalle
$ heroku config:set S3_BUCKET=name of bucket
5) push to git and heroku - done
6) heroku pg:reset DATABASE - done
7)heroku run rake db:migrate and here I get this message:
Running `rake db:migrate` attached to terminal... up, run.7906
rake aborted!
ArgumentError: Missing required arguments: aws_access_key_id, aws_secret_access_key
/app/vendor/bundle/ruby/2.0.0/gems/fog-core-1.28.0/lib/fog/core/service.rb:244:in `validate_options'
/app/vendor/bundle/ruby/2.0.0/gems/fog-core-1.28.0/lib/fog/core/service.rb:268:in `handle_settings'
/app/vendor/bundle/ruby/2.0.0/gems/fog-core-1.28.0/lib/fog/core/service.rb:98:in `new'
/app/vendor/bundle/ruby/2.0.0/gems/fog-core-1.28.0/lib/fog/storage.rb:25:in `new'
/app/vendor/bundle/ruby/2.0.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:83:in `eager_load_fog'
/app/vendor/bundle/ruby/2.0.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:96:in `fog_credentials='
/app/config/initializers/carrier_wave.rb:3:in `block in <top (required)>'
/app/vendor/bundle/ruby/2.0.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:118:in `configure'
/app/vendor/bundle/ruby/2.0.0/gems/carrierwave-0.10.0/lib/carrierwave.rb:14:in `configure'
/app/config/initializers/carrier_wave.rb:2:in `<top (required)>'
/app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:268:in `load'
/app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:268:in `block in load'
/app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:240:in `load_dependency'
/app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:268:in `load'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/engine.rb:652:in `block in load_config_initializer'
/app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.0/lib/active_support/notifications.rb:166:in `instrument'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/engine.rb:651:in `load_config_initializer'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/engine.rb:615:in `each'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/engine.rb:615:in `block in <class:Engine>'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/initializable.rb:30:in `instance_exec'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/initializable.rb:30:in `run'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/initializable.rb:55:in `block in run_initializers'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/initializable.rb:44:in `each'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/initializable.rb:44:in `tsort_each_child'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/initializable.rb:54:in `run_initializers'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/application.rb:352:in `initialize!'
/app/config/environment.rb:5:in `<top (required)>'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/application.rb:328:in `require'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/application.rb:328:in `require_environment!'
/app/vendor/bundle/ruby/2.0.0/gems/railties-4.2.0/lib/rails/application.rb:443:in `block in run_tasks_blocks'
Tasks: TOP => db:migrate => environment
(See full trace by running task with --trace)
Here is my tutorial that I created to pick up where Michael Hartl left off at the end of Ruby on Rails Tutorial (3rd Ed.) Chapter 11. It should answer your question and more.
Getting the railstutorial.org Sample App to work between Heroku and AWS was a huge pain in the ass. But I did it. If you found this tutorial, that means you're probably encountering an error you can't get past. That's fine. I had a few of them.
2020 Note: It's possible that everything here referencing a Region for S3 is no longer needed, or perhaps was never needed. When I originally got this all to work properly it was after adding the region information. However, S3 Buckets all share a global namespace. So if anybody is still reading this, try it all without the region stuff first, and leave a comment about whether it works or not. Anyway, back to the tutorial...
The first thing you need to do is go back over the code that Hartl provided. Make sure you typed it (or copy/pasted it) in exactly as shown. Out of all the code in this section, there is only one small addition you might need to make. The "region" environment variable. This is needed if you create a bucket that is not in the default US area. More on this later. Here is the code for /config/initializers/carrier_wave.rb:
if Rails.env.production?
CarrierWave.configure do |config|
config.fog_credentials = {
# Configuration for Amazon S3
:provider => 'AWS',
:aws_access_key_id => ENV['S3_ACCESS_KEY'],
:aws_secret_access_key => ENV['S3_SECRET_KEY'],
:region => ENV['S3_REGION']
}
config.fog_directory = ENV['S3_BUCKET']
end
end
That line :region => ENV['S3_REGION'] is a problem for a lot of people. More on that later.
You should be using that block of code exactly as shown. Do NOT put your actual keys in there. We'll send them to Heroku separately.
If you had to add that line of code, don't forget to commit it to git and push it to Heroku.
Now let's move on to your AWS account and security.
First of all, create your AWS account. For the most part, it is like signing up for any web site. Make a nice long password and store it someplace secure, like an encrypted password manager. When you make your account, you will be given your first set of AWS keys. You will not be using those in this tutorial, but you might need them at some point in the future so save those somewhere safe as well.
Go to the S3 section and make a bucket. It has to have a unique
name, so I usually just put the date on the end and that does it. For example, you might name it "my-sample-app-bucket-20160126". Once you
have created your bucket, click on the name, then click on Properties.
It's important for you to know what "Region" your bucket is in. Find it,
and make a note of it. You'll use it later.
Your main account probably has full permissions to everything, so let's not use that for transmitting random data between two web services. This could cost you a lot of money if it got out. We'll make a limited user instead. Make a new User in the IAM section. I named it "fog", because that's the cloud service software that handles the sending and receiving. When you create it, you will have the option of displaying and/or downoading the keys associated with the new user. It's important you keep this in a safe
and secure place. It does NOT go into your code, because that will probably
end up in a repository where other people can see it. Also, don't give this
new user a password, since it will not be logging into the AWS dashboard.
Make a new Group. I called mine "s3railsbucket". This is where the
permissions will be assigned. Add "fog" to this group.
Go to the Policies section. Click "Create Policy" then select "Create Your
Own Policy". Give it a name that starts with "Allow" so it will show up near
the top of the list of policies. It's a huge list. Here's what I did:
Policy Name: AllowFullAccessToMySampleAppBucket20160126
Description: Allows remote write/delete access to S3 bucket named
my-sample-app-bucket-20160126.
Policy Document:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-sample-app-bucket-20160126",
"arn:aws:s3:::my-sample-app-bucket-20160126/*"
]
}
]
}
Go back to the Group section, select the group you made, then add
your new policy to the group.
That's it for AWS configuration. I didn't need to make a policy to allow
"fog" to list the contents of the bucket, even though most tutorials I tried
said that was necessary. I think it's only necessary when you want a user
that can log in through the dashboard.
Now for the Heroku configuration. This stuff gets entered in at your
command prompt, just like 'heroku run rake db:migrate' and such. This is
where you enter the actual Access Key and Secret Key you got from the "fog" user you created earlier.
$ heroku config:set S3_ACCESS_KEY=THERANDOMKEYYOUGOT
$ heroku config:set S3_SECRET_KEY=an0tHeRstRing0frAnDomjUnK
$ heroku config:set S3_REGION=us-west-2
$ heroku config:set S3_BUCKET=my-sample-app-bucket-20160126
Look again at that last one. Remember when you looked at the Properties of
your S3 bucket? This is where you enter the code associated with your
region. If your bucket is not in Oregon, you will have to change us-west-2 to your actual region code. This link worked when this tutorial was written:
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
If that doesn't work, Google "AWS S3 region codes".
After doing all this and double-checking for mistakes in the code, I got
Heroku to work with AWS for storage of pictures!
In Services -> IAM, click on 1 User(s) underneath IAM Resources. Select your user you want to have the permission. In this user's profile, click onAttach User Policy. Click on Select for Amazon S3 Full Access and finally Apply Policy.
For others in future, this answer helped me a lot.
Go on Heroku, on your application, go to settings, hit Reveal Config Vars.
Click on on Edit on the right side and enter your secrets there:
S3_BUCKET: name of your bucket goes here
S3_ACCESS_KEY: xxxxx
S3_SECRET_KEY: xxxx
On config/initializers/carrierwave.rb or wherever you're entering your secrets should have:
CarrierWave.configure do |config|
config.root = Rails.root.join('tmp') # adding these...
config.cache_dir = 'carrierwave' # ...two lines
config.fog_credentials = {
:provider => 'AWS', # required
:s3_access_key_id => ENV['S3_ACCESS_KEY'], # required
:s3_secret_access_key => ENV['S3_SECRET_KEY'], # required
:region => 'eu-west-1', # optional, defaults to 'us-east-1'
:host => 's3.example.com', # optional, defaults to nil
:endpoint => 'https://s3.example.com:8080' # optional, defaults to nil
}
config.fog_directory = ENV['S3_Bucket'] # required
config.fog_public = false # optional, defaults to true
config.fog_attributes = {'Cache-Control'=>'max-age=315576000'} # optional, defaults to {}
end
I found the selected correct answer above did not work for me. This is what eventually worked for me after much trial and error.
I followed the first steps to manually enter the secret information as listed above...
Go on Heroku, on your application, go to settings, hit Reveal Config Vars.
Click on on Edit on the right side and enter your secrets there:
S3_BUCKET: name of your bucket goes here
S3_ACCESS_KEY: xxxxx
S3_SECRET_KEY: xxxx
However a slight difference in the carrier_wave file seemed to work.
Note the encpompassing if Rails.env.production? line and end.
carrier_wave.rb
if Rails.env.production?
CarrierWave.configure do |config|
config.root = Rails.root.join('tmp') # adding these...
config.cache_dir = 'carrierwave' # ...two lines
config.fog_credentials = {
:provider => 'AWS', # required
:aws_access_key_id => ENV['S3_ACCESS_KEY'],
:aws_secret_access_key => ENV['S3_SECRET_KEY'],
:region => 'eu-west-2', # optional, defaults to 'us-east-1'
:host => 's3.example.com', # optional, defaults to nil
:endpoint => 'https://s3.example.com:8080' # optional, defaults to nil
}
config.fog_directory = ENV['S3_Bucket'] # required
config.fog_public = false # optional, defaults to true
config.fog_attributes = {'Cache-Control'=>'max-age=315576000'} # optional, defaults to {}
end
end
Not sure if that is what the problem was or not.
After making that change I finished the chapter according to Michael Hartl's instructions.
We’re now ready to commit the changes on our topic branch and merge
back to master:
$ bundle exec rake test
$ git add -A
$ git commit -m "Add user microposts"
$ git checkout master
$ git merge user-microposts
$ git push
Then we deploy, reset the database, and reseed the sample data:
$ git push heroku
$ heroku pg:reset DATABASE
$ heroku run rake db:migrate
$ heroku run rake db:seed

Trouble Getting s3 set up in Rails 3 Refinery CMS App

I'm trying to get my refinery cms image storage to Amazon s3 and I'm following this guide:
http://refinerycms.com/guides/how-to-use-amazon-s3-for-storage
But I'm blocked here:
There are a number of ways to set
these with your credentials, including
unix variables or settings them
manually through Ruby using ENV.
How do I define these credentials. Do I put something like :S3_KEY =>
"my_key" in my environments.rb file? I tried this and it didn't work.
I also tried this:
AWS::S3::Base.establish_connection!(
:access_key_id => ENV['S3_KEY'] || 'key_goes_here',
:secret_access_key => ENV['S3_SECRET'] || 's3_secret_key_here',
)
Can't figure out how to do this. Any ideas are greatly appreciated.
The safest way is to specify them as environment variables, so they aren't included in your source code. If you're the only one with access to the source, then specifying them as you describe should work.
You can specify them in your ~/.bashrc
export S3_KEY=mykey
export S3_SECRET=mysecret
Or if you're just testing locally you can prepend them to your rails command.
$ S3_KEY=mykey S3_SECRET=mysecret rails server
If you don't want to/can't use environment variables, another method is to use an initializer to load credentials from a yml file: config/initializers/s3_credentials.rb
# Load AWS::S3 configuration values
#
S3_CREDENTIALS = \
YAML.load_file(File.join(Rails.root, 'config/s3_credentials.yml'))[Rails.env]
# Set the AWS::S3 configuration
#
AWS::S3::Base.establish_connection! S3_CREDENTIALS['connection']
config/s3_credentials.yml
development: &defaults
connection:
:access_key_id: AAAAAA_your-key-here
:secret_access_key: 4rpsi235js_your-secret-here
:use_ssl: true
bucket: project-development
acl: public-read
production:
<<: *defaults
bucket: project

Resources