How to setup AWS credentials in rails for development? - ruby-on-rails

I'm trying to figure out what is the best way to set up my credentials in development? I'm able to set up credentials in production because of heroku's configuration variables... but need help doing so if you're just testing app locally.
I have this in my development.rb
config.paperclip_defaults = {
:storage => :s3,
:s3_credentials => {
:bucket => ENV['BUCKET_NAME'],
:access_key_id => ENV['ACCESS_KEY_ID'],
:secret_access_key => ENV['SECRET_ACCESS_KEY']
}
}
But, I'm not sure how to reference these ENV variables?
I created this aws.yml file in my /config folder
development:
BUCKET_NAME: "somename"
ACCESS_KEY_ID: "4205823951412980"
SECRET_ACCESS_KEY: "123141ABNCEFEHUDSL2309489850"
I thought if I matched the ENV name, it'll work? I guess thats not the case...
I made sure to include aws.yml to my .ignore file
/config/aws.yml

You are looking for dotenv gem. Install the gem gem 'dotenv-rails', :groups => [:development, :test], then
just create and `.env' file and put the variables into it like this.
S3_BUCKET=YOURS3BUCKET
SECRET_KEY=YOURSECRETKEYGOESHERE

I like to use Foreman locally and store all my variables in a .ENV file. Plays well with Heroku and is easy to manage. Also in my case I have the following:
Under config/initiliazers/aws.rb
require 'aws-sdk'
require 'aws-sdk-resources'
# Rails.configuration.aws is used by AWS, Paperclip, and S3DirectUpload
Rails.configuration.aws = YAML.load(ERB.new(File.read("#{Rails.root}/config/aws.yml")).result)[Rails.env].symbolize_keys!
AWS.config(logger: Rails.logger)
AWS.config(Rails.configuration.aws)
Under config/initializers/paperclip.rb
# https://devcenter.heroku.com/articles/paperclip-s3
if Rails.env.test?
Paperclip::Attachment.default_options.merge!(
path: ":rails_root/public/paperclip/:rails_env/:class/:attachment/:id_partition/:filename",
storage: :filesystem
)
else
Paperclip::Attachment.default_options.merge!(
url: ':s3_domain_url',
path: '/:class/:filename',
storage: :s3,
s3_credentials: Rails.configuration.aws,
s3_protocol: 'https'
)
end
Under config/aws.yml
defaults: &defaults
access_key_id: <%= ENV['AWS_ACCESS_KEY_ID'] %>
secret_access_key: <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
test:
<<: *defaults
bucket: <%= ENV['AWS_TEST_BUCKET'] %>
development:
<<: *defaults
bucket: <%= ENV['AWS_TEST_BUCKET'] %>
production:
<<: *defaults
bucket: <%= ENV['AWS_BUCKET'] %>

What I like to is create a config.yml on my ./config directory.
then I tell the ./config/application.rb I have a small block that loads the variables like
config.before_initialize do
dev = File.join(Rails.root, 'config', 'config.yml')
YAML.load(File.open(dev)).each do |key,value|
ENV[key.to_s] = value
end if File.exists?(dev)
end
Here is my ./config/config.yml
BUCKET_NAME: "somename"
ACCESS_KEY_ID: "4205823951412980"
SECRET_ACCESS_KEY: "123141ABNCEFEHUDSL2309489850"
New Updates
Here is a copy of my #config/application.rb
#config/application.rb
require File.expand_path('../boot', __FILE__)
require 'rails/all'
# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)
module Appname
class Application < Rails::Application
# Settings in config/environments/* take precedence over those specified here.
# Application configuration should go into files in config/initializers
# -- all .rb files in that directory are automatically loaded.
# Set Time.zone default to the specified zone and make Active Record auto-convert to this zone.
# Run "rake -D time" for a list of tasks for finding time zone names. Default is UTC.
# config.time_zone = 'Central Time (US & Canada)'
# The default locale is :en and all translations from config/locales/*.rb,yml are auto loaded.
# config.i18n.load_path += Dir[Rails.root.join('my', 'locales', '*.{rb,yml}').to_s]
# config.i18n.default_locale = :de
# Do not swallow errors in after_commit/after_rollback callbacks.
config.active_record.raise_in_transactional_callbacks = true
initializer 'setup_asset_pipeline', :group => :all do |app|
# We don't want the default of everything that isn't js or css, because it pulls too many things in
app.config.assets.precompile.shift
# Explicitly register the extensions we are interested in compiling
app.config.assets.precompile.push(Proc.new do |path|
File.extname(path).in? [
'.html', '.erb', '.haml', # Templates
'.png', '.gif', '.jpg', '.jpeg', # Images
'.eot', '.otf', '.svc', '.woff', '.ttf', # Fonts
]
end)
end
I18n.enforce_available_locales = false
config.before_initialize do
dev = File.join(Rails.root, 'config', 'config.yml')
YAML.load(File.open(dev)).each do |key,value|
ENV[key.to_s] = value
end if File.exists?(dev)
end
end
end

If everything else fails, you can simply start your rails app with command line params. It is good to know the base case, where you know for sure that your params are indeed passed to your app:
ACCESS_KEY_ID=ABCDEFG SECRET_ACCESS_KEY=secreTaccessKey BUCKET_NAME=dev-images rails s
I actually prefer to set an alias in env:
alias railss='ACCESS_KEY_ID=ABCDEFG SECRET_ACCESS_KEY=secreTaccessKey BUCKET_NAME=dev-images rails s'

Related

Not able to change S3 path of images in refinerycms (using dragonfly) in rails

Currently I am migrating a 10 year old on-prem application to heroku.
We have good amount of data in our servers.
dragonfly/refinerycms data stored at: public/system/refinery/...
Images stored under refinery folder images/2021/11/25/486ucenknk_image.png
same is the case for resources.
But when I set in images.rb
config.s3_datastore = true
obviously files start saving to S3 bucket. But with a different path.
Which is
2021/11/25/02/01/37/5f6e0f21-658c-4cf2-9edc-da7cb8575ab8/images.png
means it is including time as well in folders.
I tried changing this path at so many places but I couldn't. I tried changing url_format as well, but looks like it is not affecting anything in the store location.
I have attached the config files for both the files.
config/initializers/dragonfly.rb
# config/initializers/dragonfly.rb
require 'dragonfly/s3_data_store'
# Configure
Dragonfly.app.configure do
protect_from_dos_attacks true
secret "Some secret"
url_format "/media/:job/:name"
datastore :s3,
bucket_name: ENV['S3_BUCKET'],
access_key_id: ENV['S3_KEY'],
secret_access_key: ENV['S3_SECRET'],
url_scheme: 'https'
end
# Logger
Dragonfly.logger = Rails.logger
# Mount as middleware
Rails.application.middleware.use Dragonfly::Middleware
# Add model functionality
if defined?(ActiveRecord::Base)
ActiveRecord::Base.extend Dragonfly::Model
ActiveRecord::Base.extend Dragonfly::Model::Validations
end
Excon.defaults[:write_timeout] = 500
config/initializers/refinery/images.rb
# config/initializers/refinery/images.rb
# encoding: utf-8
Refinery::Images.configure do |config|
# Configure S3 (you can also use ENV for this)
# The s3_backend setting by default defers to the core setting for this but can be set just for images.
config.s3_datastore = false
config.s3_bucket_name = ENV['S3_BUCKET']
config.s3_access_key_id = ENV['S3_KEY']
config.s3_secret_access_key = ENV['S3_SECRET']
config.s3_region = 'us-east-1'
# Configure Dragonfly
config.dragonfly_verify_urls = false
config.datastore_root_path = "/refinery/images"
end
If anyone encountered problem like this before, please help me. Thanks in Advance.

Heroku: Couldn't find Active Storage configuration in /app/config/storage.yml (RuntimeError)

My app deploys to Heroku but crashes every time. I don't know why. I have set up Carrierwave, fog, and aws for an app in production on Heroku before just fine. Tried to follow the same steps and I am getting an h10 error code. In the rails console it specifically says:
/app/vendor/bundle/ruby/2.3.0/gems/activestorage-5.2.1/lib/active_storage/engine.rb:76:in
`block (2 levels) in ': Couldn't find Active Storage
configuration in /app/config/storage.yml (RuntimeError)
storage.yml
test:
service: Disk
root: <%= Rails.root.join("tmp/storage") %>
local:
service: Disk
root: <%= Rails.root.join("storage") %>
# Use rails credentials:edit to set the AWS secrets (as aws:access_key_id|secret_access_key)
# amazon:
amazon:
service: S3
access_key_id: "S3_KEY"
secret_access_key: "S3_SECRET"
region: "us-east-1"
bucket: "books4reviews"
production.rb
config.active_storage.service = :amazon
carrierwave.rb
CarrierWave.configure do |config|
config.fog_provider = 'fog/aws'
config.fog_credentials = {
provider: 'AWS',
aws_access_key_id: ENV['S3_KEY'],
aws_secret_access_key: ENV['S3_SECRET'],
region: 'us-east-1'
}
config.fog_directory = 'books4reviews'
config.fog_public = false
config.storage = :fog
end
puma.rb
threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
threads threads_count, threads_count
port ENV.fetch("PORT") { 3000 }
environment ENV.fetch("RAILS_ENV") { "development" }
plugin :tmp_restart
Procfile
web: bundle exec puma -C config/puma.rb
avatar_uploader.rb
class AvatarUploader < CarrierWave::Uploader::Base
# Include RMagick or MiniMagick support:
# include CarrierWave::RMagick
# Choose what kind of storage to use for this uploader:
include CarrierWave::MiniMagick
storage :fog
process resize_to_fit: [500,500]
version :small do
process resize_to_fill: [200, 200]
end
version :medium do
# change the word 'fit' to 'fill'
process resize_to_fill: [400,600]
end
version :large do
process resize_to_fill: [1000,1000]
end
version :thumb do
process resize_to_fill: [50, 50]
end
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
def extension_white_list
%w(jpg jpeg gif png)
end
end
I've set my env variables for the aws credentials in my heroku config variables from the terminal. Can you tell me why I'm getting this active storage error? Thanks
I had this same issue when deploying a recently upgraded Rails app. The application was upgraded from Rails 5 to Rails 6. However, when I try deploying to Heroku, I got the error below:
2021-02-12T17:32:33.404828+00:00 app[web.1]: ! Unable to load application: RuntimeError: Couldn't find Active Storage configuration in /app/config/storage.yml
2021-02-12T17:32:33.404874+00:00 app[web.1]: bundler: failed to load command: puma (/app/vendor/bundle/ruby/2.7.0/bin/puma)
2021-02-12T17:32:33.404958+00:00 app[web.1]: RuntimeError: Couldn't find Active Storage configuration in /app/config/storage.yml
Here's how I fixed it:
I checked the config directory of my application and realized that it had no config/storage.yml file. All I had to do was to create the file, and copy the vanilla template that comes with Rails 6 applications into the file:
test:
service: Disk
root: <%= Rails.root.join("tmp/storage") %>
local:
service: Disk
root: <%= Rails.root.join("storage") %>
# Use rails credentials:edit to set the AWS secrets (as aws:access_key_id|secret_access_key)
# amazon:
# service: S3
# access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
# secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
# region: us-east-1
# bucket: your_own_bucket
# Remember not to checkin your GCS keyfile to a repository
# google:
# service: GCS
# project: your_project
# credentials: <%= Rails.root.join("path/to/gcs.keyfile") %>
# bucket: your_own_bucket
# Use rails credentials:edit to set the Azure Storage secret (as azure_storage:storage_access_key)
# microsoft:
# service: AzureStorage
# storage_account_name: your_account_name
# storage_access_key: <%= Rails.application.credentials.dig(:azure_storage, :storage_access_key) %>
# container: your_container_name
# mirror:
# service: Mirror
# primary: local
# mirrors: [ amazon, google, microsoft ]
This time when I deployed everything worked fine.
Note: You can modify the file content based on your storage configurations
That's all.
I hope this helps
This may not fix your issue, but I had ".yaml" instead of ".yml" because I had to manually create the file "/config/storage.yml" manually and made a typo.
Hope this helps someone, as I couldn't find many results on this error.
FYI, I think the generator didn't create the storage.yml file because I was on Rails 5.1, originally and then upgraded to 5.2

Settings returns nil for all configurations which reads from environment variables in devise.rb

I am using config gem for configuration management in Rails 4.2.5 application. I am facing a strange issue where it looks like config loads after devise.rb and so not able to find any config from settings.yml in production reading some environment variables.
Here is application.rb config
require File.expand_path('../boot', __FILE__)
require 'rails/all'
Bundler.require(*Rails.groups)
module Magnificent
class Application < Rails::Application
Config::Integration::Rails::Railtie.preload
end
end
devise.rb
Devise.setup do |config|
config.omniauth :twitter, Settings.twitter.consumer_key, Settings.twitter.consumer_secret, strategy_class: OmniAuth::Strategies::Twitter
config.omniauth :facebook, Settings.facebook.app_id, Settings.facebook.app_secret , { info_fields: 'email,first_name,last_name' , scope: 'email,public_profile' }
end
settings.yml
twitter:
consumer_key: <%= ENV['TWITTER_CONSUMER_KEY'] %>
consumer_secret: <%= ENV['TWITTER_CONSUMER_SECRET'] %>
facebook:
app_id: <%= ENV['FACEBOOK_APP_ID'] %>
app_secret: <%= ENV['FACEBOOK_APP_SECRET'] %>
P.S. This works perfectly in development environment. Also, When I access Settings.twitter.consumer_key from the console in production, it shows the value.
The application runs on Passenger 5.0.25.
EDIT
Few models also use settings from config that read values from environment variables. Such settings also have nil values. Surprisingly all those settings reading values from environment have properly value set when I check from console(RAILS_ENV=production rails c)
Created a yml file with key as environment variable name. Then added environment variables before module declaration in application.rb
require 'rails/all'
# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)
if Rails.env.production?
config_file = '/srv/shared_filesystem/pdf-config.yml'
if File.exists?(config_file)
config = YAML.load(File.read(config_file))
config.each do |key, value|
ENV[key] ||= value.to_s unless value.kind_of? Hash
end
else
raise 'Missing required configuration file /srv/shared_filesystem_ro/pdf-config.yml'
end
end
module MyApp
class Application < Rails::Application
.. ..
.. ..
end
end

Paperclip: Staging App that can read from productions bucket, but write to different bucket?

I run a staging and development environment that is a mirror of the production database.
I use the Paperclip gem for storing uploads to S3. I want to let environments that are not production READ from the production bucket (because thats where the file is stored), but WRITE to a different bucket, for safety. In addition, the app should be smart enough to now READ from the different bucket IF an upload was made.
Is there a way to achieve this?
the way that I achive this is I have a config file local that is different from my config on the server I use Heroku to host my apps.
1) I create a file called config/config.yml
#S3
S3_KEY: 'xxxxxx'
S3_SECRET: 'xxxxxxxx'
S3_REGION: 'us-east-1'
S3_ASSET_URL: 'appname-dev.s3-website-us-east-1.amazonaws.com'
S3_BUCKET_NAME: 'appname-dev'
2) I read variables form config/config.ym into config/application.rb
require File.expand_path('../boot', FILE)
require 'rails/all'
# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(:default, Rails.env)
module Appname
class Application < Rails::Application
config.before_initialize do
dev = File.join(Rails.root, 'config', 'config.yml')
YAML.load(File.open(dev)).each do |key,value|
ENV[key.to_s] = value
end if File.exists?(dev)
end
end
end
then on Heroku I set the config variables
I hope this is able to help.
After further digging I discovered that the bucket option for has_attached_filecan take a proc.
config/inititalizers/paperclip_defaults.rb
LIVE_BUCKET = 'media'
STAGING_BUCKET = 'media-demo'
Paperclip::Attachment.default_options.merge!(
storage: :s3,
path: 'assets/:class/:id/:attachment/:style.:extension',
s3_credentials: "#{Rails.root}/config/s3.yml",
s3_host_name: 's3-us-west-2.amazonaws.com',
use_timestamp: false,
bucket: proc { |attachment|
(Rails.env.production? || !attachment.dirty?) ? LIVE_BUCKET : STAGING_BUCKET
}
)
Relevant documentation:
http://www.rubydoc.info/github/thoughtbot/paperclip/master/Paperclip/Storage/S3
http://www.rubydoc.info/github/thoughtbot/paperclip/master/Paperclip/Attachment#dirty%3F-instance_method

Rails send mail with GMail

I am on rails 2.3.5 and have the latest Ruby installed and my application is running well, except, GMail emails.
I am trying to setup my gmail imap connection which has worked previously but now doesnt want to know.
This is my code:
# Be sure to restart your server when you modify this file
# Uncomment below to force Rails into production mode when
# you don't control web/app server and can't set it the proper way
# ENV['RAILS_ENV'] ||= 'production'
# Specifies gem version of Rails to use when vendor/rails is not present
RAILS_GEM_VERSION = '2.3.5' unless defined? RAILS_GEM_VERSION
# Bootstrap the Rails environment, frameworks, and default configuration
require File.join(File.dirname(__FILE__), 'boot')
Rails::Initializer.run do |config|
# Gems
config.gem "capistrano-ext", :lib => "capistrano"
config.gem "configatron"
# Make Time.zone default to the specified zone, and make Active Record store time values
# in the database in UTC, and return them converted to the specified local zone.
config.time_zone = "London"
# The internationalization framework can be changed to have another default locale (standard is :en) or more load paths.
# All files from config/locales/*.rb,yml are added automatically.
# config.i18n.load_path << Dir[File.join(RAILS_ROOT, 'my', 'locales', '*.{rb,yml}')]
#config.i18n.default_locale = :de
# Your secret key for verifying cookie session data integrity.
# If you change this key, all old sessions will become invalid!
# Make sure the secret is at least 30 characters and all random,
# no regular words or you'll be exposed to dictionary attacks.
config.action_controller.session = {
:session_key => '_base_session',
:secret => '7389ea9180b15f1495a5e73a69a893311f859ccff1ffd0fa2d7ea25fdf1fa324f280e6ba06e3e5ba612e71298d8fbe7f15fd7da2929c45a9c87fe226d2f77347'
}
config.active_record.observers = :user_observer
end
ActiveSupport::CoreExtensions::Date::Conversions::DATE_FORMATS.merge!(:default => '%d/%m/%Y')
ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.merge!(:default => '%d/%m/%Y')
require "will_paginate"
ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.smtp_settings = {
:enable_starttls_auto => true,
:address => "smtp.gmail.com",
:port => 587,
:domain => "XXXXXXXX.XXX",
:authentication => :plain,
:user_name => "XXXXXXXXXX.XXXXXXXXXX.XXX",
:password => "XXXXX"
}
But the above just results in an SMTP auth error in the production log.
I have read varied reports of this not working in Rails 2.2.2 but nothing for 2.3.5, anyone got any ideas?
Thanks,
Danny
Use action mailer optional tls

Resources