I'm currently trying to configure Paperclip with newest aws-sdk suggested gem.
On my S3.yml file I have something like this
development:
bucket: newmeeter-dev
access_key_id: ENV['S3_KEY']
secret_access_key: ENV['S3_SECRET']
But it is not recognizing the ENV variables. I'm getting the following error
AWS::S3::Errors::InvalidAccessKeyId in PhotosController#create
The AWS Access Key Id you provided does not exist in our records.
If I try to put both the access and secret directly into the file it works perfectly. At the same time I tried to print both ENV variables into the views or in the console I can see their values okay.
I'm not getting why it is not recognizing it.
Solved!
I found the reply to this question here
Ruby on Rails: Can you put Ruby code in a YAML config file?
Solution: YAML files understand code in ERB format.
Printing ENV variables inside <%= and %> works.
access_key_id: <%= ENV['S3_KEY'] %>
secret_access_key: <%= ENV['S3_SECRET'] %>
Related
I have a Rails 5 app and I am trying to import my Environment Variables in a yaml file, which I can then import in the environment.rb. My config/aws.yml looks as follows:
S3_BUCKET: <%= ENV["S3_BUCKET"] %>
AWS_ACCESS_KEY_ID: <%= ENV["AWS_ACCESS_KEY_ID"] %>
AWS_SECRET_ACCESS_KEY: <%= ENV["AWS_SECRET_ACCESS_KEY"] %>
Then in my environment.rb I have the following:
require_relative 'application'
aws_file = File.join(Rails.root, 'config', 'aws.yml')
if File.exist?(aws_file)
YAML.safe_load(File.open(aws_file)).each do |k, v|
ENV[k.to_s] = v
end
end
on running my rails c this is what I get:
[1] pry(main)> ENV.fetch('S3_BUCKET')
=> "<%= ENV[\"S3_BUCKET\"] %>"
How can I pass the actual environment variable rather than it pulling the name of the environment variable as a string?
EDIT: Adding more relevant information
My credentials are stored in my elastic-beanstalk configuration and I am trying to load the values from the configuration to my yaml file.
<%= ... %> is a feature of ERB, not YAML. You'd need to first run your YAML file through ERB to cause the <%= ... %> to execute and then load it as YAML.
But there's a deeper problem. Your YAML file is trying to get its values from environment variables. But then you set those same environment variables from the YAML values. It's circular.
Instead use something like Encrypted Credentials introduced in Rails 5.2 to manage your secrets. Then write config/initializer/aws.rb to set the necessary environment variables from those secrets. This is nice because it stores your secrets in your app rather than in the deployment platform. Then it can be run with full secrets anywhere.
Or you can set the environment variables in Elastic Beanstalk via the "Environment Properties" console.
In addition to what you have already, add this to config/application.rb:
config.x.aws = config_for(:aws)
And you'll be able to access it:
Rails.configuration.x.aws.S3_BUCKET # => ...
Have you tried fetch, it worked on my case:
S3_BUCKET: <%= ENV.fetch("S3_BUCKET") %>
AWS_ACCESS_KEY_ID: <%= ENV.fetch("AWS_ACCESS_KEY_ID") %>
AWS_SECRET_ACCESS_KEY: <%= ENV.fetch("AWS_SECRET_ACCESS_KEY") %>
I am having an issue trying to deploy my app to heroku with rails active storage. In development I have no issues using
config.active_storage.service = :local
and all works as it should.
However in my production.rb file I have set config.active_storage.service = :amazon
and followed the set up guides.
My storage.ymlis as follows:
test:
service: Disk
root: <%= Rails.root.join("tmp/storage") %>
local:
service: Disk
root: <%= Rails.root.join("storage") %>
amazon:
service: S3
access_key_id: <%= ENV['S3_ACCESS_KEY'] %>
secret_access_key: <%= ENV['S3_SECRET_ACCESS_KEY'] %>
region: <%= ENV['S3_REGION'] %>
bucket: <%= ENV['S3_BUCKET_NAME'] %>
When I run git push heroku master, the app will deploy, but the following error will appear in the logs: "Detecting rails configuration failed".
I am unable to open the app and the herkou log will display the following error: "Missing configuration for the :amazon Active Storage service. Configurations available for [:test]".
This same error occurs if I change
config.active_storage.service = :amazon
to
config.active_storage.service = :local
however if I change it to
config.active_storage.service = :test
the app will deploy without error and I am able to open the app and upload files as expected.
I have trawled the web but haven't seen anyone else with this error, so any comments or thoughts are appreciated.
Thanks in advance.
The problem is the indentation in storage.yml
test: local: are indented one space, and amazon: is indented two spaces.
I had the same problem, and it was hard to notice an extra space. In YAML the spaces matter, but Ruby devs are not accustomed to having the spaces matter.
I'm setting up a new rails 5.2 app utilising Active Storage and using AWS for the hosting of images in production.
However, I'm having an issue with the app reading the credentials:
2018-07-06T08:11:52.625415+00:00 app[web.1]: ! Unable to load application: Aws::Sigv4::Errors::MissingCredentialsError: Cannot load `Rails.config.active_storage.service`:
2018-07-06T08:11:52.625432+00:00 app[web.1]: missing credentials, provide credentials with one of the following options:
2018-07-06T08:11:52.625435+00:00 app[web.1]: - :access_key_id and :secret_access_key
2018-07-06T08:11:52.625437+00:00 app[web.1]: - :credentials
2018-07-06T08:11:52.625479+00:00 app[web.1]: - :credentials_provider
This is an existing S3 Bucket which I created a new user just for this app. I'm happy with the CORS etc.
The user is set up under the S3FullAccess group.
I've edited the credentials in my app via $EDITOR="atom --wait" rails credentials:edit
The contents of the file:
aws:
access_key_id: [my access key]
secret_access_key: [my secrect key]
# Used as the base secret for all MessageVerifiers in Rails, including the one protecting cookies.
secret_key_base: [my secret key base]
Appreciate this is in YAML format, I have played with using one space, and one tab on the keys, but this doesn't seem to make a difference.
When I save and close the file, the terminal writes New credentials encrypted and saved.
I also have gem 'aws-sdk-s3', '~>1', require: false installed.
And config/storage.yml
test:
service: Disk
root: <%= Rails.root.join("tmp/storage") %>
local:
service: Disk
root: <%= Rails.root.join("storage") %>
# Use rails credentials:edit to set the AWS secrets (as aws:access_key_id|secret_access_key)
amazon:
service: S3
access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
region: eu-west-2
bucket: [mybucket]
Any suggestions on what I might be doing wrong?
I think you're missing the master.key file in your server. Check your local repo in config/master.key (this file is added to your .gitignore by default).
Add this file to your server or set ENV["RAILS_MASTER_KEY"].
This worked for me on Heroku: in "Settings > Config vars" add a RAILS_MASTER_KEY key, with the content of your your config/master.key file (from your Rails app) as the value.
Go into config/environments/development.rb and make sure you have this:
config.active_storage.service = :local
in config/environments/production you should have
config.active_storage.service = :amazon
amazon is for Amazon S3. It can be changed to whichever storage service you want to use. See the Rails docs for more info on storage services and Active Storage.
In Rails 5.2, do the following:
Step 1. In config/storage.yml add
amazon:
service: S3
access_key_id: <%= ENV['AWS_ACCESS_KEY_ID'] %>
secret_access_key: <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
region: ap-south-1
bucket: my-bucket
Step 2:
Copy config/credentials.yml.example to config/credentials.yml
and add the following in config/credentials.yml
development:
AWS_ACCESS_KEY_ID: YOUR-KEY
AWS_SECRET_ACCESS_KEY: YOUR-SECRET
credentials.yml is already added to .gitignore by default.
Step 3:
In application.rb
Uncomment the following:
# Load ENV variables from credentials.yml file
config.before_configuration do
env_file = File.join(Rails.root, 'config', 'credentials.yml')
YAML.load(File.open(env_file))[Rails.env].each do |key, value|
ENV[key.to_s] = value
end if File.exists?(env_file)
end
Restart the server and try to upload again.
Another way of solving this issue (worked for me)
Run rake secret in the console
copy the key
go to config and open application.rb
inside the class type: config.secret_key_base = "paste the output of rake secrete"
I had the same error. In my case the problem was neither with configs, nor with master.key. Starting Redis server fixed the error. For MacOS:
$> redis-server
I created a new Rails app called sample_app and I use postgresql as my db (I already created a postgresql username and password). I use this setup guide https://gorails.com/setup/ubuntu/16.04
So I run this command rails new sample_app -d postgresql. And then I have to edit the config/database.yml to match the username and password to my postgresql's username and password I just created. But I don't want to hard-code because I will be using git.
I found this tutorial from digital ocean which suggest to use:
username: <%= ENV['APPNAME_DATABASE_USER'] %>
password: <%= ENV['APPNAME_DATABASE_PASSWORD'] %>
Is this the correct code? If so, since my app is called sample_app, my code should be?
username: <%= ENV['SAMPLE_APP_DATABASE_USER'] %>
password: <%= ENV['SAMPLE_APP_DATABASE_PASSWORD'] %>
If this is not the correct one, can you help me? Thank you!
There are many ways you can set the environment variables.
Here are two of them,
Option One: Setting ENV variables via a yml file
Create a file config/local_env.yml:
config/local_env.yml:
SAMPLE_APP_DATABASE_USER: 'your username'
SAMPLE_APP_DATABASE_PASSWORD: '******'
The above are the names you will use like,ENV['SAMPLE_APP_DATABASE_USER']. these can be names as your wish. you can take any name, but we should use the same name in the ENV reference.
add it to gitignore:
/config/local_env.yml
Change some code in application.rb
application.rb:
config.before_configuration do
env_file = File.join(Rails.root, 'config', 'local_env.yml')
YAML.load(File.open(env_file)).each do |key, value|
ENV[key.to_s] = value
end if File.exists?(env_file)
end
The code opens the config/local_env.yml file, reads each key/value pair, and sets environment variables.
Using Environment Variables:
username: <%= ENV['SAMPLE_APP_DATABASE_USER'] %>
password: <%= ENV['SAMPLE_APP_DATABASE_PASSWORD'] %>
Option Two: Use the Figaro Gem
The gem takes advantage of Ruby’s ability to set environment variables as well as read them. The gem reads a config/application.yml file and sets environment variables before anything else is configured in the Rails application.
Here’s how to use it. In your Gemfile, add:
gem 'figaro'
and run bundle install
The gem provides a generator:
$ bundle exec figaro install
The generator creates a config/application.yml file and modifies the .gitignore file to prevent the file from being checked into a git repository.
You can add environment variables as key/value pairs to config/application.yml:
SAMPLE_APP_DATABASE_USER: 'your username'
SAMPLE_APP_DATABASE_PASSWORD: '******'
The environment variables will be available anywhere in your application as ENV variables:
ENV["SAMPLE_APP_DATABASE_USER"]
Here are the remaining ways you can achieve the same.
You can call it anything you want...
username: <%= ENV['CARROTS'] %>
password: <%= ENV['BEANS'] %>
You just have to make sure your deploy script sets the variables CARROTS and BEANS correctly.
try this gem dotenv-rails
add this to Gemfile:
gem 'dotenv-rails', :groups => [:development, :test]
bundle it. Now create a .env file on your apps's directory with following content:
SAMPLE_APP_DATABASE_USER: "devuser"
SAMPLE_APP_DATABASE_PASSWORD: "devuser"
restart the server you're good to go. these variables are exported when you boot your app which you can access in your database.yml file
username: <%= ENV['SAMPLE_APP_DATABASE_USER'] %>
password: <%= ENV['SAMPLE_APP_DATABASE_PASSWORD'] %>
read dotenv-rails documentation for more info
I have a rails app running on Heroku. I am using paperclip for some simple image uploads for user avatars and some other things, I have S3 set as my backend and everything seems to be working fine except when trying to push to S3 I get the following error:
The AWS Access Key Id you provided does not exist in our records.
Thinking I mis-pasted my access key and secret key, I tried again, still no luck. Thinking maybe it was just a buggy key I deactivated it and generated a new one. Still no luck.
Now for both keys I have used the S3 browser app on OS X and have been able to connect to each and view my current buckets and add/delete buckets. Is there something I should be looking out for? I have my application's S3 and paperclip setup like so
development:
bucket: (unique name)
access_key_id: ENV['S3_KEY']
secret_access_key: ENV['S3_SECRET']
test:
bucket: (unique name)
access_key_id: ENV['S3_KEY']
secret_access_key: ENV['S3_SECRET']
production:
bucket: (unique_name)
access_key_id: ENV['S3_KEY']
secret_access_key: ENV['S3_SECRET']
has_attached_file :cover,
:styles => {
:thumb => "50x50"
},
:storage => :s3,
:s3_credentials => "#{RAILS_ROOT}/config/s3.yml",
:path => ":class/:id/:style/:filename"
EDIT NOTE: The ENV['S3_KEY'] and ENV['S3_SECRET'] are environment variables in heroku which i have tried even using my keys directly and it still doesn't work
Note: I just added the (unique name) bits, those aren't actually there--I have also verified bucket names, but I don't even think this is getting that far. I also have my heroku environment vars setup correctly and have them setup on dev
You aren't setting a bucket. It's in your s3.yml file, but you aren't reading that value from your call to has_attached_file.
Paperclip S3 docs:
http://rubydoc.info/gems/paperclip/Paperclip/Storage/S3#s3_protocol-instance_method
Also, pay attention to those people who are telling you not to use a s3.yml file with Heroku. It's a waste and just added abstraction that buys you nothing. You already have your ENV set up with the values you need, so use them.
I've done this before where I don't want to push an s3.yml file to Heroku, but I do want to use one for test and development. In an initializer you can do something like this:
# If an s3.yml file exists, use the key, secret key, and bucket values from there.
# Otherwise, pull them from the environment.
if File.exists?("#{Rails.root}/config/s3.yml")
s3_config = YAML.load_file("#{Rails.root}/config/s3.yml")
S3[:key] = s3_config[Rails.env]['key']
S3[:secret] = s3_config[Rails.env]['secret']
S3[:bucket] = s3_config[Rails.env]['bucket']
else
S3[:key] = ENV['S3_KEY']
S3[:secret] = ENV['S3_SECRET']
S3[:bucket] = ENV['S3_BUCKET']
end
Then when you're setting up Paperclip in your model, you reference the value like this:
...
:s3_credentials => {
:access_key_id => S3[:key],
:secret_access_key => S3[:secret]
},
:bucket => S3[:bucket]
Obviously, this means that you do not want to have your s3.yml file in your git repository (which really, you shouldn't anyway).
I kept getting the same AWS::S3::InvalidAccessKeyId error, and had a very similar s3.yml file. As x1a4 recommended, I used ERB in my yaml file and it worked. Here's what it looks like now:
# myapp/config/s3.yml
development: &DEFAULTS
bucket: myapp_dev
access_key_id: <%= ENV['S3_KEY'] %>
secret_access_key: <%= ENV['S3_SECRET'] %>
test:
<<: *DEFAULTS
bucket: myapp_test
production:
<<: *DEFAULTS
bucket: myapp
staging:
<<: *DEFAULTS
bucket: myapp_staging
I guess this might a tad too indirect for some folks, but it seemed like the cleanest implementation to me.
Your s3 yaml file is actually using the strings ENV['S3_KEY'] and ENV['S3_SECRET'] as the auth info for s3. They are not being evaluated as ruby code.
There are a couple of things at least that you can do outside of putting that actual info into the yaml file. You can look into enabling ERB in your yaml configs or just not use a yaml file for your credentials at all, because you're always pulling from the environment in every one of your rails_envs, so the yaml file is just an extra layer of indirection in your case that is useless.