I am using a Rails code (using AWS SDK) to do the following:
Upload a file to an S3 bucket
s3 = Aws::S3::Client.new(
access_key_id: <my key>,
secret_access_key: <my secret key>
)
s3.put_object(bucket: <my bucket>, key: <file name>, body: <file content>)
Send an email to the user stating that the file has been uploaded to S3 bucket
ses = Aws::SES::Client.new(region: 'us-west-2')
While step 1 works perfectly fine, I am getting this error when I try to instantiate the SES client in step# 2:
NameError uninitialized constant Aws::SES
Why AWS::Ses is giving a namespace error while Aws::S3 is working perfectly fine? Please help!
These are the related gems I am using:
aws-ses
aws-sdk-3
Please add gem 'aws-sdk-ses'
for more info check here https://rubygems.org/gems/aws-sdk-ses/versions/1.6.0
Related
While creating the presigned url for my private image.png file in my s3 bucket, i used the below template
require 'aws-sdk-s3'
s3 = Aws::S3::Client.new(
region: 'us-east-1',
access_key_id: Access_key_id,
secret_access_key: Secret_access_key
)
signer = Aws::S3::Presigner.new(client: s3)
url = signer.presigned_url(
:get_object,
bucket: 'mybuck1',
key: "${image.png}-#{SecureRandom.uuid}"
)
while running the code, i get the follwing error as
AuthorizationQueryParametersError
Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.
so what might be the reason of this error, and how to fix this error
thanks in advance
I've been spending hours on this but cannot come up with any solutions. I've scoured the docs to see what I'm missing but can't find anything.
I'm using JWT and ActiveStorage with Amazon S3. I need to hide this keys before I can commit to Github. Following many examples, here's how my credentials.yml.enc file looks:
amazon:
key: <my access key for AWS S3>
secret: <my secret id for AWS S3>
jwt: <my jwt token encode/decode password>
I save this then in my storage.yml file I have:
test:
service: Disk
root: <%= Rails.root.join("tmp/storage") %>
local:
service: Disk
root: <%= Rails.root.join("storage") %>
# Use rails credentials:edit to set the AWS secrets (as aws:access_key_id|secret_access_key)
amazon:
service: S3
access_key_id: <%= Rails.application.credentials[:amazon][:key] %>
secret_access_key: <%= Rails.application.credentials[:amazon][:secret] %>
region: us-east-1
bucket: your_own_bucket
I've also tried with
<%= Rails.application.credentials[Rails.env.to_sym][:amazon][:key] %>
If I goto console and try Rails.application.credentials[:amazon][:key] or Rails.application.credentials.amazon[:key] or any other variants, it's always nil.
Rails.application.credentials
gives me:
=> #<ActiveSupport::EncryptedConfiguration:0x00007faf1284aa80
#key_path=#<Pathname:/Users/demiansims/Development/Beastly/beastly-
backend/config/master.key>, #content_path=#
<Pathname:/Users/demiansims/Development/Beastly/beastly-
backend/config/credentials.yml.enc>, #env_key="RAILS_MASTER_KEY",
#raise_if_missing_key=false, #encryptor=#
<ActiveSupport::MessageEncryptor:0x00007faf1287bb80
#secret=">^\x04\x9Bh\xFEb\x00\x8B\xB3O5\xDC\x8E\xA6b",
#sign_secret=nil, #cipher="aes-128-gcm", #aead_mode=true,
#verifier=ActiveSupport::MessageEncryptor::NullVerifier,
#serializer=Marshal, #options={:cipher=>"aes-128-gcm"}, #rotations=[]>,
#config={}
You credential file might not be saving your changes. I had this same problem. I was using sublime and EDITOR="subl --wait" bin/rails credentials:edit would bring up the editor but it would not recognize when I had saved the file. I never solved this, so I switched to vi. EDITOR="vi" bin/rails credentials:edit and problem solved. Now it all works as expected. If you are not familiar with vi commands like me, these are helpful to know:
i -> "insert" will allow to move around with your arrow keys, delete, and add code
esc -> when you are finished editing hit escape
:wq -> this will save and exit the file. after you hit escape just type in :wq
hope this helps someone.
I think there's a problem in the format of your credentials.yml.enc it should be like:
amazon:
key: <my access key for AWS S3>
secret: <my secret id for AWS S3>
jwt: <my jwt token encode/decode password>
So you can access your key like Rails.application.credentials.amazon[:key]
I am setting the aws credentials on my application.yml
AWS_KEY: 'myKey'
AWS_SECRET: 'mySecret'
Then
s3 = Aws::S3::Client.new
successfully returns a client, but when I am trying to fetch an object
resp = s3.get_object(bucket:'my-bucket', key:'myFile.txt')
I am getting the following error.
Aws::Errors::MissingCredentialsError:
I've followed the instructions on a couple of pages for getting a sitemap to generate and be uploaded to my S3 Bucket. The sitemap is generating, but not uploading.
I'm using carrierwave for the upload, which is working fine for image uploads.
The key file seems to be config/sitemap.rb. Here's mine:
require 'rubygems'
require 'sitemap_generator'
# Set the host name for URL creation
SitemapGenerator::Sitemap.default_host = "https://www.driverhunt.com"
# pick a place safe to write the files
SitemapGenerator::Sitemap.public_path = 'tmp/'
# store on S3 using #Fog# Carrierwave
SitemapGenerator::Sitemap.adapter = SitemapGenerator::WaveAdapter.new
# SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new
# This is a different problem to the one in the question, but using this second adaptor gives the error: "...lib/fog/storage.rb:27:in `new': is not a recognized storage provider (ArgumentError)"
# inform the map cross-linking where to find the other maps
SitemapGenerator::Sitemap.sitemaps_host = "http://#{ENV['S3_BUCKET']}.s3.amazonaws.com/"
# pick a namespace within your bucket to organize your maps
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'
SitemapGenerator::Sitemap.create do
add '/home', :changefreq => 'daily', :priority => 0.9
# add '/contact_us', :changefreq => 'weekly'
end
# SitemapGenerator::Sitemap.ping_search_engines # Not needed if you use the rake tasks
What's going on? How do I debug a carrierwave upload?
I will answer the question as your comment for the S3Adapter brought me to this topic while I was googling the not recognized provider. If you turn back on the comment using the S3Adapter and do the following you will get it working.
If you do not specify any fog ENV VARS for the fog-aws gem you will get the error:
ArgumentError: is not a recognized provider
by using as an adapter the SitemapGenerator::S3Adapter.new
The setup you have got above is perfectly fine, just use the S3Adapter.new instead of the WaveAdapter!
The error you are getting (and I was getting as well) is due to the fact that SitemapGenerator::S3Adapter uses fog-aws and in order to make it run by default you should have the following ENV VARS:
ENV['AWS_ACCESS_KEY_ID'] = XXX
ENV['AWS_SECRET_ACCESS_KEY'] = XXX
ENV['FOG_PROVIDER'] = AWS
ENV['FOG_DIRECTORY'] = your-bucket-name
ENV['FOG_REGION'] = your-bucket-region (ex: us-west-2)
If you are missing even one of the following you will get the error:
ArgumentError: is not a recognized provider
Alternativelly, if you want to avoid using ENV VARS for some reason you should specify the values when you initialize your adapter as follows:
SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new(fog_provider: 'AWS',
aws_access_key_id: 'your-access-key-id',
aws_secret_access_key: 'your-access-key',
fog_directory: 'your-bucket',
fog_region: 'your-aws-region')
However using just the above ENV VARS you will be fine and get your sitemap up and running. This setup was tested with sitemap_generator version: 5.1.0
For your question:
The Image uploading works as it does not require the exact same configuration as the WaveAdapter. I am guessing that your carrierwave.rb file is missing the following:
config.cache_dir = "#{Rails.root}/tmp/"
config.permissions = 0666
The complete configuration for the carrierwave initializer can be found here:
Generate Sitemaps on read only filesystems like Heroku (check if you are missing something or use the other adapter)
However, I believe that your problem has to do with missing ENV VARS from the production environment.
To avoid passing in access keys and secret aws access on a yml file I use the following :
development:
bucket: development
access_key_id: <%= ENV["S3_KEY"] %>
secret_access_key: <%= ENV["S3_SECRET"] %>
and then when running i get the error
Could not log "sql.active_record" event. ArgumentError: invalid byte sequence in UTF-8
PG::Error: ERROR: invalid byte sequence for encoding "UTF8": 0xe7 0xe3 0x6f
If I write my access key and secret directly on yml, like:
development:
bucket: development
access_key_id: MYACCESSKEY
secret_access_key: MYSECRETKEY
it goes smoothly.
Why does this error happen? How can i fix it without loading my key and secret into the yml file?
Edit
To load the environment variables onto development, I'm using the solution explained here
# Load the app's custom environment variables here, so that they are loaded before environments/*.rb
app_environment_variables = File.join(Rails.root, 'config', 'app_environment_variables.rb')
load(app_environment_variables) if File.exists?(app_environment_variables)
Might this be a problem with the loading process?
Edit 2
In the meanwhile, I tried to log what seems to be on my S3_CONFIG variable, loaded with:
config/initializers/load_config.rb
S3_CONFIG = YAML.load_file("#{::Rails.root}/config/s3.yml")[Rails.env]
I get
S3 Config: {"bucket"=>"mybucket", "access_key_id"=>"<%= ENV[\"S3_KEY\"] %>", "secret_access_key"=>"<%= ENV[\"S3_SECRET\"] %>"}
Wasn't it supposed to load the environment key already? May this be my problem?
This problem was happening when I was downloading the file from S3 with :
s3=AWS::S3.new(
access_key_id: S3_CONFIG["access_key_id"],
secret_access_key: S3_CONFIG["secret_access_key"])
and S3_CONFIG["access_key_id"] is just a string <%= ENV[\"S3_KEY\"] %>.
My solution for this was using just
s3=AWS::S3.new(
access_key_id: ENV['S3_KEY'],
secret_access_key: ENV['S3_SECRET'])
Guess sometimes one just needs to understand what he is doing, before pasting in lines of code...