I am setting the aws credentials on my application.yml
AWS_KEY: 'myKey'
AWS_SECRET: 'mySecret'
Then
s3 = Aws::S3::Client.new
successfully returns a client, but when I am trying to fetch an object
resp = s3.get_object(bucket:'my-bucket', key:'myFile.txt')
I am getting the following error.
Aws::Errors::MissingCredentialsError:
Related
Want to achieve
ruby 2.6.5
Rails 6.0.4
Amazon AWS S3
Thankyou for watching.
I am creating an application in Rails.
I want to upload an image to AWS S3, but I get the following error in the log and cannot upload it
rescue500 => SocketError : getaddrinfo: Name or service not known
/home/ec2-user/test_app/web/app/services/clamav_service.rb:4:in `initialize'
config/environments/production.rb
config.x.s3.local = false
config.x.s3.region = ENV["AWS_S3_REGION"]
config.x.s3.bucket = ENV["AWS_S3_BUCKET"]
config.x.s3.access_key = ENV["AWS_S3_ACCESS_KEY"]
config.x.s3.secret_access_key = ENV["AWS_S3_SECRET_ACCESS_KEY"]
config.x.s3.password = ENV["AWS_S3_PASSWORD"]
config.x.s3.path = ENV["AWS_S3_PATH"]
Environment variables in ec2
AWS_S3_REGION=ap-northeast-1
AWS_S3_BUCKET=xxx
AWS_S3_ACCESS_KEY=yyy
AWS_S3_SECRET_ACCESS_KEY=zzz
AWS_S3_PASSWORD=
AWS_S3_PATH=uploads/
We have made the following settings.
I have confirmed that AWS_S3_ACCESS_KEY and AWS_S3_SECRET_ACCESS_KEY are correct.
In my local environment, I was able to upload successfully, so I think it is a bug in my environment.
I would appreciate it if you could tell me if I missed anything.
clamd doesn't expand environment variables, so I had to write them directly.
I am using a Rails code (using AWS SDK) to do the following:
Upload a file to an S3 bucket
s3 = Aws::S3::Client.new(
access_key_id: <my key>,
secret_access_key: <my secret key>
)
s3.put_object(bucket: <my bucket>, key: <file name>, body: <file content>)
Send an email to the user stating that the file has been uploaded to S3 bucket
ses = Aws::SES::Client.new(region: 'us-west-2')
While step 1 works perfectly fine, I am getting this error when I try to instantiate the SES client in step# 2:
NameError uninitialized constant Aws::SES
Why AWS::Ses is giving a namespace error while Aws::S3 is working perfectly fine? Please help!
These are the related gems I am using:
aws-ses
aws-sdk-3
Please add gem 'aws-sdk-ses'
for more info check here https://rubygems.org/gems/aws-sdk-ses/versions/1.6.0
I am using CKAN 2.7.2.
I have added the following configurations in my development.ini file of ckan:
ckan.oauth2.authorization_endpoint = https://wso2IP/oauth2/authorize
ckan.oauth2.token_endpoint = https://wso2IP/oauth2/token
ckan.oauth2.profile_api_url = https://wso2IP/userinfo
ckan.oauth2.client_id = *client-id*
ckan.oauth2.client_secret = *clientsecret*
ckan.oauth2.profile_api_user_field = abc
ckan.oauth2.profile_api_mail_field = abc#gmail.com
Also, have exported the following while running ckan using paster serve :
export OAUTHLIB_INSECURE_TRANSPORT=True
Also, I have added an application (ckan provider) in WSO2 Identity Server also with callback URL = (http://ckan-url:5000/oauth2/callback) where the CKAN instance is running (i.e a private IP of 172.30.66.XX type running on port 5000).
So, what configuration need to be done in WSO2 Identity Server and ckan development.ini file?
Also, configuring on both sides, I get the error Client authentication Failed in CKAN GUI.
I am using Ckan 2.7.2 , WSO2-IS 5.1 and ckanext-oauth2 (oauthlib==0.8.0)
will it be giving a problem in authentication process??Can it be a version dependency issue??
I have followed the given steps. Kindly correct if I am missing or doing something wrong:
I have used API store and published ckan application.
Entered the ckan callback url as http://oauth2/callback
Entered the generated client ID and client secret from step 1 in ckan configuration.
Also, made the given changes mentioned by you regarding claim configurations and ckanext-oauth2 configurations.
Now, it gives above error.or sometimes Invalid Client (Client Authentication Failed)
Inbound settings:
enter image description here
Error Logs :
[2018-07-25 19:29:25,432] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2018-07-25 19:29:25,431+0530]
[2018-07-25 21:55:15,930] INFO {org.wso2.carbon.identity.application.authentication.framework.store.OperationCleanUpService} - Session Operation Data cleanup task is running successfully for removing expired Operation Data
[2018-07-25 23:40:15,929] INFO {org.wso2.carbon.identity.application.authentication.framework.store.SessionCleanUpService} - Session Data cleanup task is running successfully for removing expired Data
[2018-07-26 09:21:18,948] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2018-07-26 09:21:18,948+0530]
[2018-07-26 09:21:39,307] INFO {org.wso2.carbon.core.services.authentication.AuthenticationAdmin} - 'admin#carbon.super [-1234]' logged out at [2018-07-26 09:21:39,0306]
[2018-07-26 09:55:15,975] INFO {org.wso2.carbon.identity.application.authentication.framework.store.OperationCleanUpService} - Session Operation Data cleanup task is running successfully for removing expired Operation Data
After configuring the settings I receive the following Message:
Your application settings at WSO2-IS seems to be correct. The only thing to take into consideration, it is that ckanext-oauth2 is only able to use the Code and, the Refresh Token grant types, so only those types need to be allowed.
Using this configuration for ckanext-oauth2:
ckan.oauth2.authorization_endpoint = https://wso2IP/oauth2/authorize
ckan.oauth2.token_endpoint = https://wso2IP/oauth2/token
ckan.oauth2.profile_api_url = https://wso2IP/userinfo
ckan.oauth2.client_id = *** OAuth Client Key from the Inbound Authentication Configuration at WSO2 ***
ckan.oauth2.client_secret = *** OAuth Client Secret from the Inbound Authentication Configuration at WSO2 ***
ckan.oauth2.scope = all_info openid
ckan.oauth2.profile_api_user_field = email
ckan.oauth2.profile_api_fullname_field = name
ckan.oauth2.profile_api_mail_field = email
and using your claim mapping configuration (http://wso2.org/claims/fullname and http://wso2.org/claims/emailaddress); I was able to sign in into CKAN. I have tested using CKAN v2.7.4, v2.8.0 and v2.8.1; ckanext-oauth2 v0.6.1 and WSO2-IS v5.5.0 (although it should work using WSO2-IS v5.1).
I've followed the instructions on a couple of pages for getting a sitemap to generate and be uploaded to my S3 Bucket. The sitemap is generating, but not uploading.
I'm using carrierwave for the upload, which is working fine for image uploads.
The key file seems to be config/sitemap.rb. Here's mine:
require 'rubygems'
require 'sitemap_generator'
# Set the host name for URL creation
SitemapGenerator::Sitemap.default_host = "https://www.driverhunt.com"
# pick a place safe to write the files
SitemapGenerator::Sitemap.public_path = 'tmp/'
# store on S3 using #Fog# Carrierwave
SitemapGenerator::Sitemap.adapter = SitemapGenerator::WaveAdapter.new
# SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new
# This is a different problem to the one in the question, but using this second adaptor gives the error: "...lib/fog/storage.rb:27:in `new': is not a recognized storage provider (ArgumentError)"
# inform the map cross-linking where to find the other maps
SitemapGenerator::Sitemap.sitemaps_host = "http://#{ENV['S3_BUCKET']}.s3.amazonaws.com/"
# pick a namespace within your bucket to organize your maps
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'
SitemapGenerator::Sitemap.create do
add '/home', :changefreq => 'daily', :priority => 0.9
# add '/contact_us', :changefreq => 'weekly'
end
# SitemapGenerator::Sitemap.ping_search_engines # Not needed if you use the rake tasks
What's going on? How do I debug a carrierwave upload?
I will answer the question as your comment for the S3Adapter brought me to this topic while I was googling the not recognized provider. If you turn back on the comment using the S3Adapter and do the following you will get it working.
If you do not specify any fog ENV VARS for the fog-aws gem you will get the error:
ArgumentError: is not a recognized provider
by using as an adapter the SitemapGenerator::S3Adapter.new
The setup you have got above is perfectly fine, just use the S3Adapter.new instead of the WaveAdapter!
The error you are getting (and I was getting as well) is due to the fact that SitemapGenerator::S3Adapter uses fog-aws and in order to make it run by default you should have the following ENV VARS:
ENV['AWS_ACCESS_KEY_ID'] = XXX
ENV['AWS_SECRET_ACCESS_KEY'] = XXX
ENV['FOG_PROVIDER'] = AWS
ENV['FOG_DIRECTORY'] = your-bucket-name
ENV['FOG_REGION'] = your-bucket-region (ex: us-west-2)
If you are missing even one of the following you will get the error:
ArgumentError: is not a recognized provider
Alternativelly, if you want to avoid using ENV VARS for some reason you should specify the values when you initialize your adapter as follows:
SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new(fog_provider: 'AWS',
aws_access_key_id: 'your-access-key-id',
aws_secret_access_key: 'your-access-key',
fog_directory: 'your-bucket',
fog_region: 'your-aws-region')
However using just the above ENV VARS you will be fine and get your sitemap up and running. This setup was tested with sitemap_generator version: 5.1.0
For your question:
The Image uploading works as it does not require the exact same configuration as the WaveAdapter. I am guessing that your carrierwave.rb file is missing the following:
config.cache_dir = "#{Rails.root}/tmp/"
config.permissions = 0666
The complete configuration for the carrierwave initializer can be found here:
Generate Sitemaps on read only filesystems like Heroku (check if you are missing something or use the other adapter)
However, I believe that your problem has to do with missing ENV VARS from the production environment.
I am creating an API for a backend service with Rails 4.
The service needs to upload an image file to an amazon s3 bucket.
I'd like to use a direct upload url, so that the clients manage the uploads to s3 and the server is not kept busy.
Currently I have the following prototypical rails action
def create
filename = params[:filename]
s3_direct_post = S3_BUCKET.presigned_post(key: "offers/#{SecureRandom.uuid}/#{filename}", acl: 'public-read')
s3p = s3_direct_post.fields
url = "#{s3_direct_post.url}/#{filename}?X-Amz-Algorithm=#{s3p['x-amz-algorithm']}&X-Amz-Credential=#{s3p['x-amz-credential']}&X-Amz-Date=#{s3p['x-amz-date']}&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=#{s3p['x-amz-signature']}"
render json: {success: true, url: url}, status: :ok
end
This generates such an url:
https://my-bucket.s3.eu-central-1.amazonaws.com/test.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=MYKEY/20150420/eu-central-1/s3/aws4_request&X-Amz-Date=20150420T162603Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=MYSIGNATURE
Now I try to post the test.png to this url with the following:
curl -v -T test.png "url"
and I get the following error response:
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>MYKEY</AWSAccessKeyId>...
I believe the problem comes from the fact, that the specified X-Amz-SignedHeaders Header is wrong. I am not sure which headers are used by default from the amazon rails sdk gem.
How should I change my url generation, so that a mobile client can just take the url and post a file to it?
Here is a solution:
In config/initializers/aws.rb:
AWS_CREDS = Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
Aws.config.update({
region: 'eu-central-1',
credentials: AWS_CREDS
})
S3 = Aws::S3::Resource.new('eu-central-1')
S3_BUCKET_NAME = ENV['S3_BUCKET_NAME']
S3_BUCKET = S3.bucket(S3_BUCKET_NAME)
In your model/controller/concern/or whatever:
obj = S3_BUCKET.object("offers/#{user.id}/#{self.id}")
url = obj.presigned_url(:put) # obj.presigned_url(:put, acl: 'public-read') #if you want to make the file public
Then to upload you can use a mobile client or curl:
curl -X PUT -T file_to_upload "url from above"
Note that you will have to add the x-amz-acl: public-read header if you used the public-read acl option:
curl -H "x-amz-acl: public-read" -X PUT -T file_to_upload "url from above"