Seahorse::Client::NetworkingError Amazon S3 file upload with rails - ruby-on-rails

In my rails 4 application I'm trying to download and then upload a regular png file to my s3 bucket using the aws-sdk (using gem 'aws-sdk', '~> 2').
In development environment, the code works totally fine. But if I try rails s -e production or if I test out the upload on my heroku instance, I get the following error when I test out the image upload functionality,
Seahorse::Client::NetworkingError (Connection reset by peer):
app/helpers/aws_s3.rb:73:in `upload_to_s3'
app/controllers/evaluations_controller.rb:19:in `test'
my upload_to_s3 method mentioned in the trace looks like:
def upload_to_s3(folder_name)
url = "http://i.imgur.com/WKeQQox.png"
filename = "ss-" + DateTime.now.strftime("%Y%d%m-%s") + "-" + SecureRandom.hex(4) + ".png"
full_bucket_path = Pathname(folder_name.to_s).join(filename).to_s
file = save_to_tempfile(url, filename)
s3 = Aws::S3::Resource.new(access_key_id: ENV["IAM_ID"], secret_access_key: ENV["IAM_SECRET"], region: 'us-east-1')
s3_file = s3.bucket(ENV["BUCKET"]).object(full_bucket_path)
s3_file.upload_file(file.path)
raise s3_file.public_url.to_s.inspect
end
The environment variables are the same between both environments. I don't really know where else to turn to debug this. Why would it work in development, but not in production? I have a feeling I'm missing something pretty obvious.
UPDATE:
Let's simplify this further since I'm not getting very much feedback.
s3 = Aws::S3::Resource.new
bucket = s3.bucket(ENV["BUCKET"])
bucket.object("some_file.txt").put(body:'Hello World!')
The above totally works in my development environment, but not my production environment. In production it faults when I call put(body:'Hello World!'). I know this is probably related to write permissions or something, but again, I've checked my environment variables, and they are identical. Is there some configuration that I'm not aware of that I should be checking?
I've tried using a new IAM user. I also temporarily copied the entire contents of development.rb over to production.rb just to see if the configuration for the development or production was affecting it, to no avail. I ran bundle update as well. Again, no luck.
I wish the error was more descriptive, but it just says Seahorse::Client::NetworkingError (Connection reset by peer) no matter what I try.

Well I never found the solution for this problem and had to resort to other options since I was on a deadline. I'm assuming it is a bug on Amazon's end or with the aws-sdk gem, because I have checked my configuration many times, and it is correct.
My workaround was to use the fog gem, which is actually very handy. after adding gem 'fog' to my gemfile and running bundle install my code now looks like this:
def upload_to_s3(folder_name)
filename = "ss-" + DateTime.now.strftime("%Y%d%m-%s") + "-" + SecureRandom.hex(4) + ".png"
full_bucket_path = Pathname(folder_name.to_s).join(filename).to_s
image_contents = open(url).read
connection = Fog::Storage.new({
:provider => 'AWS',
:aws_access_key_id => ENV["AWS_ACCESS_KEY_ID"],
:aws_secret_access_key => ENV["AWS_SECRET_ACCESS_KEY"]
})
directory = connection.directories.get(ENV["BUCKET"])
file = directory.files.create(key: full_bucket_path, public: true)
file.body = image_contents
file.save
return file.public_url
end
Which is simple enough and was a breeze to implement. Wish I knew what was messing up with the aws-sdk gem, but for anyone else who has problems, give fog a go.

Related

Ruby aws-sdk - ".exists?" says the file doesn't exist even though I see it in the bucket

I stuck all afternoon on checking whether an uploaded file to AWS S3 exists or not. I use Ruby On Rails and the gem called aws-sdk, v2.
First of all - the file exists in the bucket, it is located here:
test_bucket/users/10/file_test.pdf
There's no typo, this is the exact path. Also, the bucket + credentials are set up correctly.
And here's how I try to check the existence of the file:
config = {region: 'us-west-1', bucket: AWS_S3_CONFIG['bucket'], key: AWS_S3_CONFIG['access_key_id'], secret: AWS_S3_CONFIG['secret_access_key']}
Aws.config.update({region: config[:region],
credentials: Aws::Credentials.new(config[:key], config[:secret]),
:s3 => { :region => 'us-east-1' }})
bucket = Aws::S3::Resource.new.bucket(config[:bucket])
puts bucket.object("file_test.pdf").exists?
The output is always false.
I also tried puts bucket.object("test_bucket/users/10/file_test.pdf").exists?, but still false.
Also, I tried to make the file public in the AWS S3 dashboard, but no success, still false. The file is visible when click on the generated link.
But the problem is that when I check with using aws-sdk if the file exist, the output is still false.
What am I doing wrong?
Thank you.
You need to pass the full path to the object (not including the bucket name) - users/10/file_test.pdf

Paperclip S3 timestamp interpolation is one second off?

I'm having a fair bit of trouble with a Rails app using Paperclip for attachments stored in S3 and delivered with Cloudfront. I've configured Paperclip with timestamp interpolation. Everything works on a staging environment as expected. However, in production, the actual filename being stored uses a timestamp that is one second off. I have no idea why....
Paperclip Configuration
if configatron.aws.enabled
Paperclip::Attachment.default_options[:storage] = :s3
Paperclip::Attachment.default_options[:s3_credentials] = {
access_key_id: configatron.aws.access_key,
secret_access_key: configatron.aws.secret_key
}
Paperclip::Attachment.default_options[:protocol] = 'https'
Paperclip::Attachment.default_options[:bucket] = configatron.aws.bucket
Paperclip::Attachment.default_options[:s3_host_alias] = configatron.aws.cloudfront_host
Paperclip::Attachment.default_options[:url] = ":s3_alias_url"
# Change the default path
Paperclip::Attachment.default_options[:path] = "images/:class/:attachment/:id_partition/:style-:timestamp-:filename"
Paperclip.interpolates(:timestamp) do |attachment, style|
attachment.instance_read(:updated_at).to_i
end
end
Example record
> alumni = Alumni.last
> alumni.updated_at.to_i
=> 1428719699
> alumni.thumb.url
=> "http://d2kektcjb0ajja.cloudfront.net/images/alumnis/thumbs/000/000/664/original-1428719699-diana-zeng-image.jpg?1428719699"
Notice that in the above IRB session, the updated_at timestamp matches the timestamp on the filename, which is correct. However, the file actually being stored in S3 is thumb-1428719698-diana-zeng-image.jpg, notice that the timestamp is 1 second off! This means that the above URL is not found.
This only happens on Production. On our staging environment, it works perfectly. I have no idea why the above would happen.
Can anyone help?
Thanks!
Leonard
Resolved by upgrading Rails to >= 4.2.1.
The problem is that Rails 4.2.0 preserves fractional seconds, causing the drift to happen due to rounding to the nearest second.
https://github.com/thoughtbot/paperclip/issues/1772

Refinerycms on Heroku not working with AmazonS3 bucket

I try to setup Amazon S3 support to store images in the cloud with refinerycms.
I created the bucket at https://console.aws.amazon.com/s3/
I named it like the app 'bee-barcelona' and it says it is in region US Standard
In ~/config/initializers/refinery/images.rb I entered all the data (where 'xxx? stands for the actual keys I entered:
# Configure S3 (you can also use ENV for this)
# The s3_backend setting by default defers to the core setting for this but can be set just for images.
config.s3_backend = Refinery::Core.s3_backend
config.s3_bucket_name = ENV['bee-barcelona']
config.s3_access_key_id = ENV['xxx']
config.s3_secret_access_key = ENV['xxx']
config.s3_region = ENV['xxx']
Then I applied the changes to heroku with:
heroku config:add S3_KEY=xxx S3_SECRET=xxx S3_BUCKET=bee-barcelona S3_REGION=us-standard
But still, in the app I only get: "Sorry, something wen wrong" when I try to upload.
What did I miss?
What a sad error. I didn't think about that option till I went for a 10 km run…
I had the app set up to be "beekeeping"
My bucket on Amazon was named "bee-barcelona"
I did register the correct bucket in the app. Still refinery tried to keep on going to another persons bucket, named "beekeeping". With my secret key there was no way my files would end up there.
I created a new app and a new bucket, all with crazy names, BUT! They are the same on AmazonS3 and GIT!!!
No it works like a charm.
What a very rare situation...
The way I did it was as follows:
Create a bucket in region US-STANDARD!!!!!!!!!!
Did you see that? US-STANDARD, not oregon, not anywhere else.
Add gems to Gemfile
gem "fog"
gem "unf"
gem "dragonfly-s3_data_store"
In config/application.rb
config.assets.initialize_on_precompile = true
In config/environments/production.rb
Refinery::Core.config.s3_backend = true
In config/environments/development.rb
Refinery::Core.config.s3_backend = false
Configure S3 for heroku (production) and local storage for development. In config/initializers/refinery/core.rb
if Rails.env.production?
config.s3_backend = true
else
config.s3_backend = false
end
config.s3_bucket_name = ENV['S3_BUCKET']
config.s3_region = ENV['S3_REGION']
config.s3_access_key_id = ENV['S3_ACCESS_KEY']
config.s3_secret_access_key = ENV['S3_SECRET_KEY']
Add variables to heroku:
heroku config:add S3_ACCESS_KEY=xxxxxx S3_SECRET_KEY=xxxxxx S3_BUCKET=bucket-name-here S3_REGION=us-east-1
I had a lot of issues because I had before S3_REGION=us-standard. This is WRONG. Set your US-Standard bucket as shown:
S3_REGION=us-east-1
This worked flawlessly for me on Rails 4.2.1 and refinery 3.0.0. Also, make sure you are using the exact same names for the variables. Sometimes it says S3_KEY instead of S3_ACCESS_KEY or S3_SECRET instead of S3_SECRET_KEY. Just make sure you have the same ones in your files and your Heroku variables.

Set System Directory Rails Production Environment

I have an app that works fine in on my development machine, but on my production server it uses a broken link to serve an image served using the Paperclip Gem.
Production environment is Linux(Debian), Apache, Passenger and I am deploying with Capistrano.
The app is stored in (a symlink that points to the public folder of the current version of the app deployed using capistrano):
/var/www/apps/root/appname
However, when I try and access it on the production server, the Apache error log displays this as the path it is looking in:
/var/www/apps/root/system
The correct path, however, is:
/var/www/apps/appname/shared/system
One option available to me is to create a symlink in root that directs system to the correct path, but I don't want to do this in case I want to deploy another app in the same root dir.
The url for this request is generated by rails, but Apache is what fetches the static resource (image files), so I have tried placing the following in my config/environments/production.rb:
ENV["RAILS_RELATIVE_URL_ROOT"] = '/appname/'
Which has resolved all other pathing issues I've been experiencing, but when rails generates the url (via the Paperclip gem), it doesn't seem to use it.
How can I set it so Paperclip uses the right path and only uses it production?
I've a workaround, add this as one of initializers:
config/initializer/paperclip.rb
Paperclip::Attachment.class_eval do
def url(style_name = default_style, options = {})
if options == true || options == false # Backwards compatibility.
res = #url_generator.for(style_name, default_options.merge(:timestamp => options))
else
res = #url_generator.for(style_name, default_options.merge(options))
end
# replace adding uri before res, minus final /
Rails.application.config.site_relative_url[0..-2]+res
end
end
At the moment Paperclip doesn’t work with ENV['RAILS_RELATIVE_URL_ROOT'] and the. You can follow the issue here:
https://github.com/thoughtbot/paperclip/issues/889

How to copy file across buckets using aws-s3 or aws-sdk gem in ruby on rails

The aws-s3 documentation says:
# Copying an object
S3Object.copy 'headshot.jpg', 'headshot2.jpg', 'photos'
But how do I copy heashot.jpg from the photos bucket to the archive bucket for example
Thanks!
Deb
AWS-SDK gem. S3Object#copy_to
Copies data from the current object to another object in S3.
S3 handles the copy so the client does not need to fetch the
data and upload it again. You can also change the storage
class and metadata of the object when copying.
It uses copy_object method internal, so the copy functionality allows you to copy objects within or between your S3 buckets, and optionally to replace the metadata associated with the object in the process.
Standard method (download/upload)
Copy method
Code sample:
require 'aws-sdk'
AWS.config(
:access_key_id => '***',
:secret_access_key => '***',
:max_retries => 10
)
file = 'test_file.rb'
bucket_0 = {:name => 'bucket_from', :endpoint => 's3-eu-west-1.amazonaws.com'}
bucket_1 = {:name => 'bucket_to', :endpoint => 's3.amazonaws.com'}
s3_interface_from = AWS::S3.new(:s3_endpoint => bucket_0[:endpoint])
bucket_from = s3_interface_from.buckets[bucket_0[:name]]
bucket_from.objects[file].write(open(file))
s3_interface_to = AWS::S3.new(:s3_endpoint => bucket_1[:endpoint])
bucket_to = s3_interface_to.buckets[bucket_1[:name]]
bucket_to.objects[file].copy_from(file, {:bucket => bucket_from})
Using the right_aws gem:
# With s3 being an S3 object acquired via S3Interface.new
# Copies key1 from bucket b1 to key1_copy in bucket b2:
s3.copy('b1', 'key1', 'b2', 'key1_copy')
the gotcha I ran into is that if you have pics/1234/yourfile.jpg the bucket is only pics and the key is 1234/yourfile.jpg
I got the answer from here: How do I copy files between buckets using s3 from a rails application?
For anyone still looking, AWS has documentation for this. It's actually very simple with the aws-sdk gem:
bucket = Aws::S3::Bucket.new('source-bucket')
object = bucket.object('source-key')
object.copy_to(bucket: 'target-bucket', key: 'target-key')
When using the AWS SDK gem's copy_from or copy_to there are three things that aren't copied by default: ACL, storage class, or server side encryption. You need to specify them as options.
from_object.copy_to from_object.key, {:bucket => 'new-bucket-name', :acl => :public_read}
https://github.com/aws/aws-sdk-ruby/blob/master/lib/aws/s3/s3_object.rb#L904
Here's a simple ruby class to copy all objects from one bucket to another bucket: https://gist.github.com/edwardsharp/d501af263728eceb361ebba80d7fe324
Multiple images could easily be copied using aws-sdk gem as follows:
require 'aws-sdk'
image_names = ['one.jpg', 'two.jpg', 'three.jpg', 'four.jpg', 'five.png', 'six.jpg']
Aws.config.update({
region: "destination_region",
credentials: Aws::Credentials.new('AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY')
})
image_names.each do |img|
s3 = Aws::S3::Client.new()
resp = s3.copy_object({
bucket: "destinationation_bucket_name",
copy_source: URI.encode_www_form_component("/source_bucket_name/path/to/#{img}"),
key: "path/where/to/save/#{img}"
})
end
If you have too many images, it is suggested to put the copying process in a background job.
I believe that in order to copy between buckets you must read the file's contents from the source bucket and then write it back to the destination bucket via your application's memory space. There's a snippet showing this using aws-s3 here and another approach using right_aws here
The aws-s3 gem does not have the ability to copy files in between buckets without moving files to your local machine. If that's acceptable to you, then the following will work:
AWS::S3::S3Object.store 'dest-key', open('http://url/to/source.file'), 'dest-bucket'
I ran into the same issue that you had, so I cloned the source code for AWS-S3 and made a branch that has a copy_to method that allows for copying between buckets, which I've been bundling into my projects and using when I need that functionality. Hopefully someone else will find this useful as well.
View the branch on GitHub.

Resources