I am trying to instantiate a file stored on s3 with rails to parse it. The first step is to access the file. I have built a quick script to get familiar with amazon aws-sdk gem.
require 'aws-sdk'
s3 = Aws::S3::Resource.new(
access_key_id: 'XXXX',
secret_access_key: 'XXXX',
region: 'us-east-1')
bucket = s3.bucket('my_bucket')
bucket.objects.each do |obj|
puts obj.key
end
This code will give me the list of all the objects (files) on my drive. It looks like
uploads/doc1.pdf
uploads/doc2.pdf
Now I want to understand how I can retrieve one single object. I tried
object = bucket.objects['doc2.pdf']
puts object.key
But I get the error method_missing': undefined method[]' for Aws::Resources::Collection:0x0055e82a514ab0> (NoMethodError)
I also tried
object = bucket.objects.find(1)
puts object.name
But I get the error undefined method `name' for Enumerator:0x0055855a098de8> (NoMethodError)
Could anyone give me the right syntax to retrieve one single object (file)? Once I get it, I intend to use the "open" method to instantiate the file and then a pdf parser. Thanks!
This has worked for me
object = bucket.object('path to file')
Related
I'm creating a Ruby script in Rails that will:
1) create an S3 object with AWS S3 SDK
2) iterate through bucket and download (get) each file
3) through iteration, store the file in memory and then convert to a string
4) parse the string for data and re-upload the file based on parsed data to an appropriate folder in same bucket
So the code I have so far in Rails jobs:
def aws_get
io = IO.new(1)
bucket_col = []
s3 = Aws::S3::Resource.new(region: 'us-east-1', access_key_id: Rails.application.credentials.dig(:aws, :access_key_id), secret_access_key: Rails.application.credentials.dig(:aws, :secret_access_key))
s3.bucket('missouridata').objects.each do |object|
obj = s3.bucket('missouridata').object(object.key)
file = obj.get(response_target: io)
???
end
end
The questions marks is where I don't know what to do next. How do I take the file stored in memory and convert it to a string to be parsed?
I have the perfect solution for you. I have beein using fog gem to manipulate S3 bucket for a while. It can do pretty much anything for you.
Here is the reference link.
https://www.ironin.it/blog/manipulating-files-on-amazon-s3-storage-with-rubys-fog-gem.html
I stuck all afternoon on checking whether an uploaded file to AWS S3 exists or not. I use Ruby On Rails and the gem called aws-sdk, v2.
First of all - the file exists in the bucket, it is located here:
test_bucket/users/10/file_test.pdf
There's no typo, this is the exact path. Also, the bucket + credentials are set up correctly.
And here's how I try to check the existence of the file:
config = {region: 'us-west-1', bucket: AWS_S3_CONFIG['bucket'], key: AWS_S3_CONFIG['access_key_id'], secret: AWS_S3_CONFIG['secret_access_key']}
Aws.config.update({region: config[:region],
credentials: Aws::Credentials.new(config[:key], config[:secret]),
:s3 => { :region => 'us-east-1' }})
bucket = Aws::S3::Resource.new.bucket(config[:bucket])
puts bucket.object("file_test.pdf").exists?
The output is always false.
I also tried puts bucket.object("test_bucket/users/10/file_test.pdf").exists?, but still false.
Also, I tried to make the file public in the AWS S3 dashboard, but no success, still false. The file is visible when click on the generated link.
But the problem is that when I check with using aws-sdk if the file exist, the output is still false.
What am I doing wrong?
Thank you.
You need to pass the full path to the object (not including the bucket name) - users/10/file_test.pdf
I'm attempting to download files from my S3 File server via a rails app that I've written. However, I'm having a difficult time trying to figure out how to accomplish this. I've been attempting to use this reference from Amazon's blog to get it working.
in the get method in my controller, I have the following:
asset = current_user.assets.find_by_id(params[:id])
File.open('filename', 'wb') do |file|
reap = s3.get_object({ bucket:'bucket-name', key: URI.encode(asset.uploaded_file.url)}, target: file)
end
however I'm getting the following error:
uninitialized constant AssetsController::s3
I'm using the gem aws-sdk. Any suggestions would be much appreciated.
uninitialized constant AssetsController::s3
You need to define s3, the below should work
asset = current_user.assets.find_by_id(params[:id])
File.open('filename', 'wb') do |file|
s3 = Aws::S3::Client.new
reap = s3.get_object({ bucket:'bucket-name', key: URI.encode(asset.uploaded_file.url)}, target: file)
end
So, first, I'll give you an idea of what I'm trying to accomplish. I host a bunch of photos on S3. They're car photos, and organized by VIN. At creation, I don't always have the VIN, so I make a fake one. Later on, when updated with the correct VIN, I want to update the folder name on S3. Can't rename stuff in S3, so I have to copy with a new name, and delete the original. Phew! Now on to how I'm failing...
creds = ::Aws::Credentials.new(Settings.aws.access_key_id, Settings.aws.secret_access_key)
s3 = ::Aws::S3::Client.new(region: 'us-west-1', credentials: creds)
bucket = Settings.aws.bucket
s3.copy_object(bucket: bucket, copy_source: "#{bucket}/foo", key: 'bar')
s3.delete_object(bucket: bucket, key: "#{bucket}/foo")
This does not work... Aws::S3::Errors::NoSuchKey: The specified key does not exist. when calling copy_object. copy_source is a folder, so it doesn't work like a regular object, fine.
So, I looked around and found that I have to call:
s3 = ::Aws::S3::Resource.new(region: 'us-west-1', credentials: creds)
bucket = bucket = s3.bucket(Settings.aws.bucket)
bucket.objects.with_prefix('foo/').each do |object|
object.copy_from(copy_source: "#{bucket}/foo")
end
This does not work... notice that I'm calling ::Resource now, as I don't know how to get a bucket from ::Client. With above, I get NoMethodError: undefined method 'with_prefix' for #<Aws::Resources::Collection:0x007fad587255a8> with baffles me, as everything I read seems to point to that solution.
I'm using v2 of the AWS SDK. Not sure if I'm looking at v1 solutions?
I have no idea how the copy_from works, honestly... what I really want is a copy_to which doesn't seem to be in the documentation.
aws-sdk-core-ruby github
aws sdk docs
The 'objects' method of 'bucket' class in aws-sdk v2 returns a Collection of ObjectSummary (See: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Bucket.html#objects-instance_method).
To query a list of objects with a prefix you should use:
bucket.objects({prefix: 'some_prefix'})
I use
bucket.objects.find_all { |object| object.key.include?('foo/') }
for aws sdk v2
Haven't seen much on testing for the soundcloud gem. Most everything has been splendid - however I found that uploading files to SoundCloud through my specs is a bit tricky.
Here's the problem:
spec:
file = "#{Rails.root}/spec/factories/files/sample.mp3"
title = "From the test suite"
track = #client.post('/tracks', track: {title: title, asset_data: File.new(file)})
Whether or not I use
File.new
or
File.expand_path
the error is the same: NoMethodError:
undefined method `map' for #String:0x007fa1ce8425c0
The successful code in the controller is:
file = params["soundcloud_file"]
title = params["title"]
track = client.post('/tracks', track: {
title: title
asset_data: File.new(file.tempfile, 'rb')
})
Now I understand that params["soundcloud_file"] is an ActionDispatch::Http::UploadedFile, so that's what I've been attempting to work with for the past hour or so with no luck.
I'm open to any way to successfully test an upload.
The problem was with the 'webmock' gem. Even using the required version for VCR (<1.10.0) it throws:
NoMethodError: undefined method `map' for <#String: ... >
Replacing with the 'fakeweb' gem corrects the error.