I'm trying to upload image png/jpg less than 10 kilobytes to s3 amazon.
The upload succeed but the file uploaded and stored with 0 bytes.
When i'm trying to see the image in the link provided by s3- i get blank.
If i upload image more bigger than 10 kilobytes size it's ok.
Can someone have any idea what is the problem please?
file_name = account[:img_token] + File.extname(img.original_filename)
file = Tempfile.new(file_name, encoding: 'ascii-8bit')
file.write(img.read)
path = file.path
bucket_name = 'bucket'
s3 = AWS::S3.new(access_key_id: ENV['S3_ACCESS_KEY'], secret_access_key: ENV['S3_SECRET_ACCESS_KEY'])
link = 'https://s3-eu-west-1.amazonaws.com/' + bucket_name + '/' + file_name
key = file_name
object = s3.buckets[bucket_name].objects[key].write(file: path, acl: 'public-read')
It looks like you are passing a file to the SDK that is seeked to the end of the file. I suspect the SDK is calling #read and getting nil back. Try rewinding the file first.
According to this answer
Write, read, and delete objects containing from 1 byte to 5 terabytes
of data each. The number of objects you can store is unlimited.
Related
I'm creating a Ruby script in Rails that will:
1) create an S3 object with AWS S3 SDK
2) iterate through bucket and download (get) each file
3) through iteration, store the file in memory and then convert to a string
4) parse the string for data and re-upload the file based on parsed data to an appropriate folder in same bucket
So the code I have so far in Rails jobs:
def aws_get
io = IO.new(1)
bucket_col = []
s3 = Aws::S3::Resource.new(region: 'us-east-1', access_key_id: Rails.application.credentials.dig(:aws, :access_key_id), secret_access_key: Rails.application.credentials.dig(:aws, :secret_access_key))
s3.bucket('missouridata').objects.each do |object|
obj = s3.bucket('missouridata').object(object.key)
file = obj.get(response_target: io)
???
end
end
The questions marks is where I don't know what to do next. How do I take the file stored in memory and convert it to a string to be parsed?
I have the perfect solution for you. I have beein using fog gem to manipulate S3 bucket for a while. It can do pretty much anything for you.
Here is the reference link.
https://www.ironin.it/blog/manipulating-files-on-amazon-s3-storage-with-rubys-fog-gem.html
I am using Libreconv gem to convert word to doc but it's not working with S3
bucket = Aws::S3::Bucket.new('bucket-name')
object = bucket.object file.attachment.blob.key
path = object.presigned_url(:get)
Libreconv.convert(path, "public/test.pdf")
If I try to convert this path to PDF using Libreconv then it's give me filename too long error. I have wrriten this code under ActiveJobs. So kindly provide me solutions as per ActiveJobs.
Can someone please suggest me how can I convert word file to pdf.
Here path is https://domain.s3.amazonaws.com/Bf5qPUP3znZGCHCcTWHcR5Nn?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZ6RZ7J425ORVUYQ%2F20181206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181206T051240Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=b89c47a324b2aa423bf64dfb343e3b3c90dce9b54fa9fe1bc4efa9c248e912f9
and error I am getting is
Error: source file could not be loaded
*** Errno::ENAMETOOLONG Exception: File name too long # rb_sysopen - /tmp/Bf5qPUP3znZGCHCcTWHcR5Nn?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZ6RZ7J425ORVUYQ%2F20181206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181206T051240Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=b89c47a324b2aa423bf64dfb343e3b3c90dce9b54fa9fe1bc4efa9c248e912f9.pd
It seems that you PDF is created with all the params needed to fetch docx from S3.
I suppose it happens in this line:
target_tmp_file = "#{target_path}/#{File.basename(#source, ".*")}.#{File.basename(#convert_to, ":*")}"
#source is https://domain.s3.amazonaws.com/Bf5qPUP3znZGCHCcTWHcR5Nn?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZ6RZ7J425ORVUYQ%2F20181206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181206T051240Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=b89c47a324b2aa423bf64dfb343e3b3c90dce9b54fa9fe1bc4efa9c248e912f9 and
> File.basename(#source, ".*")
=> "Bf5qPUP3znZGCHCcTWHcR5Nn?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZ6RZ7J425ORVUYQ%2F20181206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181206T051240Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=b89c47a324b2aa423bf64dfb343e3b3c90dce9b54fa9fe1bc4efa9c248e912f9"
As a result Libreconv gem tries to create a tmp file with this long name and it's too long - that's why an error is raised.
Possible solution: split the process into separate steps of fetching file and converting it. Something like:
require "open-uri"
bucket = Aws::S3::Bucket.new('bucket-name')
object = bucket.object file.attachment.blob.key
path = object.presigned_url(:get)
doc_file = open(path)
begin
Libreconv.convert(doc_file.path, "public/test.pdf")
ensure
doc_file.delete
end
following is the answer using combine pdf gem
tape = Tape.new(file)
result = tape.preview
tempfile = Tempfile.new(['foo', '.pdf'])
File.open(tempfile, 'wb') do |f|
f.write result
end
path = tempfile.path
combine_pdf(path)
and for load file for S3 I have used
object = #bucket.object object_key
path = object.presigned_url(:get)
response = Net::HTTP.get_response(URI.parse(path)).body
I am using amazon s3, rails 4, and the FOG gem. I have an amazon bucket called uipstudy with 100 folders, each containing about 20 images. I use the following to get all the images in a specific folder (In my application_helper.rb which is included in the application_controller.rb).
def get_files(image_folder)
connection = Fog::Storage.new(
provider: 'AWS',
aws_access_key_id: '######',
aws_secret_access_key: '#######'
)
connection.directories.get('uipimages', prefix:image_folder).files.map do |file|
file.key
end
end
In my controller I have this....in this example I am looking in the folder "1" in the uipstudy bucket.
#Amazon solution:
#images = get_files('1')
#images.each do |image|
image = "https://s3.amazonaws.com/uipstudy/#{image}"
#image_array << image
end
The problem is that its returning the files inside the folder labelled "1" but also in 10, 11, 12,13....etc. I assumed that the prefix was an absolute but it appears not. Is there a way to enforce that the prefix gets exactly the folder specified in the prefix?
I think you should be able to make a small change in your script to get the behavior you want. Simply append a forward slash to the prefix so that it clearly shows you want things that are like a directory instead of any/all things that begin with a particular character.
So, that would get you something like:
directory = connection.directories.get('upimages', prefix: image_folder + '/')
directory.files.map do |file|
file.key
end
(I just split it into two commands to make it format/read easier)
Below is my solution using the aws-sdk gem.
initialize s3 client
s3 = AWS::S3.new
bucket = s3.buckets[ENV['AWS_BUCKET']]
regex for ipa files in _inbox folder
regex = %r{_inbox/(?:[^/]+/)*[^/]+\.ipa}i
get and process ipa files
bucket.objects.select { |o| o.key.match(regex) }.each do |ipa|
The code below resizes my image. But I am not sure how to write it out to a temp file or blob so I can upload it to s3.
origImage = MiniMagick::Image.open(myPhoto.tempfile.path)
origImage.resize "200x200"
thumbKey = "tiny-#{key}"
obj = bucket.objects[thumbKey].write(:file => origImage.write("tiny.jpg"))
I can upload the original file just fine to s3 with the below command:
obj = bucket.objects[key].write('data')
obj.write(:file => myPhoto.tempfile)
I think I want to create a temp file, read the image file into it and upload that:
thumbFile = Tempfile.new('temp')
thumbFile.write(origImage.read)
obj = bucket.objects[thumbKey].write(:file => thumbFile)
but the origImage class doesn't have a read command.
UPDATE: I was reading the source code and found this out about the write command
# Writes the temporary file out to either a file location (by passing in a String) or by
# passing in a Stream that you can #write(chunk) to repeatedly
#
# #param output_to [IOStream, String] Some kind of stream object that needs to be read or a file path as a String
# #return [IOStream, Boolean] If you pass in a file location [String] then you get a success boolean. If its a stream, you get it back.
# Writes the temporary image that we are using for processing to the output path
And the s3 api docs say you can stream the content using a code block like:
obj.write do |buffer, bytes|
# writing fewer than the requested number of bytes to the buffer
# will cause write to stop yielding to the block
end
How do I change my code so
origImage.write(s3stream here)
http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html
UPDATE 2
This code successfully uploads the thumbnail file to s3. But I would still love to know how to stream it up. It would be much more efficient I think.
#resize image and upload a thumbnail
smallImage = MiniMagick::Image.open(myPhoto.tempfile.path)
smallImage.resize "200x200"
thumbKey = "tiny-#{key}"
newFile = Tempfile.new("tempimage")
smallImage.write(newFile.path)
obj = bucket.objects[thumbKey].write('data')
obj.write(:file => newFile)
smallImage.to_blob ?
below code copy from https://github.com/probablycorey/mini_magick/blob/master/lib/mini_magick.rb
# Gives you raw image data back
# #return [String] binary string
def to_blob
f = File.new #path
f.binmode
f.read
ensure
f.close if f
end
Have you looked into the paperclip gem? The gem offers direct compatibility to s3 and works great.
I have this code, which writes a zip file to disk, reads it back, uploads to s3, then deletes the file:
compressed_file = some_temp_path
Zip::ZipOutputStream.open(compressed_file) do |zos|
some_file_list.each do |file|
zos.put_next_entry(file.some_title)
zos.print IO.read(file.path)
end
end # Write zip file
s3 = Aws::S3.new(S3_KEY, S3_SECRET)
bucket = Aws::S3::Bucket.create(s3, S3_BUCKET)
bucket.put("#{BUCKET_PATH}/archive.zip", IO.read(compressed_file), {}, 'authenticated-read')
File.delete(compressed_file)
This code works already but what I want is to not create the zip file anymore, to save a few steps. I was wondering if there is a way to export the zipfile data directly to s3 without having to first create a tmpfile, read it back, then delete it?
I think I just found the answer to my question.
It's Zip::ZipOutputStream.write_buffer. I'll check this out and update this answer when I get it working.
Update
It does work. My code is like this now:
compressed_filestream = Zip::ZipOutputStream.write_buffer do |zos|
some_file_list.each do |file|
zos.put_next_entry(file.some_title)
zos.print IO.read(file.path)
end
end # Outputs zipfile as StringIO
s3 = Aws::S3.new(S3_KEY, S3_SECRET)
bucket = Aws::S3::Bucket.create(s3, S3_BUCKET)
compressed_filestream.rewind
bucket.put("#{BUCKET_PATH}/archive.zip", compressed_filestream.read, {}, 'authenticated-read')
The write_buffer returns a StringIO and needs to rewind the stream first before reading it. Now I don't need to create and delete the tmpfile.
I'm just wondering now if write_buffer would be more memory extensive or heavier than open? Or is it the other way around?