I have a image in s3 bucket and url to access it.
I want to read the image from the s3 and to create a thumbnail icon and push the thumbnail_icon to s3.
If the image is in local, I can read the image and convert it to StringIO. After that I can push the StringIO to create thumbnail image in s3:
item = File.read(url)
data_io = StringIO.new(item)
s3_connection.interface.put(data_io, ...)
how can I open remote file and process it?
File.open(remote_url) returns No such file or directory
with OpenURI I can read the file. But I couldn't convert it to StringIO
response = open(remote_url) #Tempfile
data_io = StringIO.new(response)
#can't convert Tempfile into String`
What am I missing ?
The StringIO initialize method expects a string as the only parameter. The object you are giving it is a Tempfile. Try this:
data_io = StringIO.new(response.read)
Related
I am hitting an api to get an image that they have stored and use it as the profile pic for our application's users. I'm using Ruby on Rails and ActiveStorage with AWS to attach and store the image. What they send back is this:
{"status"=>"shared", "values"=>[{"$objectType"=>"BinaryData", "data"=>"/9j/4AAQSkZJRgABAQAASABIAAD/4QBMRXhpZgAAT.....KK5tT/9k=", "mime_type"=>"image/jpeg", "metadata"=>{"cropped"=>false}}]}
I tried a lot of different ways to attach it and manipulate the data such as just attaching it as it is, Base64.decode64, Base64.encode64. I also tried creating a new file and then attaching that. Here are some examples:
data = Base64.decode64(Base64.encode64(response[:selfie_image]["values"][0]["data"]))
user.profile_pic.attach!(
io: StringIO.new(open(data).read),
filename: "#{user.first_name}_#{user.last_name}_#{user.id}.jpg",
content_type: "image/jpeg"
)
data = Base64.decode64(Base64.encode64(response[:selfie_image]["values"][0]["data"]))
out_file = File.new("#{user.first_name}_#{user.last_name}_# . {user.id}.jpg", "w")
out_file.puts(data)
out_file.close
user.profile_pic.attach(
io: StringIO.new(open(out_file).read),
filename: "#{user.first_name}_#{user.last_name}_#{user.id}.jpg",
content_type: "image/jpeg"
)
I also tried:
user.profile_pic.attach(out_file)
It keeps either saying attachment is nil or depending on how I manipulate the data it will say not a jpeg file content header type wrong and throw that as an image magick error.
How can I manipulate this data to be able to attach it as an image for our users with ActiveStorage?
To get this to work I had to add gem "mini_magick" to my gemfile and then use it to decode the image data I was receiving from the api call and then turn it into a blob so that ActiveStorage could handle it.
data = response[:selfie_image]["values"][0]["data"]
decoded_data = Base64.decode64(data)
image = MiniMagick::Image.read(decoded_data)
image.format("png")
user.profile_pic.attach(
io: StringIO.new(image.to_blob),
filename: "#{user.id}_#{user.first_name}_#{user.last_name}.png",
content_type: "image/jpeg"
)
In command line ImageMagick you can use the inline: feature to decode base64 data into a gif or jpg. The base64 image here has transparency, it is most proper to save to gif or png.
convert 'inline:data:image/gif;base64,
R0lGODlhIAAgAPIEAAAAAB6Q/76+vvXes////wAAAAAAAAAAACH5BAEAAAUALAAA
AAAgACAAAAOBWLrc/jDKCYG1NBcwegeaxHkeGD4j+Z1OWl4Yu6mAYAu1ebpwL/OE
YCDA0YWAQuJqRwsSeEyaRTUwTlxUqjUymmZpmeI3u62Mv+XWmUzBrpeit7YtB1/r
pTAefv942UcXVX9+MjNVfheGCl18i4ddjwwpPjEslFKDUWeRGj2fnw0JADs=
' b64_noseguy.gif
But you can still save it to jpg. The transparency will be lost and the background will be black.
convert 'inline:data:image/jpeg;base64,
R0lGODlhIAAgAPIEAAAAAB6Q/76+vvXes////wAAAAAAAAAAACH5BAEAAAUALAAA
AAAgACAAAAOBWLrc/jDKCYG1NBcwegeaxHkeGD4j+Z1OWl4Yu6mAYAu1ebpwL/OE
YCDA0YWAQuJqRwsSeEyaRTUwTlxUqjUymmZpmeI3u62Mv+XWmUzBrpeit7YtB1/r
pTAefv942UcXVX9+MjNVfheGCl18i4ddjwwpPjEslFKDUWeRGj2fnw0JADs=
' b64_noseguy.jpg
See https://imagemagick.org/Usage/files/#inline
Sorry, I do not know the equivalent in RMagick
I use webdav protocol. gem 'net_dav' it's old but usefull
when i get file through it i receive it like this \xFF\xD8\xFF\xE0\x00\x10JFIF\x00\x01\x01\x01\x00H\x00H\x00\x00\xFF\...
How to convert to real image .jpg or .png?
you will have to decode this base64 string and then save file, something like
File.open('file_name.jpg', 'wb') do|f|
f.write(Base64.decode64(base_64_encoded_string))
end
Is there anyway to load a yaml file which is stored in the Amazon S3 bucket? I tried giving the url in the YAML.load_file() method but it is not picking up the file. Please give your inputs.
Load your website as string using open-uri, then load that string as YAML.
require "open-uri"
yaml_content = open("http://www.google.com"){|f| f.read}
yaml_data = YAML::load(yaml_content)
yaml_data["content"]
The code below resizes my image. But I am not sure how to write it out to a temp file or blob so I can upload it to s3.
origImage = MiniMagick::Image.open(myPhoto.tempfile.path)
origImage.resize "200x200"
thumbKey = "tiny-#{key}"
obj = bucket.objects[thumbKey].write(:file => origImage.write("tiny.jpg"))
I can upload the original file just fine to s3 with the below command:
obj = bucket.objects[key].write('data')
obj.write(:file => myPhoto.tempfile)
I think I want to create a temp file, read the image file into it and upload that:
thumbFile = Tempfile.new('temp')
thumbFile.write(origImage.read)
obj = bucket.objects[thumbKey].write(:file => thumbFile)
but the origImage class doesn't have a read command.
UPDATE: I was reading the source code and found this out about the write command
# Writes the temporary file out to either a file location (by passing in a String) or by
# passing in a Stream that you can #write(chunk) to repeatedly
#
# #param output_to [IOStream, String] Some kind of stream object that needs to be read or a file path as a String
# #return [IOStream, Boolean] If you pass in a file location [String] then you get a success boolean. If its a stream, you get it back.
# Writes the temporary image that we are using for processing to the output path
And the s3 api docs say you can stream the content using a code block like:
obj.write do |buffer, bytes|
# writing fewer than the requested number of bytes to the buffer
# will cause write to stop yielding to the block
end
How do I change my code so
origImage.write(s3stream here)
http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html
UPDATE 2
This code successfully uploads the thumbnail file to s3. But I would still love to know how to stream it up. It would be much more efficient I think.
#resize image and upload a thumbnail
smallImage = MiniMagick::Image.open(myPhoto.tempfile.path)
smallImage.resize "200x200"
thumbKey = "tiny-#{key}"
newFile = Tempfile.new("tempimage")
smallImage.write(newFile.path)
obj = bucket.objects[thumbKey].write('data')
obj.write(:file => newFile)
smallImage.to_blob ?
below code copy from https://github.com/probablycorey/mini_magick/blob/master/lib/mini_magick.rb
# Gives you raw image data back
# #return [String] binary string
def to_blob
f = File.new #path
f.binmode
f.read
ensure
f.close if f
end
Have you looked into the paperclip gem? The gem offers direct compatibility to s3 and works great.
I just started using resque to do some processing on some very large files in the background, and I'm having trouble figuring out how to pass a file to a resque worker. I use rails to handle the file upload, and rails creates an ActionDispatch::Http::UploadedFile object for each file uploaded from the form.
How do send this file to a resque worker? I tried sending a custom hash of just the pathname of the temporary file and original filename, but I can't reopen the temporary file in the resque worker anymore (just a normal Errno::ENOENT - No such file or directory) because rails seems to delete that temporary file after the request ends.
Http::UploadedFileisn't accessible once the request finishes. You need to write the file somewhere (or use s3 as temp storage). Pass resque the path to the file that you wrote.
I just spent two days trying to do this and finally figured it out. You need to Base64 encode the file so that it can be serialized into json. Then you need to decode it in the worker and create a new
ActionDispatch::Http::UploadedFile
Here's how to encode and pass to resque:
// You only need to encode the actual file, everything else in the
// ActionDispatch::Http::UploadedFile object is just string or a hash of strings
file = params[:file] // Your ActionDispatch::Http::UploadedFile object
file.tempfile.binmode
file.tempfile = Base64.encode64(file.tempfile.read)
Resque.enqueue(QueueWorker, params)
And Here's how to decode and convert back to an object within your worker
class QueueWorker
#queue = :main_queue
def self.perform(params)
file = params['file']
tempfile = Tempfile.new('file')
tempfile.binmode
tempfile.write(Base64.decode64(file['tempfile']))
// Now that the file is decoded you need to build a new
// ActionDispatch::Http::UploadedFile with the decoded tempfile and the other
// attritubes you passed in.
file = ActionDispatch::Http::UploadedFile.new(tempfile: tempfile, filename: file['original_filename'], type: file['content_type'], head: file['headers'])
// This object is now the same as the one in your controller in params[:file]
end
end