Rails 5.1 -
Ruby 2.3.4 -
AWS S3 - Carrierwave & Minimagick
I would like to import a CSV file to my Rails app and also upload Images with each record.
The images are URL's from a current AWS S3 Bucket. I am able to upload from my local drive but only in development.
def self.import(file)
CSV.foreach(file.path, headers: true, header_converters: :symbol) do |row|
Model.find_or_create_by(name: row[:name]) do |m|
m.name = row[:name]
m.description = row[:description]
m.image = URI.parse(row[:image])
m.summary = row[:summary]
end
end
end
The CSV is simply
name,descripton,image,summary
Test,Test-Description,https://s3.amazonaws.com/main/production/model/7/image/image-formated.jpg,Test-Summary
Though whenever I try and upload on my live server. The images are not processed. How can I reference and upload the images correctly?
I can't load the URL that you have but it should work with something like:
m.image = URI.parse(row[:image]).open
This will return a tempfile which Rails will associate to the model.
Background:
URI.parse() returns URI::HTTPS the url encoded URI for example:
URI.parse('https://google.com/search?q=test this')
# => #<URI::HTTPS https://google.com/search?q=test%20this> # where %20 is a url encoded space
So assuming your model expects an image file, you want to open and save the file rather than the URI. If this does not solve this, please share the config for your model
Related
I'm integrating with a third party ID verification provider. Once the ID verification checks have run a report is generated, which I can access with a get request.
The response is binary text (screenshot attached), which I want to save to a PDF file.
Function:
def generate_pdf
resources = "applicants/61f84499b7b92f00014d5c6d/summary/report?report=applicantReport"
response = RestClient.get(request_env_url(resources), signed_header(resources, nil, 'GET', 'application/pdf'))
puts response
end
How do I take the binary response, create a new file and add the binary into that file in a friendly and readable format?
If you can access a filesystem, you might just want to write that data to a file:
File.open("thing.pdf", "wb") { |f| f.write response }
How do I upload an S3 URL image properly for Google Vision?
I am attempting to send an image (saved at an AWS S3 URL) to Google Vision with Base64 encoding per the 2nd option in the documentation listed below:
Images sent to the Google Cloud Vision API can be supplied in two ways:
Using Google Cloud Storage URIs of the form gs://bucketname/path/to/image_filename
As image data sent within the JSON request. Because image data must be supplied as ASCII text, all image data should be escaped using base64 encoding.
I am using the Google-Cloud-Vision Gem.
I have tried this previous answer about Base64 encoding, with a minor modification:
require 'google/cloud/vision'
require 'base64'
require 'googleauth'
require 'open-uri'
encoded_image = Base64.strict_encode64(open(image_url, &:read))
#vision = Google::Cloud::Vision.new
image = #vision.image(encoded_image)
annotation = #vision.annotate(image, labels: true, text: true)
I have tried images at AWS URLs and images at other urls.
Every time I get this error from the Google-Cloud-Vision gem:
ArgumentError: Unable to convert (my_base_64_encoded_image) to an image
Update - successfully encoded and decoded image in ruby only
I have confirmed that this code: encoded_image = Base64.strict_encode64(open(image_url, &:read)) works via the following:
# Using a random image from the interwebs
image_url = "https://storage.googleapis.com/gweb-uniblog-publish-prod/static/blog/images/google-200x200.7714256da16f.png"
encoded_image = Base64.strict_encode64(open(image_url, &:read))
### now try to decode the encoded_image
File.open("my-new-image.jpg", "wb") do |file|
file.write(Base64.strict_decode64(encoded_image))
end
### great success
So what's google's problem with this? I am properly encoded.
If you are going to use Google-Cloud-Vision gem, you need to follow the gem documentation (uses non encoded images), maybe he did that under the hood..
Based on the documentation of the gem google-cloud-vision , you can convert your image like the code bellow
open image_url do |img|
#vision = Google::Cloud::Vision.new
image = #vision.image(img)
# you can also use the class method from_io
# image = Google::Cloud::Vision::Image.from_io(img, #vision)
annotation = #vision.annotate(image, labels: true, text: true)
end
I'm currently storing files privately on S3. In my Rails app, in the attachment.rb model I can obtain a public URL for the private file like so:
def cdn_url ( style='original' )
attachment.s3_object(style).url_for( :read, secure: true, response_content_type: self.meta['file_content_type'], expires: 1.hour ).to_s
end
The problem is this is providing a URL to S3 and rewriting the URL to use my Cloudfront origin url is erroring with:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
How can I get a public URL asset like below but serve the asset via Cloudfront?
First Way (Easy)
Just use aws_cf_signer gem. Put it in you bundler.
WIth this you can do something like
def cdn_url (options = {})
style = options[:style] || 'original'
cloudfront_domain = options[:cloudfront_domain] || 'example.cloudfront.net'
cloudfront_pem_key_path = options[:cloudfront_pem_key_path]
cloudfront_key_paid_id = options[:cloundfrount_key_paid_id]
path = attachment.path(style) #path of the file
# you can get this values from your aws a/c , most probably by going int
# https://console.aws.amazon.com/iam/home?#security_credential
signer = AwsCfSigner.new(cloudfront_pem_key_path, cloudfront_key_paid_id)
# this configuration may vary.
# visit https://github.com/dylanvaughn/aws_cf_signer
# and check all available settings/options
url = signer.sign(path, :ending => Time.now + 3600)
cloudfront_domain + url
end
With this you can access the url with something like this
cdn_url(cloudfront_pem_key_path: '/users/downloads/pri.pem' , cloudfront_key_paid_id: '33243424XXX')
Second way
# A simple function to return a signed, expiring url for Amazon Cloudfront.
# This will require openssl, digest/sha1, base64 and maybe other libraries.
module CloudFront
def get_signed_expiring_url(domain,path, expires_in, private_key_filename, key_pair_id)
# AWS works on UTC, so make sure you are not using local time
expires = (Time.now.getutc + expires_in).to_i.to_s
private_key = OpenSSL::PKey::RSA.new(File.read(private_key_filename))
# path should be your S3 path without a leading slash and without a file extension.
# e.g. files/private/52
policy = %Q[{"Statement":[{"Resource":"#{path}","Condition":{"DateLessThan":{"AWS:EpochTime":#{expires}}}}]}]
signature = Base64.strict_encode64(private_key.sign(OpenSSL::Digest::SHA1.new, policy))
# I'm not sure exactly why this is required, but it's in Amazon's perl script and seems necessary
# Different base64 implementations maybe?
signature.tr!("+=/", "-_~")
"#{domain}#{path}?Expires=#{expires}&Signature=#{signature}&Key-Pair-Id=#{key_pair_id}"
end
end
With this you can do something like
def cdn_url ( style='original',cloudfront_pem_key_path,key_pair_id)
path = attachment.path(style) #path of the file
# you can get this values from your aws a/c , most probably by going int
CloudFront.get_signed_expiring_url 'example.cloudfront.net', path, 45.seconds ,'/users/downloads/pri.pem', 'as12XXXXX')
end
Give a try, may be it will work. Be sure to properly set properly bucket access policy. check this out if you are seeing accessDenied error http://www.jppinto.com/2011/12/access-denied-to-file-amazon-s3-bucket/
Use the aws sdk gem.
See the API Documentation
Details about generating a presigned URL for an operation on the object
Provide the access-key-id and secret-access-key:-
S3 = AWS::S3.new(
:access_key_id => 'access_key_id',
:secret_access_key => 'secret_access_key')
In controller put these lines:--
bucket = S3.buckets['bucket_name']
s3_obj = bucket.objects["Path-to-file"]
return s3_obj.url_for(:read, :expires => 60*60).to_s
This link will expires in 1 Hour. After that the link will be not accessible.
I have a problem with S3 and CarrierWave:
I have a pseudo-form that uploads data and files, I wrote "pseudo" because it's an ajax form so data is sent with jquery to rails with a POST request. Files cannot be uploaded in this way...so I have a popup windows that upload files to rails, I save in the session the reference to the uploaded files and when the ajax request uploads the rest of the form, I link the files uploaded to the rest of the data.
With storage :file it works without any problems, when i receive the file I do:
uploader = ImgObjUploader.new
uploader.store!(params[:image_form][:image])
session["image"] = uploader.url
and then when I get the rest of the data:
if (session[:image] != nil) then
obj.image = File.open(session[:image])
end
And my model is:
mount_uploader :image, ImgObjUploader
This code work without any problems, for amazon s3 I switched to:
uploader = ImgObjUploader.new
uploader.retrieve_from_store!(session[:image])
puts uploader
#obj.image = uploader
obj.image = uploader.url
but it doesn't work...I didn't receive an error but I don't have the image saved inside obj object. Puts uploader prints the url of amazon S3.
Anyone can help me?
Thank You.
In my app, I have a requirement that is stumping me.
I have a file stored in S3, and when a user clicks on a link in my app, I log in the DB they've clicked the link, decrease their 'download credit' allowance by one and then I want to prompt the file for download.
I don't simply want to redirect the user to the file because it's stored in S3 and I don't want them to have the link of the source file (so that I can maintain integrity and access)
It looks like send_file() wont work with a remote source file, anyone recommend a gem or suitable code which will do this?
You would need to stream the file content to the user while reading it from the S3 bucket/object.
If you use the AWS::S3 library something like this may work:
send_file_headers!( :length=>S3Object.about(<s3 object>, <s3 bucket>)["content-length"], :filename=><the filename> )
render :status => 200, :text => Proc.new { |response, output|
S3Object.stream(<s3 object>, <s3 bucket>) do |chunk|
output.write chunk
end
}
This code is mostly copied form the send_file code which by itself works only for local files or file-like objects
N.B. I would anyhow advise against serving the file from the rails process itself. If possible/acceptable for your use case I'd use an authenticated GET to serve the private data from the bucket.
Using an authenticated GET you can keep the bucket and its objects private, while allowing temporary permission to read a specific object content by crafting a URL that includes an authentication signature token. The user is simply redirected to the authenticated URL, and the token can be made valid for just a few minutes.
Using the above mentioned AWS::S3 you can obtain an authenticated GET url in this way:
time_of_exipry = Time.now + 2.minutes
S3Object.url_for(<s3 object>, <s3 bucket>,
:expires => time_of_exipry)
Full image download method using temp file (tested rails 3.2):
def download
#image = Image.find(params[:image_id])
open(#image.url) {|img|
tmpfile = Tempfile.new("download.jpg")
File.open(tmpfile.path, 'wb') do |f|
f.write img.read
end
send_file tmpfile.path, :filename => "great-image.jpg"
}
end
You can read the file from S3 and write it locally to a non-public directory, then use X-Sendfile (apache) or X-Accel-Redirect (nginx) to serve the content.
For nginx you would include something like the following in your config:
location /private {
internal;
alias /path/to/private/directory/;
}
Then in your rails controller, you do the following:
response.headers['Content-Type'] = your_content_type
response.headers['Content-Disposition'] = "attachment; filename=#{your_file_name}"
response.headers['Cache-Control'] = "private"
response.headers['X-Accel-Redirect'] = path_to_your_file
render :nothing=>true
A good writeup of the process is here