I use Shrine gem with Rails 5. I enabled plugins upload_endpoint, versions, processing and recache. I expected to get generated versions in upload endpoint response.
class VideoUploader < Shrine
plugin :processing
plugin :versions
plugin :recache
plugin :upload_endpoint
plugin :upload_endpoint, rack_response: -> (uploaded_file, request) do
# ??? I expected uploaded_file to have thumbnail version here ???
body = { data: uploaded_file.data, url: uploaded_file.url }.to_json
[201, { "Content-Type" => "application/json" }, [body]]
end
process(:recache) do |io, context|
versions = { original: io }
io.download do |original|
screenshot = Tempfile.new(["screenshot", ".jpg"], binmode: true)
movie = FFMPEG::Movie.new(original.path)
movie.screenshot(screenshot.path)
screenshot.open # refresh file descriptors
versions[:thumbnail] = screenshot
end
versions
end
end
Why process callback process(:recache) happens only when saving whole record? And how to make it generate versions right after direct uploading?
The :recache action only happens when you assign a file to a model instance, and after validation succeeded. So the recache plugin is not what you want here.
Whenever Shrine uploads a file, it includes an :action parameter in that upload, and this is what's matched when you register a process block. It's not currently documented, but the upload_endpoint includes action: :upload, so just use process(:upload):
process(:upload) do |io, context|
# ...
end
In your :rack_response block, uploaded_file will now be a hash of uploaded files, so you won't be able to call #data on it. But you can just include them in the hash directly, and they should automatically convert to JSON.
plugin :upload_endpoint, rack_response: -> (uploaded_file, request) do
body = { data: uploaded_file, url: uploaded_file[:original].url }.to_json
[201, { "Content-Type" => "application/json" }, [body]]
end
Related
So here is the thing: currently our files, when user downloads them, have names like 897123uiojdkashdu182uiej.pdf. I need to change that to file-name.pdf.
And logically I go and change paperclip.rb config from this:
Paperclip::Attachment.default_options.update({
path: '/:hash.:extension',
hash_secret: Rails.application.secrets.secret_key_base
})
to this:
Paperclip::Attachment.default_options.update({
path: "/attachment/#{SecureRandom.urlsafe_base64(64)}/:filename",
hash_secret: Rails.application.secrets.secret_key_base
})
which works just fine, filenames are great. However, old files are now unaccessable due to the change in the path. So I came up with the following decision
First I made a rake task which will store the old paths in the database:
namespace :paperclip do
desc "Set old urls for attachments"
task :update_old_urls => :environment do
Asset.find_each do |asset|
if asset.attachment
attachment_url = asset.attachment.try!(:url)
file_url = "https:#{attachment_url}"
puts "Set old url asset attachment #{asset.id} - #{file_url}"
asset.update(old_url: file_url)
else
puts "No attachment found in asset #{asset.id}"
end
end
end
end
Now the asset.old_url stores the current url of the file. Then I go and change the config, making the file unaccessable.
Now it's time for the new rake task:
require 'uri'
require 'open-uri'
namespace :paperclip do
desc "Recreate attachments and save them to new destination"
task :move_attachments => :environment do
Asset.find_each do |asset|
unless asset.old_url.blank?
url = asset.old_url
filename = File.basename(asset.attachment.path)
file = File.new("#{Rails.root}/tmp/#{filename}", "wb")
file.write(open(url).read)
if File.exists? file
puts "Re-saving asset attachment #{asset.id} - #{filename}"
asset.attachment = file
asset.save
# if there are multiple styles, you want to recreate them :
asset.attachment.reprocess!
file.close
else
puts "Missing file attachment #{asset.id} - #{filename}"
end
File.delete(file)
end
end
end
end
But my plan didn't work at all, I didn't get access to the files, and the asset.url still isn't equal to asset.old_url.
Would appreciate help very much!
With S3, you can set the "filename upon saving" as a header. Specifically, the user will get to an url https://foo.bar.com/mangled/path/some/weird/hash/whatever?options and when the browser will offer to save, you can control the filename (not the url).
The trick to that relies on the browser reading the Content-Disposition header from the response, if it reads Content-Disposition: attachment; filename="filename.jpg" it will save (or ask the user to save as) filename.jpg, independently on the original URL.
You can force S3 to add this header by adding one more parameter to the URL or by setting a metadata on the file.
The former can be done by passing it to the url method:
has_attached_file :attachment,
s3_url_options: ->(instance) {
{response_content_disposition: "attachment; filename=\"#{instance.filename}\""}
}
Check https://github.com/thoughtbot/paperclip/blob/v6.1.0/lib/paperclip/storage/s3.rb#L221-L225 for the relevant source code.
The latter can be done in bulk via paperclip (and you should also configure it to do it on new uploads). It will also take a long time!!
Asset.find_each do |asset|
next unless asset.attachment
s3_object = asset.attachment.s3_object
s3_object.copy_to(
s3_object,
metadata_directive: 'REPLACE',
content_disposition: "attachment; filename=\"#{asset.filename}\")"
)
end
# for new uploads
has_attached_file :attachment,
s3_headers: ->(att) {
{content_disposition: "attachment; filename=\"#{att.model.filename}\""}
}
I recently upgraded from Carrierwave 1.3 to 2.1, and I got a couple of specs failing due to the invalid mime type.
I store on the database, CSV Uploads, and I validate on the model if the mime type is text/csv.
validates :file, presence: true, file_content_type: {
allow: [
'text/csv',
'application/vnd.ms-excel',
'application/vnd.ms-office',
'application/octet-stream',
'text/comma-separated-values'
]
}
and on the spec, I created a fixture
let(:file) { fixture_file_upload('files/fixture.csv', 'text/csv') }
when I debug,
#file=
#<CarrierWave::SanitizedFile:0x00007f8c731791f0
#content=nil,
#content_type="invalid/invalid",
#file="/Users/tiagovieira/code/work/tpc/public/uploads/csv_file_upload/file/1/1605532759-308056149220914-0040-7268/fixture.csv",
#original_filename="fixture.csv">,
#filename="fixture.csv",
#identifier="fixture.csv",
Is this related to the fact that carrierwave stopped using mime-types gem as a dependency?
Seems the problem is found.
In previous carrierwave version "CarrierWave::SanitizedFile" content_type was calculated by extension
https://github.com/carrierwaveuploader/carrierwave/blob/1.x-stable/lib/carrierwave/sanitized_file.rb
def content_type
return #content_type if #content_type
if #file.respond_to?(:content_type) and #file.content_type
#content_type = #file.content_type.to_s.chomp
elsif path
#content_type = ::MIME::Types.type_for(path).first.to_s
end
end
And now it has more complicated way. It uses algorithms to recognize the file type by what data this file contains.
https://github.com/carrierwaveuploader/carrierwave/blob/master/lib/carrierwave/sanitized_file.rb
def content_type
#content_type ||=
existing_content_type ||
mime_magic_content_type ||
mini_mime_content_type
end
And i have "invalid/invalid" content-type after mime_magic_content_type which seems could not fetch file type using "MimeMagic.by_magic".
PS i see that "plain/text" content_type is returned for usual css file.
https://github.com/minad/mimemagic/blob/master/lib/mimemagic/tables.rb#L1506
use Rack::Test::UploadedFile
assign the mounted model a Rack::Test::UploadedFile object
assume your model is:
class User < ApplicationRecord
mount_uploader :file, FileUploader
end
to test the uploader you can use something like:
user.file = Rack::Test::UploadedFile.new(File.open('test_file.csv'), "text/csv")
user.save
whitelist by Carrierwave's content_type_whitelist or extension_whitelist
class FileUploader < CarrierWave::Uploader::Base
private
def extension_whitelist
%w(csv xlsx xls)
end
def content_type_whitelist
[
'text/csv',
'application/vnd.ms-excel',
'application/vnd.ms-office',
'application/octet-stream',
'text/comma-separated-values'
]
end
end
also check:
https://til.codes/testing-carrierwave-file-uploads-with-rspec-and-factorygirl/
I am working on an app that require to upload videos. I added Shrine and s3 storage.
Till here everything is working. Now I need to transcode the videos and I added the following code to the video_uploader file
class VideoUploader < Shrine
plugin :processing
plugin :versions
process(:store) do |io|
transcoder = Aws::ElasticTranscoder::Client.new(
access_key_id: ENV['AWS_ACCESS_KEY_ID'],
secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
region: 'us-east-1',
)
pipeline = transcoder.create_pipeline(options = {
:name => "name",
:input_bucket => "bucket",
:output_bucket => "bucket",
:role => "arn:aws:iam::XXXXX:role/Elastic_Transcoder_Default_Role",
})
PIPELINE_ID = pipeline[:pipeline][:id]
transcode_hd = transcoder.create_job({
:pipeline_id=>PIPELINE_ID,
:input=> {
:key=> "cache/"+io.id,
:frame_rate=> "auto",
:resolution => "auto",
:aspect_ratio => "auto",
:container => 'auto'
},
:outputs=>[{
:key=>"store/"+io.id,
:preset_id=>"1351620000001-000010",
}]
})
end
end
The transcoding is working and basically is transcoding the new file uploaded to cache folder and put in the store folder with the same name.
The issue now is to attach this file to the record in the database. As of now the record is updated with a different name it creates a new file in the store folder of 0mb.
How can I attach the results of processing into Shrine's uploaded file for storage?
The process(:store) block expects you to return a file for Shrine to upload to permanent storage, so this flow won't work Amazon Elastic Transcoder, because Amazon Elastic Transcoder is now the one that will upload the cached file to permanent storage.
You can delay the transcoding request into a background job, poll the transcoding job every N seconds, and create a Shrine::UploadedFile from the results and update the record. Something like the following should work:
# superclass for all uploaders that use Amazon Elastic Transcoder
class TranscoderUploader < Shrine
plugin :backgrounding
Attacher.promote { |data| TranscodeJob.perform_async(data) }
end
class VideoUploader < TranscoderUploader
plugin :versions
end
class TranscodeJob
include Sidekiq::Worker
def perform(data)
attacher = TranscoderUploader::Attacher.load(data)
cached_file = attacher.get #=> #<Shrine::UploadedFile>
# create transcoding job, use `cached_file.id`
transcoder.wait_until(:job_complete, id: job.id)
response = transcoder.read_job(id: job.id)
output = response.output
versions = {
video: attacher.shrine_class::UploadedFile.new(
"id" => cached_file.id,
"storage" => "store",
"metadata" => {
"width" => output.width,
"height" => output.height,
# ...
}
),
...
}
attacher.swap(versions)
end
end
If you'll by any chance be interested in making a Shrine plugin for Amazon Elastic Transcoder, take a look at shrine-transloadit which provides integration for Transloadit, which uses practically the same flow as the Amazon Elastic Transcoder, and it works with webhooks rather than polling for the response.
I'm trying to determine whether a remote url is an image. Most url's have .jpg, .png etc...but some images, like google images, have no extension...i.e.
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSbK2NSUILnFozlX-oCWQ0r2PS2gHPPF7c8XaxGuJFGe83KGJkhFtlLXU_u
I've tried using FastImage to determine whether a url is an image. It works when any URL is fed into it...
How could I ensure that remote urls use FastImage and uploaded files use the whitelist? Here is what have in my uploader. Avatar_remote_url isn't recognized...what do I do in the uploader to just test remote urls and not regular files.
def extension_white_list
if defined? avatar_remote_url && !FastImage.type(CGI::unescape(avatar_remote_url)).nil?
# ok to process
else # regular uploaded file should detect the following extensions
%w(jpg jpeg gif png)
end
end
if all you have to work with is a url like that you can send a HEAD request to the server to obtain the content type for the image. From that you can obtain the extension
require 'net/http'
require 'mime/types'
def get_extension(url)
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true if uri.scheme == 'https'
request = Net::HTTP::Head.new(uri.request_uri)
response = http.request(request)
content_type = response['Content-Type']
MIME::Types[content_type].first.extensions.first
end
I'm working with the code you provided and some of the code provided in the CarrierWave Wiki for validating remote URLs.
You can create a new validator in lib/remote_image_validator.rb.
require 'fastimage'
class RemoteImageValidator < ActiveModel::EachValidator
def validate_each(object, attribute, value)
raise(ArgumentError, "A regular expression must be supplied as the :format option of the options hash") unless options[:format].nil? || options[:format].is_a?(Regexp)
configuration = { :message => "is invalid or not responding", :format => URI::regexp(%w(http https)) }
configuration.update(options)
if value =~ configuration[:format]
begin
if FastImage.type(CGI::unescape(avatar_remote_url))
true
else
object.errors.add(attribute, configuration[:message]) and false
end
rescue
object.errors.add(attribute, configuration[:message]) and false
end
else
object.errors.add(attribute, configuration[:message]) and false
end
end
end
Then in your model
class User < ActiveRecord::Base
validates :avatar_remote_url,
:remote_image => {
:format => /(^$)|(^(http|https):\/\/[a-z0-9]+([\-\.]{1}[a-z0-9]+)*\.[a-z]{2,5}(([0-9]{1,5})?\/.*)?$)/ix,
:unless => remote_avatar_url.blank?
}
end
I was having a similar issue where creating the different versions from the original was failing because ImageMagick could not figure out the correct encoder to use due to the missing extension. Here is a monkey-patch I applied in Rails that fixed my problem:
module CarrierWave
module Uploader
module Download
class RemoteFile
def original_filename
value = File.basename(file.base_uri.path)
mime_type = Mime::Type.lookup(file.content_type)
unless File.extname(value).present? || mime_type.blank?
value = "#{value}.#{mime_type.symbol}"
end
value
end
end
end
end
end
I believe this will address the problem you are having as well since it ensures the existence of a file extension when the content type is set appropriately.
UPDATE:
The master branch of carrierwave has a different solution to this problem that uses the Content-Disposition header to figure out the filename. Here is the relevant pull request on github.
Does anyone know how to post a JSON to a Rails server with a file attached? Would the content be base64 encoded? Multipart? I honestly have no idea and havent really found anything here to help. Idea is to have a client posting a JSON to a rails API with the file attached, as well as having the Rails (with paperclip would be perfect) getting the JSON and saving the file properly. Thanks in advance
Here is how I solved this problem. First I created a rake task to upload the file within the json content:
desc "Tests JSON uploads with attached files on multipart formats"
task :picture => :environment do
file = File.open(Rails.root.join('lib', 'assets', 'photo.jpg'))
data = {title: "Something", description: "Else", file_content: Base64.encode64(file.read)}.to_json
req = Net::HTTP::Post.new("/users.json", {"Content-Type" => "application/json", 'Accept' => '*/*'})
req.body = data
response = Net::HTTP.new("localhost", "3000").start {|http| http.request(req) }
puts response.body
end
And then got this on the controller/model of my rails app, like this:
params[:user] = JSON.parse(request.body.read)
...
class User < ActiveRecord::Base
...
has_attached_file :picture, formats: {medium: "300x300#", thumb: "100#100"}
def file_content=(c)
filename = "#{Time.now.to_f.to_s.gsub('.', '_')}.jpg"
File.open("/tmp/#{filename}", 'wb') {|f| f.write(Base64.decode64(c).strip) }
self.picture = File.open("/tmp/#{filename}", 'r')
end
end
JSON is a data serializing format. There is no standard pattern for uploading data or files as data in the serialized object. JSON has expectations that the data fields will be basic objects so you probably want to use Base64 encoding of the file to turn it into a string.
You are free to define your structure however you want, and processing it is your responsibility.