RESTful file uploads with CarrierWave - ruby-on-rails

I'm trying to build an API backend for file uploads. I want to be able to upload files with a POST request that has a Base64-encoded string of the file. The server should decode the string, and save the file using CarrierWave. Here's what I have so far:
photo.rb:
class Photo
include Mongoid::Document
include Mongoid::Timestamps
mount_uploader :image_file, ImageUploader
end
image_uploader.rb:
class ImageUploader < CarrierWave::Uploader::Base
storage :file
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
end
Rails console:
(summary)
ruby-1.8.7-p334 :001 > img = File.open("../image.png") {|i| i.read}
=> "\377���JFIF\000\001\002\001\000H\000H\000\000\377�Photoshop 3.0\0008BIM\003...
ruby-1.8.7-p334 :003 > encoded_img = Base64.encode64 img
=> 3af8A\nmLpplt5U8q+a7G2...
ruby-1.8.7-p334 :005 > p = Photo.new
=> #<Photo _id: 4e21b9a31d41c817b9000001, created_at: nil, updated_at: nil, _type: nil, user_id: nil, image_file_filename: nil>
ruby-1.8.7-p334 :006 > p.user_id = 1
=> 1
ruby-1.8.7-p334 :007 > p.image_file = Base64.decode64 encoded_img
\255��=\254\200�7u\226���\230�-zh�wT\253%����\036ʉs\232Is�M\215��˿6\247\256\177...
ruby-1.8.7-p334 :008 > p.save
=> true
ruby-1.8.7-p334 :009 > p.image_file.url
=> nil
full
The problem appears to be related to the process of converting a Base64-decoded string to a file. CarrierWave seems to expect a File object, and instead I'm giving it a String. So how do I convert that String to a File object. I'd like this conversion not to save anything to the file system, simply create the object and let CarrierWave do the rest.

CarrierWave also accepts a StringIO, but it expects a original_filename method, since it needs it for figuring out the file name and doing the extension check. How you do it changed between Rails 2 and 3, here's both methods:
Rails 2
io = StringIO.new(Base64.decode64(encoded_img))
io.original_filename = "foobar.png"
p.image_file = io
p.save
In Rails 3, you need to make a new class and then manually add original_filename back
class FilelessIO < StringIO
attr_accessor :original_filename
end
io = FilelessIO.new(Base64.decode64(encoded_img))
io.original_filename = "foobar.png"
p.image_file = io
p.save

You don't have to monkeypatch StringIO or put any of this in your model. You can override the cache!() method in your uploader definition. Or you can take it a step further and make a module for yourself to include. My file is a serialized string coming from a json document. The object passed in looks like this { :filename => 'something.jpg', :filedata => base64 string }.
Here is my module:
module CarrierWave
module Uploader
module BackboneLink
def cache!(new_file=sanitized_file)
#if new_file isn't what we expect just jump to super
if new_file.kind_of? Hash and new_file.has_key? :filedata
#this is from a browser, so it has all that 'data:..' junk to cut off.
content_type, encoding, string = new_file[:filedata].split(/[:;,]/)[1..3]
sanitized = CarrierWave::SanitizedFile.new(
:tempfile => StringIO.new(Base64.decode64(string)),
:filename => new_file[:filename],
:content_type => content_type
)
super sanitized
else
super
end
end
end
end
end
And then I can include it in an uploader.
uploaders/some_uploader.rb:
class SomeUploader < CarrierWave::Uploader::Base
include CarrierWave::Uploader::BackboneLink

class AppSpecificStringIO < StringIO
attr_accessor :filepath
def initialize(*args)
super(*args[1..-1])
#filepath = args[0]
end
def original_filename
File.basename(filepath)
end
end
also see the carrierwave wiki https://github.com/carrierwaveuploader/carrierwave/wiki/How-to%3A-Upload-from-a-string-in-Rails-3

Related

Carrierwave sets mime type to invalid/invalid

I recently upgraded from Carrierwave 1.3 to 2.1, and I got a couple of specs failing due to the invalid mime type.
I store on the database, CSV Uploads, and I validate on the model if the mime type is text/csv.
validates :file, presence: true, file_content_type: {
allow: [
'text/csv',
'application/vnd.ms-excel',
'application/vnd.ms-office',
'application/octet-stream',
'text/comma-separated-values'
]
}
and on the spec, I created a fixture
let(:file) { fixture_file_upload('files/fixture.csv', 'text/csv') }
when I debug,
#file=
#<CarrierWave::SanitizedFile:0x00007f8c731791f0
#content=nil,
#content_type="invalid/invalid",
#file="/Users/tiagovieira/code/work/tpc/public/uploads/csv_file_upload/file/1/1605532759-308056149220914-0040-7268/fixture.csv",
#original_filename="fixture.csv">,
#filename="fixture.csv",
#identifier="fixture.csv",
Is this related to the fact that carrierwave stopped using mime-types gem as a dependency?
Seems the problem is found.
In previous carrierwave version "CarrierWave::SanitizedFile" content_type was calculated by extension
https://github.com/carrierwaveuploader/carrierwave/blob/1.x-stable/lib/carrierwave/sanitized_file.rb
def content_type
return #content_type if #content_type
if #file.respond_to?(:content_type) and #file.content_type
#content_type = #file.content_type.to_s.chomp
elsif path
#content_type = ::MIME::Types.type_for(path).first.to_s
end
end
And now it has more complicated way. It uses algorithms to recognize the file type by what data this file contains.
https://github.com/carrierwaveuploader/carrierwave/blob/master/lib/carrierwave/sanitized_file.rb
def content_type
#content_type ||=
existing_content_type ||
mime_magic_content_type ||
mini_mime_content_type
end
And i have "invalid/invalid" content-type after mime_magic_content_type which seems could not fetch file type using "MimeMagic.by_magic".
PS i see that "plain/text" content_type is returned for usual css file.
https://github.com/minad/mimemagic/blob/master/lib/mimemagic/tables.rb#L1506
use Rack::Test::UploadedFile
assign the mounted model a Rack::Test::UploadedFile object
assume your model is:
class User < ApplicationRecord
mount_uploader :file, FileUploader
end
to test the uploader you can use something like:
user.file = Rack::Test::UploadedFile.new(File.open('test_file.csv'), "text/csv")
user.save
whitelist by Carrierwave's content_type_whitelist or extension_whitelist
class FileUploader < CarrierWave::Uploader::Base
private
def extension_whitelist
%w(csv xlsx xls)
end
def content_type_whitelist
[
'text/csv',
'application/vnd.ms-excel',
'application/vnd.ms-office',
'application/octet-stream',
'text/comma-separated-values'
]
end
end
also check:
https://til.codes/testing-carrierwave-file-uploads-with-rspec-and-factorygirl/

Issues with Globalize and CarrierWave uploads in Rails 4.2 app

I have issues with translated files in my Rails 4.2 app.
Background
Here are the gem versions I'm using:
gem "rails", "4.2.1"
gem "carrierwave" # 0.10.0
gem "globalize" # 5.0.1
And my model:
class Download < ActiveRecord::Base
belongs_to :download_type
has_and_belongs_to_many :products
translates :title, :part_number, :file
mount_uploader :file, DownloadFileUploader
validates :title, presence: true
def to_param
"#{id}-#{title.parameterize}"
end
end
The Issues
In my view, I want to list a Download and all of the current translations for that download, but all I get is the current locale data for each translation. In the Rails console:
> I18n.locale
=> :en
> download = Download.find(481)
=> #<Download id: 481, title: "SmartSensor HD Quick-reference Guide (User)", part_number: "WX-500-0171", download_type_id: 3, created_at: "2015-01-16 22:49:13", updated_at: "2015-04-20 16:59:25", file: "smartsensor_hd_user_quick-reference_guide-20150116...", download_updated_at: nil>
> download.translations.count
=> 8
> download.translated_locales
=> [:de, :en, :es, :fr, :it, :pt, :ru, :"zh-CN"]
> download.file.class
=> DownloadFileUploader
> download.file.url
=> "/uploads/download/file/481/smartsensor_hd_user_quick-reference_guide-20150116154913-en.pdf"
> download.title
=> "SmartSensor HD Quick-reference Guide (User)"
> download.part_number
=> "WX-500-0171"
And when the locale changes:
> I18n.locale = :de
=> :de
> download.file.class
=> DownloadFileUploader
> download.file.url
=> "/uploads/download/file/481/smartsensor_hd_user_quick-reference_guide-20150116154913-en.pdf"
> download.title
=> "SmartSensor HD Kurzanleitung"
> download.part_number
=> "WX-502-0006"
If I try and access the translation directly:
> I18n.locale = :de
=> :de
> download.translation.file.class
=> String
If I change how the uploader is mounted in my model:
Translation.mount_uploader :file, DownloadFileUploader
Existing translations list correctly—including the file, but I can no longer upload files. What gets stored in the database is this:
/uploads/download/translation/file/401/%23%3CActionDispatch%3A%3AHttp%3A%3AUploadedFile%3A0x007f9c12e6fe00%3E
Notice that it inserts /translation into the path, which I can fix in the uploader, but the filename isn't actually a file.
If I move translates :title, :part_number, :file below mount_uploader :file, DownloadFileUploader in my model, Globalize overrides the mounted uploader and that column is returned as class String when accessing it.
The Cry for Help
Help! 😮
I wrote this simple gem https://github.com/dalpo/carrierwave_globalize. which it should allow to use Globalize and Carrierwave together.
You have to extend your model with the CarrierwaveGlobalize module and use the mount_translated_uploader class method to mount your carrierwave uploader instead of the mount_uploader mehtod.
Follow the instructions in the readme for more info.
Old question, but... do not use Globalize with CarrierWave on the same attribute, they both make overrides for the default behavior.
You can do this in your model:
class Download < ActiveRecord::Base
belongs_to :download_type
has_and_belongs_to_many :products
translates :title, :part_number
mount_uploader :file, DownloadFileUploader
validates :title, presence: true
def to_param
"#{id}-#{title.parameterize}"
end
end
And then override file storing for your uploader in system to use I18n.locale:
def store_dir
"uploads/#{I18n.locale}/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end

Cloudinary URLs break with Mongoid 3?

I've using the Cloudinary gem, which up until now has worked fine as an embedded Photo model in Mongoid:
class PhotoUploader < CarrierWave::Uploader::Base
include Cloudinary::CarrierWave
def public_id
return model.id
end
end
class Photo
include Mongoid::Document
include Mongoid::Timestamps
embedded_in :place#, :inverse_of => :photos, :autosave => true
mount_uploader :image, PhotoUploader
#...
end
However, after upgrading to Mongoid 3, getting the image URL no longer works. Consider this Photo model:
1.9.3p194 :019 > p
=> #<Photo _id: 507bc3c82a450b14bd00e00a, _type: nil, created_at: 2012-10-15 08:05:28 UTC, updated_at: 2012-10-15 08:05:28 UTC, image_filename: nil, caption: nil, original_url: "http://www.reactionny.com//images/assets/101755_316529.JPG", image: "v1350288333/507bc3c82a450b14bd00e00a.jpg">
1.9.3p194 :020 > p.image
=> /assets/fallback/default.png
1.9.3p194 :021 > p.image_url
=> "/assets/fallback/default.png"
1.9.3p194 :022 > p['image']
=> "v1350288333/507bc3c82a450b14bd00e00a.jpg"
Why is it no longer returning the fully-qualified Cloudinary URL, in the form http://res.cloudinary.com/XXXXXX/image/upload/v1350288842/507bc5ca2a450b14bd00e896.jpg?
It seems that carrierwave-mongoid changed the name of the field in which the image is saved from yyy to yyy_filename.
In your model inspection printing you can see that image_filename is nil.
The integration with the Cloudinary GEM and CarrierWave seems to be working fine, but you need to migrate your model.
The relevant field used to be called 'image' and it's now called 'image_filename'. You should probably update your model to use the new field name and copy all values.
Another possible solution would be to specifically set the attribute name using mount_on:
mount_uploader :image, PhotoUploader, mount_on: :image

Rails to_json uses different DATE_FORMATS that .to_s

With the following in my rails_defaults.rb:
Date::DATE_FORMATS[:default] = '%m/%d/%Y'
Time::DATE_FORMATS[:default]= '%m/%d/%Y %H:%M:%S'
Why do the following results differ:
ruby-1.9.2-p180 :005 > MyModel.find(2).to_json(:only => :start_date)
=> "{\"start_date\":\"2012-02-03\"}"
ruby-1.9.2-p180 :006 > MyModel.find(2).start_date.to_s
=> "02/03/2012"
And more importantly, how do I get to_json to use %m/%d/%Y?
Because the standard JSON format for a date is %Y-%m-%d and there's no way to change it unless you override Date#as_json (don't do so or your application will start misbehaving).
See https://github.com/rails/rails/blob/master/activesupport/lib/active_support/json/encoding.rb#L265-273
class Date
def as_json(options = nil) #:nodoc:
if ActiveSupport.use_standard_json_time_format
strftime("%Y-%m-%d")
else
strftime("%Y/%m/%d")
end
end
end

Storing image using open URI and paperclip having size less than 10kb

I want to import some icons from my old site. The size of those icons is less than 10kb. So when I am trying to import the icons its returning stringio.txt file.
require "open-uri"
class Category < ActiveRecord::Base
has_attached_file :icon, :path => ":rails_root/public/:attachment/:id/:style/:basename.:extension"
def icon_from_url(url)
self.icon = open(url)
end
end
In rake task.
category = Category.new
category.icon_from_url "https://xyz.com/images/dog.png"
category.save
Try:
def icon_from_url(url)
extname = File.extname(url)
basename = File.basename(url, extname)
file = Tempfile.new([basename, extname])
file.binmode
open(URI.parse(url)) do |data|
file.write data.read
end
file.rewind
self.icon = file
end
To override the default filename of a "fake file upload" in Paperclip (stringio.txt on small files or an almost random temporary name on larger files) you have 2 main possibilities:
Define an original_filename on the IO:
def icon_from_url(url)
io = open(url)
io.original_filename = "foo.png"
self.icon = io
end
You can also get the filename from the URI:
io.original_filename = File.basename(URI.parse(url).path)
Or replace :basename in your :path:
has_attached_file :icon, :path => ":rails_root/public/:attachment/:id/:style/foo.png", :url => "/:attachment/:id/:style/foo.png"
Remember to alway change the :url when you change the :path, otherwise the icon.url method will be wrong.
You can also define you own custom interpolations (e.g. :rails_root/public/:whatever).
You are almost there I think, try opening parsed uri, not the string.
require "open-uri"
class Category < ActiveRecord::Base
has_attached_file :icon, :path =>:rails_root/public/:attachment/:id/:style/:basename.:extension"
def icon_from_url(url)
self.icon = open(URI.parse(url))
end
end
Of course this doesn't handle errors
You can also disable OpenURI from ever creating a StringIO object, and force it to create a temp file instead. See this SO answer:
Why does Ruby open-uri's open return a StringIO in my unit test, but a FileIO in my controller?
In the past, I found the most reliable way to retrieve remote files was by using the command line tool "wget". The following code is mostly copied straight from an existing production (Rails 2.x) app with a few tweaks to fit with your code examples:
class CategoryIconImporter
def self.download_to_tempfile (url)
system(wget_download_command_for(url))
##tempfile.path
end
def self.clear_tempfile
##tempfile.delete if ##tempfile && ##tempfile.path && File.exist?(##tempfile.path)
##tempfile = nil
end
def self.set_wget
# used for retrieval in NrlImage (and in future from other sies?)
if !##wget
stdin, stdout, stderr = Open3.popen3('which wget')
##wget = stdout.gets
##wget ||= '/usr/local/bin/wget'
##wget.strip!
end
end
def self.wget_download_command_for (url)
set_wget
##tempfile = Tempfile.new url.sub(/\?.+$/, '').split(/[\/\\]/).last
command = [ ##wget ]
command << '-q'
if url =~ /^https/
command << '--secure-protocol=auto'
command << '--no-check-certificate'
end
command << '-O'
command << ##tempfile.path
command << url
command.join(' ')
end
def self.import_from_url (category_params, url)
clear_tempfile
filename = url.sub(/\?.+$/, '').split(/[\/\\]/).last
found = MIME::Types.type_for(filename)
content_type = !found.empty? ? found.first.content_type : nil
download_to_tempfile url
nicer_path = RAILS_ROOT + '/tmp/' + filename
File.copy ##tempfile.path, nicer_path
Category.create(category_params.merge({:icon => ActionController::TestUploadedFile.new(nicer_path, content_type, true)}))
end
end
The rake task logic might look like:
[
['Cat', 'cat'],
['Dog', 'dog'],
].each do |name, icon|
CategoryIconImporter.import_from_url {:name => name}, "https://xyz.com/images/#{icon}.png"
end
This uses the mime-types gem for content type discovery:
gem 'mime-types', :require => 'mime/types'

Resources