I am essentially writing the project in Railscast 383 - the second part, when the photo is uploaded directly to AWS S3, and the photo is then processed in the background by Sidekiq to create a thumbnail version of the photo. I'm on Rails 4.
My issue is that the Sidekiq job, after completing successfully, keeps repeating over and over, instead of just stopping.
Where am I going wrong? I can't see any difference between my code and that of the Railscast, other than I'm on Rails 4 (so strong parameters instead of attr_accessible)
Photo class:
class Photo < ActiveRecord::Base
mount_uploader :image, ImageUploader
default_scope order('updated_at DESC')
after_save :enqueue_image
def image_name
File.basename(image.path || image.filename) if image
end
def enqueue_image
ImageWorker.perform_async(id, key) if key.present?
end
end
ImageWorker:
class ImageWorker
include Sidekiq::Worker
sidekiq_options retry: false
# sidekiq_options retry: 3
def perform(id, key)
photo = Photo.find(id)
photo.key = key
photo.remote_image_url = photo.image.direct_fog_url(with_path: true)
photo.save!
photo.update_column(:image_processed, true)
end
end
Uploader:
class ImageUploader < CarrierWave::Uploader::Base
include CarrierWaveDirect::Uploader
include CarrierWave::RMagick
# storage :fog
#### storage defaults to fog when CarrierWaveDirect::Uploader is included ####
include CarrierWave::MimeTypes
process :set_content_type
version :thumb do
process :resize_to_limit => [200, 200]
end
version :medium do
process :resize_to_limit => [400, 400]
end
end
One reason for the sidekiq worker to be called again and again is because the perform_async is called every time you save your photo object, which occurs within the sidekiq worker itself.
So every time the ImageWorker is called, it saves the photo, calling the ImageWorker again, creating the loop you are experiencing.
Are you sure you aren't missing a check for the :image_processed tag to be true before calling the ImageWorker again.
Try this:
def enqueue_image
ImageWorker.perform_async(id, key) if key.present? && !image_processed
end
This will check if the image was processed once before. I think this was probably meant to be set in the rails cast but the author forgot about it, otherwise the image_processed flag is obsolete.
Related
I upgraded carrierwave from 0.11.0 to 1.2.3 and realised that a, for me crucial, behaviour has changed and broke my logic. Here is the example of my uploader.
class FileUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :fog
version :thumb do
process :convert => :jpg
def default_url
'/assets/document_thumb.png'
end
end
end
And the model that it's mounted too:
class Material < ActiveRecord::Base
attr_accessible :name, :file
mount_uploader :file, FileUploader, validate_processing: false
before_create :create_file_hash
def create_file_hash
self.hash_digest = Digest::MD5.hexdigest(file.read)
end
end
In the old carrierwave, even if the version processing (e.g. in this case the convert) failed the main version of the file would still be uploaded and stored. However, now in cases when processing fails (not always, but I can't do conditional processing as my case is more complex then here illustrated) nothing gets stored. The file attribute remains an empty (blank) uploader and nothing is uploaded to the fog storage.
Any idea on how to get back the old behaviour?
In other words, how to ignore any errors with processing of versions. Or not trigger processing of versions in the after_cache callback but rather some later time down the line?
I think I've tracked down this issue to the following change in Mounter#cache method:
def cache(new_files)
return if not new_files or new_files == ""
#uploaders = new_files.map do |new_file|
uploader = blank_uploader
uploader.cache!(new_file)
uploader
end
#integrity_error = nil
#processing_error = nil
rescue CarrierWave::IntegrityError => e
#integrity_error = e
raise e unless option(:ignore_integrity_errors)
rescue CarrierWave::ProcessingError => e
#processing_error = e
raise e unless option(:ignore_processing_errors)
end
Which used to just do the uploader.cache!(new_file) directly (not in map) and then uploader got updated along the way and returned to the model when needed. However, now the processing error causes the map block to exit and #uploaders array never gets updated with the uploader that worked (i.e. for the original file).
One possible solution would be overriding the cache! method in you uploader instead:
class FileUploader < CarrierWave::Uploader::Base
def cache!(*)
super
rescue CarrierWave::ProcessingError => e
Rails.logger.debug "FileUploader: Error creating thumbnail: #{e}"
nil
end
...
end
That way, it works for every model
Half a day of effort later here is the solution I've come up with that doesn't involve monkey patching carrierwave.
class Material < ActiveRecord::Base
attr_accessible :name, :file
mount_uploader :file, FileUploader, validate_processing: false
#----> This is to manually trigger thumbnail creation <----
before_create :create_thumbnail
def create_thumbnail
file.thumb.cache!(file.file)
rescue CarrierWave::ProcessingError => e
Rails.logger.debug "FileUploader: Error creating thumbnail: #{e}"
end
# rest of the model code
end
So here we have create_thumbnail method triggered in before_create callback that manually calls the cache! method on the thumb uploader. The file.file is at this moment (i.e. before create, so before the file has been uploaded to the storage) pointing to the temporary cached file. Which is exactly what we want (we don't want to re-download the file from the storage just to create thumbnails.
class FileUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :fog
#----> Add the if condition to versions <----
version :thumb, if: :has_versions? do
process :convert => :jpg
#----> This is needed to trigger processing later <----
after :cache, :process!
def default_url
'/assets/document_thumb.png'
end
end
#---> This is to avoid versions before the main version is fully processed and cached <---
def has_versions?
!(model.new_record? && model[:file].nil?)
end
end
Now this is the tricky party. We need to initially disable the version creations and for that reason we have the has_versions? method that checks if the file is a new record. Now that check is not enough, because in our before_create callback the model is still new record (i.e. it hasn't yet been persisted).
However, what's the difference between the first time the uploader tries to create the versions (and which, if it fails, prevents original file from caching as described in the question) and the moment we call it in our before_create callback is that in the second case the file attribute of the model will be set.
Be careful, however, because you cannot do model.file since that points to the uploader (and if called here where I'm calling it it would actually cause a stack overflow). You need to access it as model[:file].
The final trick is that for some reason just calling cache! in the model would not actually trigger the processing. The processing was supposed to be triggered during the initial run (which we prevented for other versions) and since the original file is cached, carrierwave expects the versions are as well, so they don't need processing. But adding the after :cache, :process! ensures that it's triggered.
Don't know if anybody will find this useful or if I've gone about the problem the wrong way.
Either way, I'd love to hear comments.
Happy I made it work for my case and that I can continue using latest gem version.
I have been struggling to find any tutorial or question that explains how to upload image and resize it according to certain conditions provided by user.
I am easily able to upload and resize image using hard coded values, however I am stuck at using user provided parameters to be accessed from the Uploader.
I want the image to be resized to either 800x600 or 300x300 based on whether the user checks the image as Large or Small.
For that I have a boolean column named "large" in the model structure.
In the Uploader I am able to access model and its values easily in the store_dir block, but anywhere outside this block any model attribute returns as nil.
This is what I want to do:-
class BannerUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :file
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
resize_to_fit(800,600) if model.large==true
resize_to_fit(300,300) if model.large!=true
end
However this returns the error
undefined local variable or method `model' for BannerUploader:Class
How to go about this issue.
To process an original file you can specify custom method:
class BannerUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :file
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
process :process_original_version
def process_original_version
if model.large
resize_to_fit(800,600)
else
resize_to_fit(300,300)
end
end
end
For a specific version:
class BannerUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :file
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
version :normal do
if model.large
process resize_to_fit: [800,600]
else
process resize_to_fit: [300,300]
end
end
end
Ok so #Alex Kojin has the right answer. However I was faced with another problem as well. When the user submitted the form, the image would be resized always as small (300x300) because for some reason the image resize process would be executed first and then the "large" attribute would be set as true. Therefore the Uploader would always get model.large as false.
So this is how I had to change my action controller
def create
#banner=Banner.new
#banner.large=params[:large]
#banner.update_attributes(banner_params)
#banner.save
redirect_to :back
end
Not sure if this is the right approach but does work for me.
I have an Event model that has many photographs. I have an image uploader mounted to the Photographs attribute, and regular uploads and everything is working fine.
However, when I try and duplicate an event, recreating a new photograph object for the new event, the new image is darker than the original, and if I duplicate the duplicate event, it gets darker still.
I have played around with it, but have no solution.
My Uploader code:
class ImageUploader < CarrierWave::Uploader::Base
include CarrierWave::RMagick
include CarrierWave::Processing::RMagick
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
def cache_dir
"#{Rails.root}/tmp/carrierwave"
end
process :colorspace => :rgb
# Remove EXIF data
process :strip
# Create different versions of your uploaded files:
version :thumb do
process :resize_to_limit => [640, 640]
end
version :preview_thumb do
process :resize_to_limit => [600, 600]
end
version :wine_thumb do
process :resize_to_limit => [160, 440]
end
version :logo_thumb do
process :resize_to_limit => [90, 90]
end
end
And my duplcation code (in Active Admin):
member_action :create_duplicate_event, method: :post do
old_event = Event.find(params[:id])
photograph_urls = old_event.photographs.map(&:image_url)
attributes = old_event.attributes.except("photographs", "id")
new_photos = []
photograph_urls.each do |photo|
new_photo = Photograph.new({
remote_image_url: photo
})
if new_photo.save
new_photos << new_photo
end
end
#event = Event.new(attributes)
#event.photograph_ids = new_photos.map(&:id)
render "/admin/events/_events_form/"
end
The :rgb tag was an attempt to fix. But no luck.
Ruby 2.1 and Rails 4.0
Ok, after a lot of playing around and searching I managed to fix this problem.
First, you need to download the .icc color profiles, which can be found here. It says for windows but they seemed to work for me on my Mac.
Once you have put the .icc files into a /lib/color_profiles directory, add the following code to your uploader:
class ImageUploader < CarrierWave::Uploader::Base
include CarrierWave::RMagick
process :convert_image_from_cmyk_to_rgb
#versions, and any other uploader code go here
def convert_image_from_cmyk_to_rgb
manipulate! do |image|
if image.colorspace == Magick::CMYKColorspace
image.strip!
image.add_profile("#{Rails.root}/lib/USWebCoatedSWOP.icc")
image.colorspace == Magick::SRGBColorspace
image.add_profile("#{Rails.root}/lib/sRGB.icc")
end
image
end
end
end
This converts CMYK images to RGB, and keeps the profiles keeping nice, while keeping RGB images as they were, and not ruining them.
I hope this helps someone in the future, and saves them the hours I spent working this out.
I'm using Carrierwave to upload some images and generate different versions of them (size, color...). Every image is given a unique name when uploaded and I'm now trying to process the images in the background in order to avoid blocking web processes. I use carrierwave_backgrounder for that.
The problem is that when my image is processed, a new name is given to it. So the model doesn't know this new name and the original image is now twice on the server.
I end up with this kind of files:
f9f97657-eaab-40ce-b965-31bb128066ee.jpg // First uploaded images
e4244551-7f43-4c03-8747-e8f2f2e57156.jpg // Copy of the original image created while processed in the background
thumb_e4244551-7f43-4c03-8747-e8f2f2e57156.jpg
And event.image returns f9f97657-eaab-40ce-b965-31bb128066ee.jpg.
How can I avoid to generate a new filename if we are just processing the different versions in the background and not uploading it?
My model:
class Event < ActiveRecord::Base
mount_uploader :cover_image, CoverImageUploader
process_in_background :cover_image
end
My uploader:
class CoverImageUploader < CarrierWave::Uploader::Base
include CarrierWave::RMagick
include CarrierWave::MimeTypes
include Sprockets::Helpers::RailsHelper
include Sprockets::Helpers::IsolatedHelper
include ::CarrierWave::Backgrounder::Delay
def filename
"#{secure_token}.#{file.extension}" if original_filename
end
process :set_content_type
version :thumb do
process resize_to_fill: [200, 200]
end
# ...
# Some other versions processing...
# ...
protected
def secure_token
var = :"##{mounted_as}_secure_token"
model.instance_variable_get(var) || model.instance_variable_set(var, SecureRandom.uuid)
end
end
can try this:
def filename
if original_filename.present?
if model.respond_to?("#{mounted_as}_processing") && model.send("#{mounted_as}_processing")
#name ||= model.send("#{mounted_as}_identifier")
else
#name ||= "#{secure_token}.#{file.extension}"
end
end
end
only works if you have a "processing" column in your table
I'm try to set up carrierwave_direct for some image uploading in my rails 3.2 application but seem unable to render the form by using the direct_upload_form_for tags.
I'm getting the error "undefined method `direct_fog_url' for #< PostmarkerImage:0x007fdbe07b39f0 >" when loading the page containing the form.
PostmarkerImage is the model in which I've mounted the uploader like so:
class PostmarkerImage < ActiveRecord::Base
attr_accessible :image, :image_cache
belongs_to :postmarker
validates :image, :presence => true
mount_uploader :image, PostmarkerImageUploader
end
The image column is a string and I've also ensured that the migration has been run to create the column. The uploader now looks like this:
class PostmarkerImageUploader < CarrierWave::Uploader::Base
include CarrierWaveDirect::Uploader
# Include RMagick or MiniMagick support:
# include CarrierWave::RMagick
include CarrierWave::MiniMagick
# Include the Sprockets helpers for Rails 3.1+ asset pipeline compatibility:
include Sprockets::Helpers::RailsHelper
include Sprockets::Helpers::IsolatedHelper
# Choose what kind of storage to use for this uploader:
# storage :file
# storage :fog
include CarrierWave::MimeTypes
process :set_content_type
# Provide a default URL as a default if there hasn't been a file uploaded:
# def default_url
# # For Rails 3.1+ asset pipeline compatibility:
# # asset_path("fallback/" + [version_name, "default.png"].compact.join('_'))
#
# "/images/fallback/" + [version_name, "default.png"].compact.join('_')
# end
# Process files as they are uploaded:
# process :scale => [200, 300]
#
# def scale(width, height)
# # do something
# end
# Create different versions of your uploaded files:
# version :thumb do
# process :scale => [50, 50]
# end
# Add a white list of extensions which are allowed to be uploaded.
# For images you might use something like this:
def extension_white_list
%w(jpg jpeg gif png)
end
# Override the filename of the uploaded files:
# Avoid using model.id or version_name here, see uploader/store.rb for details.
# def filename
# "something.jpg" if original_filename
# end
end
I've also ensured that carrierwave_direct, fog, mini_magick and Imagemagick have already been installed. May I know what else I have missed out? It almost seems like the mounting of the uploader has not been done properly.
Managed to figure it out :) Just in case anyone has this problem:
I am implementing a separate model to hold my images instead of adding a column to an existing table. So the direct_upload_form_for expects the column (e.g. Image.new.image) instead of the Object (e.g. Image.new) which I was returning. I guess I was thrown off by the form_for part which usually expects an Object!
For anyone else who runs into this error, you are likely not passing the correct object to direct_upload_form_for. Let's say we had a User model with an avatar column. It should look like.
Model:
class User < ActiveRecord::Base
mount_uploader :avatar, AvatarUploader
end
Controller:
def edit # Could be any action, but I always direct upload to existing objects
#uploader = #user.avatar
#uploader.success_action_redirect = edit_registration_url(resource)
end
View (erb):
<%= direct_upload_form_for #uploader do |f| %>
<%= f.file_field :image %>
<% end %>
Notice how the #uploader object is actually the column #user.avatar, not the object #user. That's what the OP is talking about in his answer. It's just more confusing in his case because the object is not a User, it's a PostmarkerImage, which I assume is a model that only exists to hold images. This is common when you want one object (ie. a Postmarker) to have multiple images. So you create a model only for images, and set up a has_many relationship between them.
Pitfall: I was also getting this error a lot in production because I was forgetting to copy the code in the controller from the edit action to the update action. If the upload fails due to validation (like file type), you need to remember to set #uploader again! Similarly, if you have your uploader in new, remember to assign #uploader in create, as well.