I'm trying to create a helper method that will display a variant if the variant exists, otherwise it will enqueue a job to resize the photo. The problem I'm facing is that the job is enqueued and seems to be performed every time the page loads. I would assume that after the job runs the first time it wouldn't need to run again because the variant would exist.
application_helper.rb:
def render_img(img, params, opts = {})
variant = img.variant(resize_to_fill: params, auto_orient: true, format: :jpg)
if img.service.exist?(variant.key)
image_tag variant, opts
else
ResizePhotoJob.perform_later(img.blob, resize_cmd: params)
image_tag img, opts
end
end
resize_photo_job.rb
# frozen_string_literal: true
class ResizePhotoJob < ApplicationJob
queue_as :default
def perform(file, resize_cmd:nil)
if resize_cmd.nil?
file.variant(auto_orient: true, format: :jpg, quality: 90).processed
else
file.variant(resize_to_fit: resize_cmd, auto_orient: true, format: :jpg).processed
end
end
end
You shouldn't need to do any of this. .processed looks like it can handle it for you:
When you do want to actually produce the variant needed, call processed. This will check that the variant has already been processed and uploaded to the service, and, if so, just return that. Otherwise it will perform the transformations, upload the variant to the service, and return itself again.
Related
Is there anyway that I can force this method, ActiveStorage::Attached#attach to not enqueue a background job? In other words, I would like to disable the async behavior which seems to be included in ActiveStorage::Attached#attach so that the method executes synchronously and not asynchronously with either ActiveJob or something like Sidekiq
This seems to work currently:
def sync_attach(record, name, attachable)
blob = ActiveStorage::Blob.create_and_upload!(
io: attachable.open, filename: attachable.original_filename
)
blob.analyze
attached = record.send(name)
attached.purge
attached.attach(blob)
end
# Given
class Thing < ActiveRecord::Base
has_one_attached :image
end
# You can do
thing = Thing.create!
sync_attach(thing, :image, attachable)
You should also include mirroring in the method if you use that.
I welcome and seek any warnings, corrections, additions, etc. This is the best I've come up with so far.
I upgraded carrierwave from 0.11.0 to 1.2.3 and realised that a, for me crucial, behaviour has changed and broke my logic. Here is the example of my uploader.
class FileUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :fog
version :thumb do
process :convert => :jpg
def default_url
'/assets/document_thumb.png'
end
end
end
And the model that it's mounted too:
class Material < ActiveRecord::Base
attr_accessible :name, :file
mount_uploader :file, FileUploader, validate_processing: false
before_create :create_file_hash
def create_file_hash
self.hash_digest = Digest::MD5.hexdigest(file.read)
end
end
In the old carrierwave, even if the version processing (e.g. in this case the convert) failed the main version of the file would still be uploaded and stored. However, now in cases when processing fails (not always, but I can't do conditional processing as my case is more complex then here illustrated) nothing gets stored. The file attribute remains an empty (blank) uploader and nothing is uploaded to the fog storage.
Any idea on how to get back the old behaviour?
In other words, how to ignore any errors with processing of versions. Or not trigger processing of versions in the after_cache callback but rather some later time down the line?
I think I've tracked down this issue to the following change in Mounter#cache method:
def cache(new_files)
return if not new_files or new_files == ""
#uploaders = new_files.map do |new_file|
uploader = blank_uploader
uploader.cache!(new_file)
uploader
end
#integrity_error = nil
#processing_error = nil
rescue CarrierWave::IntegrityError => e
#integrity_error = e
raise e unless option(:ignore_integrity_errors)
rescue CarrierWave::ProcessingError => e
#processing_error = e
raise e unless option(:ignore_processing_errors)
end
Which used to just do the uploader.cache!(new_file) directly (not in map) and then uploader got updated along the way and returned to the model when needed. However, now the processing error causes the map block to exit and #uploaders array never gets updated with the uploader that worked (i.e. for the original file).
One possible solution would be overriding the cache! method in you uploader instead:
class FileUploader < CarrierWave::Uploader::Base
def cache!(*)
super
rescue CarrierWave::ProcessingError => e
Rails.logger.debug "FileUploader: Error creating thumbnail: #{e}"
nil
end
...
end
That way, it works for every model
Half a day of effort later here is the solution I've come up with that doesn't involve monkey patching carrierwave.
class Material < ActiveRecord::Base
attr_accessible :name, :file
mount_uploader :file, FileUploader, validate_processing: false
#----> This is to manually trigger thumbnail creation <----
before_create :create_thumbnail
def create_thumbnail
file.thumb.cache!(file.file)
rescue CarrierWave::ProcessingError => e
Rails.logger.debug "FileUploader: Error creating thumbnail: #{e}"
end
# rest of the model code
end
So here we have create_thumbnail method triggered in before_create callback that manually calls the cache! method on the thumb uploader. The file.file is at this moment (i.e. before create, so before the file has been uploaded to the storage) pointing to the temporary cached file. Which is exactly what we want (we don't want to re-download the file from the storage just to create thumbnails.
class FileUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :fog
#----> Add the if condition to versions <----
version :thumb, if: :has_versions? do
process :convert => :jpg
#----> This is needed to trigger processing later <----
after :cache, :process!
def default_url
'/assets/document_thumb.png'
end
end
#---> This is to avoid versions before the main version is fully processed and cached <---
def has_versions?
!(model.new_record? && model[:file].nil?)
end
end
Now this is the tricky party. We need to initially disable the version creations and for that reason we have the has_versions? method that checks if the file is a new record. Now that check is not enough, because in our before_create callback the model is still new record (i.e. it hasn't yet been persisted).
However, what's the difference between the first time the uploader tries to create the versions (and which, if it fails, prevents original file from caching as described in the question) and the moment we call it in our before_create callback is that in the second case the file attribute of the model will be set.
Be careful, however, because you cannot do model.file since that points to the uploader (and if called here where I'm calling it it would actually cause a stack overflow). You need to access it as model[:file].
The final trick is that for some reason just calling cache! in the model would not actually trigger the processing. The processing was supposed to be triggered during the initial run (which we prevented for other versions) and since the original file is cached, carrierwave expects the versions are as well, so they don't need processing. But adding the after :cache, :process! ensures that it's triggered.
Don't know if anybody will find this useful or if I've gone about the problem the wrong way.
Either way, I'd love to hear comments.
Happy I made it work for my case and that I can continue using latest gem version.
I'm using ActiveJob with SideKiq and with the code below I'm getting differing results when using perform_later than perform_now.
When using perform_now my code creates the specified folder and file.
When using perform_later I can see the code executes and doesn't get thrown into the Retry queue of Sidekiq, yet doesn't create the folder or file.
If there is anything else I could produce to help troubleshoot the issue please let me know, as I've likely just overlooked it.
app/controllers/concerns/pdf_player_reports.rb
module PdfPlayerReports
extend ActiveSupport::Concern
included do
# before_action :set_player
# before_action :set_user_report
end
def director_report_pdf
#players = Player.where(id: params["id"])
html = render_to_string template: "players/director_summary_report.pdf.erb"
CreatePdfJob.perform_later(html)
#CreatePdfJob.perform_now(html)
end
end
app/jobs/create_pdf_job.rb
class CreatePdfJob < ActiveJob::Base
queue_as :high_priority
def perform(*args)
generate_pdf_from_string(args.first)
end
def generate_pdf_from_string(html)
# create pdf from passed in HTML string
pdf = WickedPdf.new.pdf_from_string(html)
# Create the directory for the pdfs that will be gernereated incases where it doesn't already exist
pdf_directory_path = Rails.root.join('public','uploads','pdfs')
unless File.directory? pdf_directory_path
FileUtils.mkdir_p(pdf_directory_path)
end
# Open the specified file and then write the contents to disk
File.open(pdf_directory_path.join('output.pdf'),'wb') do |f|
f << pdf
end
end
end
The issue (to my understanding) seems to be that I need to cancel my sidekiq process and restart it (sidekiq -C config/sidekiq.yml) anytime I change code on the Job file.
I want to create 1 uploader for multiple types of files (images, pdf, video)
For each content_type will different actions
How I can define what content_type of file?
For example:
if image?
version :thumb do
process :proper_resize
end
elsif video?
version :thumb do
something
end
end
I came across this, and it looks like an example of how to solve this problem: https://gist.github.com/995663.
The uploader first gets loaded when you call mount_uploader, at which point things like if image? or elsif video? won't work, because there is no file to upload defined yet. You'll need the methods to be called when the class is instantiated instead.
What the link I gave above does, is rewrite the process method, so that it takes a list of file extensions, and processes only if your file matches one of those extensions
# create a new "process_extensions" method. It is like "process", except
# it takes an array of extensions as the first parameter, and registers
# a trampoline method which checks the extension before invocation
def self.process_extensions(*args)
extensions = args.shift
args.each do |arg|
if arg.is_a?(Hash)
arg.each do |method, args|
processors.push([:process_trampoline, [extensions, method, args]])
end
else
processors.push([:process_trampoline, [extensions, arg, []]])
end
end
end
# our trampoline method which only performs processing if the extension matches
def process_trampoline(extensions, method, args)
extension = File.extname(original_filename).downcase
extension = extension[1..-1] if extension[0,1] == '.'
self.send(method, *args) if extensions.include?(extension)
end
You can then use this to call what used to be process
IMAGE_EXTENSIONS = %w(jpg jpeg gif png)
DOCUMENT_EXTENSIONS = %(exe pdf doc docm xls)
def extension_white_list
IMAGE_EXTENSIONS + DOCUMENT_EXTENSIONS
end
process_extensions IMAGE_EXTENSIONS, :resize_to_fit => [1024, 768]
For versions, there's a page on the carrierwave wiki that allows you to conditionally process versions, if you're on >0.5.4. https://github.com/jnicklas/carrierwave/wiki/How-to%3A-Do-conditional-processing. You'll have to change the version code to look like this:
version :big, :if => :image? do
process :resize_to_limit => [160, 100]
end
protected
def image?(new_file)
new_file.content_type.include? 'image'
end
I'm using FasterCSV on a Ruby on Rails application and currently it throws an Exception if the file is invalid.
I've looked over the FasterCSV doc, and it seems that if I use FasterCSV::parse with a block, it'll read the file one line at a time, without allocating too much memory. It'll throw a FasterCSV::MalformedCSV exception if there is any kind of error on the file.
I've implemented a custom solution, but I'm not sure it's the best possible one (see my answer below). I'd be interested in knowing alternatives
This is my current solution. I'm really interested in knowing improvements / alternatives.
# /lib/fastercsv_is_valid.rb
class FasterCSV
def self.is_valid?(file, options = {})
begin
FasterCSV.parse(file, options) { |row| }
true
rescue FasterCSV::MalformedCSV
false
end
end
end
I use that method like this:
# /models/csv_importer.rb
class CsvImporter
include ActiveRecord::Validations
validates_presence_of :file
validate check_file_format
...
private
def check_file_format
errors.add :file, "Malformed CSV! Please check syntax" unless FasterCSV::is_valid? file
end
end
I made some tests yesterday and it turns out that my solution didn't quite work; I kept getting empty arrays on valid CSVs after implementing the first is_valid . I'm not sure whether it's a FasterCSV caching issue or something in my code, and I don't know if it's related with my test setup, but I decided to go implement a safe_parse instead:
#/lib/faster_csv_safe_parse.rb
class FasterCSV
def self.safe_parse(file, options = {})
begin
FasterCSV.parse(file, options)
rescue FasterCSV::MalformedCSVError
nil
end
end
end
This will return a parsed array if the file is valid, or nil otherwise. I could then implement my validations as follows:
# /models/csv_importer.rb
class CsvImporter
include ActiveRecord::Validations
validates_presence_of :file
validate check_file_format
attr_accessor csv_data
def csv_data
#csv_data ||= FasterCSV.safe_parse(file)
end
...
private
def check_file_format
errors.add :file, "Malformed CSV! Please check syntax" if csv_data.nil?
end
end
I guess it would be possible to implement a safe_parse that accepts a block and parses the file line by line, but for my purposes this simple implementation was enough, and it works in all cases.
I assume you want to parse the CSV and do something with the parsed results. Worst case is that your CSV is valid and that you parse the file again. I would write something like this to stash away the parsed result so you only have to parse the CSV once:
module FasterCSV
def self.parse_and_validate(file, options = {})
begin
#parsed_result = FasterCSV.parse(file, options) { |row| }
rescue FasterCSV::MalformedCSV
#invalid = true
end
end
def self.is_valid?
!#invalid
end
def self.parsed_result
#parsed_result if self.valid?
end
end
And then:
class CsvImporter
include ActiveRecord::Validations
validates_presence_of :file
validate check_file_format
# I assume you use the parsed result after the validations so in a before_save or something
def do_your_parse_stuff
here you would use FasterCSV::parsed_result
end
...
private
def check_file_format
FasterCSV::parse_and_validate(file)
errors.add :file, "Malformed CSV! Please check syntax" unless FasterCSV::is_valid?
end
end
In the above case, you might want to move stuff into a different class that takes care of communicating with FasterCSV and stashing away the parsed result, because I don't think my example is thread safe :)