I use fixture_file_upload in my FactoryGirl methods to test file uploads. Problem is that after cleaning the database, all these uploaded files remain on Cloudinary.
I've been using Cloudinary::Api.delete_resources using a rake task to get rid of them, but I'd rather immediately clean them up before DatabaseCleaner removes all related public id's.
Where should I interfere with DatabaseCleaner as to remove these files from Cloudinary?
Based on #phoet's input, and given the fact that cloudinary limits the amount of API calls you can do on a single day, as well as the amount of images you can cleanup in a single call, I created a class
class CleanupCloudinary
##public_ids = []
def self.add_public_ids
Attachinary::File.all.each do |image|
##public_ids << image.public_id
clean if ##public_ids.count == 100
end
end
def self.clean
Cloudinary::Api.delete_resources(##public_ids) if ##public_ids.count > 0
##public_ids = []
end
end
which I use as follows: in my factory girl file, I make a call to immediately add any public_ids after creating an advertisement:
after(:build, :create) do
CleanupCloudinary.add_public_ids
end
in env.rb, I added
at_exit do
CleanupCloudinary.clean
end
as well as in spec_helper.rb
config.after(:suite) do
CleanupCloudinary.clean
end
This results in, during testing, cleanup after each 100 cloudinary images, and after testing, to clean up the remainder
i would have two ways of doing things here.
firstly, i would not upload anything to cloudinary unless it is a integration test. i would use a mock, stub or test-double.
secondly, if you really really really need to upload the files for whatever reason, i would write a hook that does automatic cleanup in an after_all hook of you tests.
To make #Danny solution work in Minitest, instead of at_exit and config.after, add in test_helper.rb:
class ActiveSupport::TestCase
...
Minitest.after_run do
puts 'Cloudinary cleanup'
CleanupCloudinary.clean
end
end
If you need to cleanup more often, you can use teardown { CleanupCloudinary.clean } either globally in test_helper.rb or in specific test files.
And of course in the factory you still need:
after(:create) do
CleanupCloudinary.add_public_ids
end
Related
I'm writing tests with rspec and am a bit struggling with Paperclip 4. At the moment I'm using webmock to stub requests but image processing is slowing tests down.
Everything I read suggest to use stubs like so:
Profile.any_instance.stub(:save_attached_files).and_return(true)
It doesn't work since :save_attached_files disappeared with Paperclip 4 as far as I can tell.
What's the proper way to do it now ?
Thanks
Add this to your rails_helper.rb in your config block:
RSpec.configure do |config|
# Stub saving of files to S3
config.before(:each) do
allow_any_instance_of(Paperclip::Attachment).to receive(:save).and_return(true)
end
end
It doesn't exactly answer my question, but I found a way to run faster specs thanks to this dev blog, so I'm posting it if it can help someone else.
Just add this piece of code at the beginning of your spec file or in a helper. My tests are running 3x faster now.
# We stub some Paperclip methods - so it won't call shell slow commands
# This allows us to speedup paperclip tests 3-5x times.
module Paperclip
def self.run cmd, params = "", expected_outcodes = 0
cmd == 'convert' ? nil : super
end
end
class Paperclip::Attachment
def post_process
end
end
Paperclip > 3.5.2
For newer versions of Paperclip, use the following:
module Paperclip
def self.run cmd, arguments = "", interpolation_values = {}, local_options = {}
cmd == 'convert' ? nil : super
end
end
class Paperclip::Attachment
def post_process
end
end
I had a heckuva time dealing with this. I didn't want to test Paperclip per se -- rather, I needed a file to exist on my model so I can test a very sensitive custom class.
None of the other answers worked for me (Rails 5, Rspec 3.5) so I gave up on stubbing AWS or using the Paperclip::Shoulda::Matchers. Nothing worked and it ate up an entire day! Finally I gave up on Paperclip altogether and went this route:
list = build(:list)
allow(list).to receive_message_chain("file.url") { "#{Rails.root}/spec/fixtures/guess_fullname.csv" }
importer = Importer.new(list)
expect(importer.set_key_mapping).to eq({nil=>:email_address, :company=>:company, :fullname=>:full_name})
Where my models/list.rb has has_attached_file :file and Importer is a class I wrote that takes the file, which was already uploaded to S3 by paperclip before the class is initialized, and processes it. The file.url would otherwise be the S3 URL, so I give it the path in my repo to a testing csv file. set_key_mapping is a method in my class that I can use to verify the processing part worked.
Hope this saves somebody a few hours...
What about adding AWS.stub! in your spec/spec_helper.rb? Give that a try.
I have a number of cleanup scripts in my Rails application which are all classes that inherit from a common CleanupScript class. In order to run a cleanup script, I usually write Script.run unless Rails.env.test? at the bottom of each file (so that it can be run through rails runner).
Obviously, this is not particularly DRY, and I'm thinking that there must be a way within the superclass to set this up automatically. I'm aware of Kernel#at_exit, but I'm not sure if I can somehow use this within CleanupScript to know which script class to use, or if this is even the correct process?
at_exit does seem to be an appropriate way to solve this problem, this is what I ended up doing:
class CleanupScript
cattr_accessor :autorun
self.autorun = !Rails.env.test?
def self.inherited(klass)
at_exit do
klass.run if klass.autorun
end
end
end
This allows subclasses to specify SubclassedCleanupScript.autorun = false if I want to disable autorun for some reason (and also automatically disables autorunning for the test env), but in general will automatically run the class when its script is loaded by rails runner.
One thing I've noticed is that in most of the projects I do, the one spec that always takes a long time (30 seconds +) is this shoulda/paperclip helper:
it { should validate_attachment_content_type(:bannerimage)
.allowing('image/png', 'image/jpeg', 'image/gif', 'image/jpg')
.rejecting('text/plain')
}
I'd quite like to keep content type validation in, but I'm wondering if there's a speedier way to do it. I already tag these tests with a :slow and run rspec without :slow specs, but nonetheless, I'm hoping someone has a swifter way of testing image content types.
Looks like you're running your own tests against paperclip.
Generally I let gem providers (especially for large products like this one) certify that their specs will run successfully before pushing a release.
I stub out actual paperclip stuff from my tests to make them faster like this, placed in spec_helper.rb
# stub out paperclip? http://pivotallabs.com/stubbing-out-paperclip-imagemagick-in-tests/
# only like .1 seconds faster anyways though...
module Paperclip
def self.run cmd, params = "", expected_outcodes = 0
case cmd
when "identify"
return "100x100"
when "convert"
return
else
super
end
end
end
class Paperclip::Attachment
def post_process
end
end
My app does a lot with images. I use paperclip to attach them to models. I have tons of tests (Test::Unit) that involve creating images, these run pretty slowly.
I use FactoryGirl to create models in my tests. This is how I create image attachments:
factory :product_image_100_100 do
image File.new(File.join(::Rails.root.to_s, "/test/fixtures/images", "100_100.jpg"))
end
How can I fake the image upload or otherwise speed things up?
This snippet worked for me:
require 'test_helper'
class PhotoTest < ActiveSupport::TestCase
setup do
Paperclip::Attachment.any_instance.stubs(:post_process).returns(true)
end
# tests...
end
Upd. My current preference is to stub out ImageMagic globally, by adding the following to my test_helper.rb:
module Paperclip
def self.run(cmd, *)
case cmd
when "identify"
return "100x100"
when "convert"
return
else
super
end
end
end
(Adapted from here – btw, you may want to take a look at this article if you're interested in speeding up your tests)
#Here is how I have delayed job set up.
Delayed::Worker.backend = :active_record
#Delayed::Worker.logger = Rails.logger
Delayed::Worker.logger = ActiveSupport::BufferedLogger.new("log/
##{Rails.env}_delayed_jobs.log", Rails.logger.level)
Delayed::Worker.logger.auto_flushing = 1
class Delayed::Job
def logger
Delayed::Worker.logger
end
end
if JobsCommon::check_job_exists("PeriodicJob").blank?
Delayed::Job.enqueue PeriodicJob.new(), 0, 30.seconds.from_now
end
#end
#Here is my simple job.
class PeriodicJob
def perform
Rails.logger.info "Periodic job writing #{Time.now}"
Delayed::Job.enqueue PeriodicJob.new(), 0,
30.seconds.from_now
end
end
I don't see any log messages from delayed job in my rails logs or delayed job log file, the only messages I see are jobs starting/success/failure in the delayed_jobs.log file.
this is causing big problems, including detecting bugs and memory leaks in workers almost impossible! Please help!
We've gotten it to work on Rails 3/Delayed Job 2.0.3 by hacking Rails.logger itself to use a different log file (the one we want for delayed_job entries) and also setting the delayed job logger to use the exact same object:
file_handle = File.open("log/#{Rails.env}_delayed_jobs.log", (File::WRONLY | File::APPEND | File::CREAT))
# Be paranoid about syncing, part #1
file_handle.sync = true
# Be paranoid about syncing, part #2
Rails.logger.auto_flushing = true
# Hack the existing Rails.logger object to use our new file handle
Rails.logger.instance_variable_set :#log, file_handle
# Calls to Rails.logger go to the same object as Delayed::Worker.logger
Delayed::Worker.logger = Rails.logger
If the above code doesn't work, try replacing Rails.logger with RAILS_DEFAULT_LOGGER.
This may be a simple workaround but it works well enough for me:
system("echo #{your message here} >> logfile.log")
Simple but works
I have it working with the following setup in initializer:
require 'delayed/worker'
Delayed::Worker.logger = Rails.logger
module Delayed
class Worker
def say_with_flushing(text, level = Logger::INFO)
if logger
say_without_flushing(text, level)
logger.flush
end
end
alias_method_chain :say, :flushing
end
end
i simply did this:
/config/environments/development.rb
MyApp::Application.configure do
[...]
[...]
[...]
Delayed::Worker.logger = Rails.logger
end
In every next request you do the mail will be appear on the log.
NOTE: Sometimes you have to refresh the page to the mail be logged on the log. Don't forget to restart the server ;)
DelayedJob does not seem to output if there is something wrong:
1- Non-active record classes need to be required and initialized:
How:
Create a file config/initializers/load_classes_for_dj.rb
Add to it the lines:
require 'lib/libtest/delayed_test.rb'
DelayedTest
Note that if you have '#{config.root}/lib/libtest' in config.autoload_paths in config/application.rb, you do not need to do the require.
Source:
Rails Delayed Job & Library Class
2- Classes that implement the Singleton module won't work by calling:
SingletonClass.instance.delay.sayhello
In order to fix that, do the following:
class SingletonClass
include Singleton
# create a class method that does the call for you
def self.delayed_sayhello
SingletonClass.instance.sayhello
end
def sayhello
# this is how you can actually write to delayed_job.log
# https://stackoverflow.com/questions/9507765/delayed-job-not-logging
Delayed::Worker.logger.add(Logger::INFO, "hello there")
end
end
In order to call the delayed class method, do the following:
SingletonClass.delay.delayed_sayhello
Yes, I am calling .delay on a class here. Since classes in Ruby are also objects, the call is valid here and allows me to access the class method "delayed_sayhello"
3- Do not pass ActiveRecord objects or some complex objects to your call but rather pass ids, look up the objects in the database in your delayed method, and then do your work:
DO NOT DO:
DelayedClass.new.delay.do_some_processing_on_album Album.first
DO THIS INSTEAD:
DelayedClass.new.delay.do_some_processing_on_album Album.first.id
and inside DelayedClass do_some_processing_on_album , do
a = Album.find_by_id id
There was a stackoverflow post about this that I saw a while ago -- not sure which :-)
4- For completion, this is how to do mailers (do not call the deliver method):
Notifier.delay.signup(user_id)
As per 3, do not pass the user's object but rather their id, do the lookup inside the signup method.
Now once you've ensured you have followed the guidelines above, you can use:
Delayed::Worker.logger.add(Logger::INFO, "hello")
If you are still facing issues, I suggest you break down your problem:
a- Create a new class
b- Make sure you have it included and initialized as per step 1
c- Add logging from step 4 and try calling MyNewClass.new.delay to see if it works
I hope this helps you guys :-)
Don't forget to change the ActiveRecord::Base.logger
Delayed::Worker.logger = Logger.new("log/delayed_job.log", 5, 104857600)
if caller.last =~ /script\/delayed_job/ or (File.basename($0) == "rake" and ARGV[0] =~ /jobs\:work/)
ActiveRecord::Base.logger = Delayed::Worker.logger
end
More detailed answer: Have delayed_job log "puts", sql queries and jobs status