I have a wrapper that sends user updates to an external service, on a regular basis, running inside a Sidekiq worker. Each User has its own Sidekiq Job. Sidekiq is setup to use 20 threads. It's a Rails 5.0.1 app, MRI Ruby 2.3.0. Webserver is Passenger 5 Community.
If I over simplify, the code looks like this:
class ProviderUserUpdateJob < ApplicationJob
queue_as :default
def perform(user_id)
user = User.find(user_id)
Provider::User.new(user).push_update
end
end
class Provider::User
def initialize(user)
#user = user
end
def push_update
SomeApiWrapper.call(
user_id: #user.id,
status: #user.status
)
end
....
end
Now, the BIG problem that I only have on production and I finally catched up by looking at the logs can be summarized like this :
class Provider::User
def initialize(user)
#user = user
end
def push_update
SomeApiWrapper.call(
user_id: #user.id, # Some user
status: #user.status # NOT THE SAME USER !!! (and I have no idea where he is coming from)
)
end
....
end
2 Questions:
How it this even possible? Does it comes from Provider::User being by essence a globally accessible object, so, from threads to threads, everything gets mixed up in a mutating soup?!
If I only use "functional" style, without any instance, passing parameters and outputs from static methods to static methods, can it solve my problem or am I completely wrong? How can I fix this?
Ultimately, is there any way to really battle test this kind of code so I can be sure not to mix users data?
Ok... turns out it's a dummy data issue. I just lost 2 hours trying to figure it out with complicated explanations but the answer was, simply, in my DB. Well done :-/
I working on project using Rails 4.1.6 now. And I have strange problem. Method to_param for my models (Product, Category) sometimes not calling. I use it for SEO-friendly urls.
Here is my to_param method for Category model:
def to_param
puts 'I am in Category to_param'
"#{id}-#{title.to_slug.normalize.to_s}"
end
I use puts for find out is this method working or no. So, when my urls looks good (/categories/39-средства-дезинфекции) I can see the string 'I am in Category to_param' on my server console. This is correct case and all it's great.
But sometimes I have urls like /categories/39 for the same objects. When I look into console for this case, I don't see any prints from my to_param method form Category model.
These two cases I have on the same pages, same views and using the same helpers for category url (category_path).
Most complicated for this situation is that I can't reproduce this bug and don't see any regularity. For the same objects I have correct urls most of times, but sometimes it's not. If I restart rails server and refresh browser with clear cache – problem may out and urls will be correct again.
During my debug and research I found source code for base class. But I can't see there any reasons for the situation described above.
def to_param(method_name = nil)
if method_name.nil?
super()
else
define_method :to_param do
if (default = super()) &&
(result = send(method_name).to_s).present? &&
(param = result.squish.truncate(20, separator: /\s/, omission: nil).parameterize).present?
"#{default}-#{param}"
else
default
end
end
end
end
Also I can tell that this problem was appear, when I used FriendlyID before, using regex for clear and build slugs, and now for babosa gem. So, I think the problem is my to_param sometimes not calling for my model.
So, I found the reason of this behaviour. Now it's resolved!
The reason was I have redefined to_param for Product and Category in my ActiveAdmin files:
before_filter do
Product.class_eval do
def to_param
puts "I am in admin Product to_param"
id.to_s
end
end
Category.class_eval do
def to_param
puts "I am in admin Category to_param"
id.to_s
end
end
end
So, when I was log in Admin panel and go to Product page – "bug" will appear on front-end views.
So, I need to remove Product.class_eval and Category.class_eval blocks from my admin classes.
Earlier, I had posted this question – and thought it was resolved:
Rails background worker always fails first time, works second
However, after continuing with tests and development, the error is back again, but in a slightly different way.
I'm using Sidekiq (with Rails 3.2.8, Ruby 1.9.3) to run background processes, after_save. Below is the code for my model, worker, and controller.
Model:
class Post < ActiveRecord::Base
attr_accessible :description,
:name,
:key
after_save :process
def process
ProcessWorker.perform_async(id, key) if key.present?
true
end
def secure_url
key.match(/(.*\/)+(.*$)/)[1]
end
def nonsecure_url
key.gsub('https', 'http')
end
end
Worker:
class ProcessWorker
include Sidekiq::Worker
def perform(id, key)
post = Post.find(id)
puts post.nonsecure_url
end
end
(Updated) Controller:
def create
#user = current_user
#post = #user.posts.create(params[:post])
render nothing: true
end
Whenever jobs are first dispatched, no matter the method, they fail initially:
undefined method `gsub' for nil:NilClass
Then, they always succeed on the first retry.
I've come across the following github issue, that appears to be resolved – relating to this same issue:
https://github.com/mperham/sidekiq/issues/331
Here, people are saying that if they create initializers to initialize the ActiveRecord methods on the model, that it resolves their issue.
To accomplish this, I've tried creating an initializer in lib/initializers called sidekiq.rb, with the following, simply to initialize the methods on the Post model:
Post.first
Now, the first job created completes successfully the first time. This is good. However, a second job created fails the first time – and completes upon retry... putting me right back to where I started.
This is really blowing my mind – has anyone had the same issue? Any help is appreciated.
Change your model callback from after_save to after_commit for the create action. Sometimes, sidekiq can initialize your worker before the model actually finishes saving to the database.
after_commit :process, :on => :create
so some context, I got some advice here:
Scheduling events in Ruby on Rails
aand have been tying to implement it today. I cant seem to make it work though. this is my scheduler job that is used to move my questions around between a delayed queue and a ready to send out queue (i've since decided to use email instead of SMS)
require 'Assignment'
require 'QuestionMailer'
module SchedulerJob
#delayed_queue = :delayed_queue
#ready_queue
def self.perform()
#delayed_queue.each do |a|
if(Time.now >= a.question.schedule)
#ready_queue << a
#delayed_queue.delete(a)
end
end
push_questions
end
def self.gather()
assignments = Assignment.find :all
assignments.each do |a|
#delayed_queue << a unless #delayed_queue.include? a
end
end
private
def self.push_questions
#ready_queue.each do |a|
QuestionMailer.question(a)
end
end
end
I use a callback on_create to call the gather method every time an assignment is created, and then the perform action actually does the sending of emails when resque runs.
I'm getting a strange error from the callback though.
undefined method `include?' for :delayed_queue:Symbol
here is the code from the assignment model
class Assignment < ActiveRecord::Base
belongs_to :user
belongs_to :question
attr_accessible :title, :body, :user_id, :question_id , :response , :correct
after_create :queue_assignments
def grade
self.correct = (response == self.question.solution) unless response == nil
end
def queue_assignments
SchedulerJob.gather
end
Any ideas what's going on? I think this is a problem with my understanding of how these queue's work with resque-scheduler. I assumed that if the queues were list-like objects then I could operate on them , but it appears that it a symbol instead of something with methode like include? I assume the << notation for adding something to it is also invalid.
Also please advise if this isn't the way to go about handling this kind of job scheduling
It appears you may have not restarted your Rails app after adding the new method gather to the SchedulerJob module. Try restarting your app to resolve this.
You may also be able to add the directory containing your Resque worker to Rails' watchable_dirs array so that changes you make to Resque worker modules in development don't require restarting your app. See this blog post for details:
http://wondible.com/2012/01/13/rails-3-2-autoloading-in-theory/
I'm working on a Rails app, where I'm using page caching to store static html output. The caching works fine. I'm having trouble expiring the caches, though.
I believe my problem is, in part, because I'm not expiring the cache from my controller. All of the actions necessary for this are being handled within the model. This seems like it should be doable, but all of the references to Model-based cache expiration that I'm finding seem to be out of date, or are otherwise not working.
In my environment.rb file, I'm calling
config.load_paths += %W( #{RAILS_ROOT}/app/sweepers )
And I have, in the /sweepers folder, a LinkSweeper file:
class LinkSweeper < ActionController::Caching::Sweeper
observe Link
def after_update(link)
clear_links_cache(link)
end
def clear_links_cache(link)
# expire_page :controller => 'links', :action => 'show', :md5 => link.md5
expire_page '/l/'+ link.md5 + '.html'
end
end
So ... why isn't it deleting the cached page when I update the model? (Process: using script/console, I'm selecting items from the database and saving them, but their corresponding pages aren't deleting from the cache), and I'm also calling the specific method in the Link model that would normally invoke the sweeper. Neither works.
If it matters, the cached file is an md5 hash off a key value in the Links table. The cached page is getting stored as something like /l/45ed4aade64d427...99919cba2bd90f.html.
Essentially, it seems as though the Sweeper isn't actually observing the Link. I also read (here) that it might be possible to simply add the sweeper to config.active_record.observers in environment.rb, but that didn't seem to do it (and I wasn't sure if the load_path of app/sweepers in environment.rb obviated that).
So I've tried a number of different approaches, to see what works, and what doesn't.
Again, to summarize the situation: My goal is to expire cached pages when an object updates, but to expire them without relying on a Controller action. Conventional sweepers use a line in the controller to notify the sweeper that it needs to function. In this case, I can't use a line in the controller, as the update is happening within the model. Normal sweeper tutorials aren't working, as they presume that your main interaction with the database object is through the controller.
If, in reading this, you see a way to tighten up my code, please comment and let me know.
First, let's look at the things that DO work, in case you're stuck on this, too, and need help.
Of all the things I tried, the only thing that really seemed to work was to declare an after_update command in the Observer for the model. In that command, I used the explicit command for the expire_page action, and included a path that had been declared in routes.rb.
So. This works:
In config/routes.rb:
map.link 'l/:md5.:format', :controller => 'links', :action => 'show'
In app/models/link_observer.rb:
def after_update(link)
ActionController::Base.expire_page(app.link_path(:md5 => link.md5))
end
Note that that "md5" is specific to my app. You might want to use :id or some other unique identifier.
I also found that declaring that ActionController::Base... line from the method in the model that's doing the updating worked. That is, within Link.rb, in the method that's actually updating the database, if I just stuck that whole line in, it worked. But since I might want to expire that page cache on other methods in the future, I'd rather have it extracted into the Observer.
Now, let's look at some things that DID NOT work, in case you're Googling around for this.
Calling "expire_page(...)" within the after_update(link) method within link_observer.rb did not work, as it returned an "undefined method `expire_page'" error
Creating a Sweeper file that observed the Model did not work. I couldn't find any error codes, but it just seemed to not even be aware that it had a job to do. This was after explicitly calling "config.load_paths += %W( #{RAILS_ROOT}/app/sweepers )" within environment.rb. Just in case I fat-fingered something in that code, here it is:
class LinkSweeper < ActionController::Caching::Sweeper
observe Link
def after_update(link)
clear_links_cache(link)
end
def clear_links_cache(link)
# DID NOT WORK expire_page :controller => 'links', :action => 'show', :md5 => link.md5
# DID NOT WORK expire_page '/l/'+ link.md5 + '.html'
# DID NOT WORK ActionController::Base.expire_page(app.link_path(:md5 => link.md5))
end
end
That above example had the link_sweeper.rb file in a directory, /app/sweepers. I also tried putting link_sweeper.rb within the app/models directory, and tried calling it with the config.active_record.observers command in environment.rb:
config.active_record.observers = :link_observer, :link_sweeper
But that didn't work, either.
So, yeah. It's quite possible that one of these methods would work, and that I messed up something in the code. But I think I did everything by the book.
Ultimately, to summarize: Rather than using a Sweeper to expire page caching, you want to set up an after_ callback in the model's Observer. You'll want to use the explicit path to the Base.expire_page method:
def after_update(<model>) # where <model> is the name of the model you're observing
ActionController::Base.expire_page(app.<model>_path(:id => <model>.id)) # where <model> is the name of the model you're observing
end
Hopefully this will help someone else down the road. Again, if you see anywhere in my not-working code where I should have done something differently, please let me know. If you see something in my working code that can be tighter, please let me know that, too.
Just a note: you can use cache_sweeper in ApplicationController.
class ApplicationController < ActionController::Base
cache_sweeper :my_sweeper
end
class MySweeper < ActionController::Caching::Sweeper
observe MyModel
def after_update(my_model)
expire_page(...)
end
end
I was experiencing the same problem when trying to do fragment caching (rails 3). Couldn't get the sweeper to observe, so I settled for the solution to make it an AR Observer as described above and calling ApplicationController.new.expire_fragment(...).
I did get this working. The only slight difference in my setup is that the sweeper is part of a Rails engine; which accounts for slight differences (loading the sweeper file with a require in the engine's init instead of adding it to the load path in environment.rb, etc).
So, the sweeper is loaded in the init.rb of the engine like this:
require File.join(File.dirname(__FILE__), 'app', 'sweepers', cached_category_count_sweeper')
I called it a sweeper because it "sweeps" the cache, but I guess its just an observer on the model:
class CachedCategoryCountSweeper < ActiveRecord::Observer
observe CategoryFeature
def before_save(cf)
expire_cache(cf.category_id_was) if cf.category_id_changed?
end
def after_save(cf)
expire_cache(cf.category_id)
end
def after_destroy(cf)
expire_cache(cf.category_id)
end
def expire_cache(c)
ApplicationController.expire_page("/categories/#{c}/counts.xml") if !c.nil?
end
end
Frankly, I don't like having to hard-code the path, but I tried adding:
include ActionController:UrlWriter
and then using the path method, but it only worked for me in development. It didn't work in production, because my production server uses a relative url root (instead of virtual hosts) and the internal method "page_cache_path" would consistently get the file path wrong so it couldn't expire.
Since this is an observer, I added to the environment.rb:
config.active_record.observers = :cached_category_count_sweeper
Finally the controller that uses the cache (doesn't expire it, that is done through the model observer):
class CachedCategoryCountsController < ApplicationController
caches_page :index
# GET /cached_category_counts.xml
def index
...
end
end
Anyhow, hope this helps.
Andres Montano
I've been able to get it to work, by way of adding
ActionController::Base.expire_page(app.link_path(:md5 => #link.md5))
to the method in the Model itself that's updating the database. This feels somewhat hacky, though, and I'd love to know if anyone can explain why it's not working with the normal sweeper setup, and if there's a more elegant way to handle this.
That snippet of code (apart from customizations I put in for my own app) came from this post on ruby-forum.com.
I wrote a bit about this topic here: Rails Cache Sweeper Confusion. Would love to hear your opinions.
Based on #moiristo and #ZoogieZork 's answers, I am guessing this would work (untested).
class LinkSweeper < ActiveRecord::Observer
include ActionController::Caching::Pages
# or if you want to expire fragments
#include ActionController::Caching::Fragments
observe Link
def after_update(link)
expire_page( ... )
#expire_fragment( ... )
end
end