I can't figure out what's going on with sidekiq. I could've swore this worked yesterday, but I must have been dreaming.
Here's my worker class:
class TagPostWorker
include Sidekiq::Worker
sidekiq_options queue: "tag"
sidekiq_options retry: false
def perform(options = {})
current_user = User.find(options[:user_id])
end
end
I've tried running this command on my show method in the Post:
TagPostWorker.perform_async({:user_id => current_user.id})
But I get this error:
2013-08-17T22:45:45Z 4029 TID-ors6jfr54 TagPostWorker JID-ae203958bb3bcee01c8f83ef INFO: start
2013-08-17T22:45:45Z 4029 TID-ors6jfr54 TagPostWorker JID-ae203958bb3bcee01c8f83ef INFO: fail: 0.003 sec
2013-08-17T22:45:45Z 4029 TID-ors6jfr54 WARN: {"retry"=>false, "queue"=>"tag", "class"=>"TagPostWorker", "args"=>[{"user_id"=>7}], "jid"=>"ae203958bb3bcee01c8f83ef", "enqueued_at"=>1376779545.9099338}
2013-08-17T22:45:45Z 4029 TID-ors6jfr54 WARN: Couldn't find Post without an ID
I don't understand how sidekiq could even be attempting to a Post since I'm not even calling it in the perform method. Any ideas what could be going on?
Any help is appreciated.
You may want to take a look at this blog post.
It sounds ridiculous, but Sidekiq is so fast that it can run your
worker before your model even finishes saving.
Also strange, I could've sworn I restarted Rails/sidekiq.
But I renamed the worker to TagWorker, restarted Rails/sidekiq and it started working again!
Related
I have 2 Sidekiq workers:
Foo:
# frozen_string_literal: true
class FooWorker
include Sidekiq::Worker
sidekiq_options queue: :foo
def perform
loop do
File.open(File.join(Rails.root, 'foo.txt'), 'w') { |file| file.write('FOO') }
end
end
end
Bar:
# frozen_string_literal: true
class BarWorker
include Sidekiq::Worker
sidekiq_options queue: :bar
def perform
loop do
File.open(File.join(Rails.root, 'bar.txt'), 'w') { |file| file.write('BAR') }
end
end
end
Which has pretty the same functionality, both runs on different queues and the yaml file looks like this:
---
:queues:
- foo
- bar
development:
:concurrency: 5
The problem is, even both are running and showing in the Busy page of Sidekiq UI, only one of them will actually create a file and put contents in. Shouldn't Sidekiq be multi-threaded?
Update:
this happens only on my machine
i created a new project with rails new and same
i cloned a colleague project and ran his sidekiq and is working!!!
i used his sidekiq version, not working!
New Update:
this happens also on my colleague machine if he clone my project
if I run 2 jobs with a finite loop ( like 10 times do something with a sleep), first job will be executed and then the second, but after the second finishes and start again both will work on same time as expected -- everyone who cloned the project from: github.com/ArayB/sidekiq-test encountered the problem.
It's not an issue with Sidekiq. It's an issue somewhere in Ruby/MRI/Thread/GIL. Google for more info, but my understanding is that sometimes threads aren't real threads (see "green threads") so really just simulate threading. The important thing is that only one thread can execute at a time.
It's interesting that with only two threads the system isn't giving time to the second thread. No idea why, but it must realize it's mistake when you run it again.
Interestingly if you run your same app but instead fire off 10 TestWorkers (and tweak the output so you can tell the difference) sidekiq will run all 10 "at once".
10.times {|i| TestWorker.perform_async(i) }
Here is the tweaked worker. Be sure to flush the output cause that can also cause issues with threading and TTY buffering not reflecting reality.
class TestWorker
include Sidekiq::Worker
def perform(n)
10.times do |i|
puts "#{n} - #{i} - #{Time.current}"
$stdout.flush
sleep 1
end
end
end
Some interesting links:
https://en.wikipedia.org/wiki/Green_threads
http://ruby-doc.org/core-2.4.1/Thread.html#method-c-pass
https://github.com/ruby/ruby/blob/v2_4_1/thread.c
Does ruby have real multithreading?
I have a sidekiq worker which will request 3rd party api(Mailchimp) and got some response. Sometimes it will response an error message which the api gem will raise an error.
However, those Errors are normal and no need to retry. So I would like Sidekiq prevent retry when those Errors raised.
I have tried a simply rescue, but it won't prevent the sidekiq capture the error raised.
def preform(id)
UpdateMailchimpService.new.(id)
rescue
Mailchimp::ListInvalidBounceMemberError
end
Any way to do this? Thanks
Update
Finally found that my problem was caused by the broken of our deploy tool(deployment failed but not realised). Actually, the Sidekiq will ignore any rescued error/exception and they are not be retried and reported to Bugsnag.
In Bugsnag's documentation, it clearly said:
Bugsnag should be installed and configured, and any unhandled exceptions will be automatically detected and should appear in your Bugsnag dashboard.
This post on github didn't have an clear explanation so that's why I am confused by this question.
Your assumption/example is incorrect. Do the normal Ruby thing: rescue the error and ignore it.
def perform(id)
UpdateMailchimpService.new.(id)
rescue NormalError
# job will succeed normally and Sidekiq won't retry it.
end
Use retry: false advanced option:
class UpdateMailchimpWorker
include Sidekiq::Worker
sidekiq_options retry: false # ⇐ HERE
def perform(id)
UpdateMailchimpService.new.(id)
end
end
I found these logs here that I'm trying to duplicate in my code, specifically the start time and done times:
PID-- ThreadID----- LLvl YourKlass- JobID--------
TID-oveahmcxw INFO: HardWorker JID-oveaivtrg start
TID-oveajt7ro INFO: HardWorker JID-oveaish94 start
TID-oveahmcxw INFO: HardWorker JID-oveaivtrg done: 10.003 sec
TID-oveajt7ro INFO: HardWorker JID-oveaish94 done: 10.002 sec
Does anyone know how to access these values? I know there is a jid that gets popluated (which I'm using), but I can't for the life of me find any documentation on where start and done come from.
You can add logging by using a middleware:
class JobMiddleware
def call(worker, msg, queue)
Rails.logger.info "Job #{worker.jid} stated at #{Time.now}"
yield
Rails.logger.info "Job #{worker.jid} ended at #{Time.now}"
end
end
Add this to config/initializers/sidekiq.rb and restart your sidekiq workers.
Click here for more info about sidekiq middlewares
I applied "mongoid-locker" gem on my app but during concurrent request it got failed and got error "LockError: could not get lock".So can anyone help me out.
By default, with_lock does not wait for other locks to complete, so if you actually have a concurrent access, you will get the LockError raised if you don't tell it to wait.
Try it like so:
object = Object.first
object.with_lock wait:true do
object.foo = "bar"
object.save!
end
I use a Sidekiq queue to process communications with an unreliable, 3rd party API. Since this API is often down for a couple minutes at a time and then back up again, Sidekiq has been handy. When a connection issue happens, an error is raised and Sidekiq throws the job back in the queue to be retried again later, after some time has passed.
I use NewRelic to not only help debug crashes, but also for monitoring. My problem is that this current methodology above creates errors in NewRelic. If the 3rd party API is down for more than a couple of minutes, the error count accumulates enough to cause notifications to send out through NewRelic.
What I'd like to do is only raise an error from my worker when a certain number of retries have occurred for a job. I'm using sidekiq_retries_exhausted to do this. My problem is that I'm not quite sure how to put jobs back in the queue after they have an error without raising an error.
Does Sidekiq provide any facilities to return a job to a queue, increment the number of retries for the job, and have it sit there until it's due to run again, as if an exception was raised in the worker class?
You raise a specific error and tell the error service to ignore errors of that type. For NewRelic:
https://docs.newrelic.com/docs/agents/ruby-agent/installation-configuration/ruby-agent-configuration#error_collector.ignore_errors
Here is what I did to keep intentional retry errors out of AirBrake:
class TaskWorker
include Sidekiq::Worker
class RetryNotAnError < RuntimeError
end
def perform task_id
task = Task.find(task_id)
task.do_cool_stuff
if task.finished?
#log.debug "Task #{task_id} was successful."
return false
else
#log.debug "Task #{task_id} will try again later."
raise RetryNotAnError, task_id
end
end
end
Tell Airbrake to ignore it:
Airbrake.configure do |config|
config.ignore << 'RetryNotAnError'
end
It's good to make your exception name OBVIOUSLY not an error (e.g. RetryLaterNotAnError), as it will still show up in logs and such, and you don't want to freak people out when they see a bunch of them.
ps. That said, I would really like to see Sidekiq to provide an explicit, errorless retry mechanism.
If using Sidekiq Enterprise, one other option might be to utilize the optional set of additional error types that will then get treated as Sidekiq::Limiter::OverLimit violations.
For my purposes, I've used a new error class and then added it to the list in the config. Here are the notes from the sidekiq-ent code (not in the public sidekiq repo) on how to modify your config file:
# An optional set of additional error types which would be
# treated as a rate limit violation, so the job would automatically
# be rescheduled as with Sidekiq::Limiter::OverLimit.
#
# Sidekiq::Limiter.errors << MyApp::TooMuch
# Sidekiq::Limiter.errors = [Foo::Error, MyApp::Limited]
Inside the specific job you can specify the max_retries, or it will default to 20:
sidekiq_options max_limiter_retries: 10
Inside the job, I'll rescue the "expected" intermittent error that I'd rather not ignore completely and then raise the error I've added to the list, something like this:
rescue RestClient::RequestTimeout => e
raise SidekiqSoftRetry.new(e.inspect)
end
Here's what that looks like in my initialization file-- and Mike Perham was kind enough to respond with the option to update the global retry limit.
class SidekiqSoftRetry < RuntimeError
end
Sidekiq::Limiter::DEFAULT_OPTIONS[:reschedule] = 10
Sidekiq::Limiter.configure do |config|
config.errors.concat(
[
SidekiqSoftRetry,
]
)
end