This is the code that I have
class ExampleTask
extend Resque::Plugins::ExponentialBackoff
#backoff_strategy = [0, 20, 3600]
#queue = :example_tasks
def self.perform
raise
end
end
I am running into a problem where, whenever I enqueue this task locally, Resque seems to retry the task immediately without respecting the backoff strategy. Has anyone ever experienced this problem before?
upgrading to 1.0.0 actually solves this problem.
For any future readers, the first integer in the array #backoff_strategy is how long Resque-Retry will wait before retrying the first time. From the github readme:
key: m = minutes, h = hours
no delay, 1m, 10m, 1h, 3h, 6h
#backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
#retry_delay_multiplicand_min = 1.0
#retry_delay_multiplicand_max = 1.0
The first delay will be 0 seconds, the 2nd will be 60 seconds, etc... Again, tweak to your own needs.
Related
So I'm building a website that calls a third-party API that can take from 20 seconds to 30 minutes to return a result. But I can't know this duration in advance so need to poll it frequently to check if the work is done (returns "COMPLETE" and the result) or not (returns "IN_PROGRESS"). Also, this API might be called many times from many users at the same time.
So I created a Sidekiq worker that checks the API every 5 seconds until it receives "COMPLETE", and only then it ends. But I've read that Sidekiq should only be doing short-lived jobs, and I'm struggling to get my head around how should I do it. Also I've been trying to search for an answer but I suspect I don't know the words to find what I'm looking for.
I'm sure there is a way I can tell my workers to call the API once, and if the result is "IN_PROGRESS" end but make sure another worker will do another API call to check, and so on and so on until the result is "COMPLETE".
Also, I guess this is also handy to better distribute the load in case many users demand the use of said API, because fewer workers can do more of this short-lived jobs.
This is my worker, which I hope clarifies what I'm doing right now:
class ThingProgressWorker
include Sidekiq::Worker
def perform(id)
#thing = Thing.find(id)
#thing_api_call = ThingAPICall.new // This uses the ruby library of the API
completed = false
while completed == false
result = #thing_api_call.get_result( { thing_job_name: #thing.job_name })
if !result.include? "COMPLETED"
completed = false
sleep 5
else
completed = true
#thing.status = "completed"
#thing.save
break
end
end
end
end
So if the API takes ten minutes to go from "IN_PROGRESS" to "COMPLETED" this worker will be busy for that long, which I recon is not advised at all.
I've been thinking about this for some hours now and can't think of how should I do to make each API call its own job without having a worker busy until the API is done.
The only solution I've thought so far is having a master worker that calls another worker for each API call, but then I'll still have a worker busy for as long as the API takes to send the result.
I'd appreciate any help or directions!
Thanks in advance
Try to call the worker with a delay. for example:
class ThingProgressWorker
include Sidekiq::Worker
def perform(id)
#thing = Thing.find(id)
#thing_api_call = ThingAPICall.new // This uses the ruby library of the API
result = #thing_api_call.get_result( { thing_job_name: #thing.job_name })
if !result.include? "COMPLETED"
ThingProgressWorker.perform_in(1.minute, id)
else
completed = true
#thing.status = "completed”
#thing.save
end
end
end
This will add the worker to the queue but will not run it immediately but in the time you specify.
I am trying to build quizup like app and want to send broadcast every 10 second with a random question for 2 minutes. How do I do that using rails ? I am using action cable for sending broadcast. I can use rufus-scheduler for running an action every few seconds but I am not sure if it make sense to use it for my use case .
Simplest solution would be to fork a new thread:
Thread.new do
duration = 2.minutes
interval = 10.seconds
number_of_questions_left = duration.seconds / interval.seconds
while(number_of_questions_left > 0) do
ActionCable.server.broadcast(
"some_broadcast_id", { random_question: 'How are you doing?' }
)
number_of_questions_left -= 1
sleep(interval)
end
end
Notes:
This is only a simple solution of which you are actually ending up more than 2.minutes of total run time, because each loop actually ends up sleeping very slightly more than 10 seconds. If this discrepancy is not important, then the solution above would be already sufficient.
Also, this kind-of-scheduler only persists in memory, as opposed to a dedicated background worker like sidekiq. So, if the rails process gets terminated, then all currently running "looping" code will also be terminated as well, which you might intentionally want or not want.
If using rufus-scheduler:
number_of_questions_left = 12
scheduler = Rufus::Scheduler.new
# `first_in` is set so that first-time job runs immediately
scheduler.every '10s', first_in: 0.1 do |job|
ActionCable.server.broadcast(
"some_broadcast_id", { random_question: 'How are you doing?' }
)
number_of_questions_left -= 1
job.unschedule if number_of_questions_left == 0
end
I am trying to scrape some tweets using Tweepy, but the connection crash after few hundreds requests with the following error:
tweepy.error.TweepError:
Failed to send request: ('Connection aborted.', error("(104, 'ECONNRESET')",))
My code is like this:
for status in tweepy.Cursor(api.search,
q="",
count=100,
include_entities=True,
monitor_rate_limit=True,
wait_on_rate_limit=True,
wait_on_rate_limit_notify = True,
retry_count = 5, #retry 5 times
retry_delay = 5, #seconds to wait for retry
geocode ="34.0207489,-118.6926066,100mi", # los angeles
until=until_date,
lang="en").items():
try:
towrite = json.dumps(status._json)
output.write(towrite + "\n")
except Exception, e:
log.error(e)
c+=1
if c % 10000 == 0: # 100 requests, sleep
time.sleep(900) # sleep 15 min
I can capture the error with try/except, but I am not able to restart the cursor from the point where it crashed.
Does anyone know how to solve this error, or restart the cursor from last known status?
Thanks!
Tweepy documentation says the Requests / 15-min window (user auth) are 180, but apparently sleeping for too long affect the connection reliability (after some requests) so if you run a request every 5 sec everything seems to work just fine:
for status in tweepy.Cursor(api.search,
q="",
count=100,
include_entities=True,
monitor_rate_limit=True,
wait_on_rate_limit=True,
wait_on_rate_limit_notify = True,
retry_count = 5, #retry 5 times
retry_delay = 5, #seconds to wait for retry
geocode ="34.0207489,-118.6926066,100mi", # los angeles
until=until_date,
lang="en").items():
try:
towrite = json.dumps(status._json)
output.write(towrite + "\n")
except Exception, e:
log.error(e)
c+=1
if c % 100 == 0: # first request completed, sleep 5 sec
time.sleep(5)
It seems to me that the tweepy call should be inside the try block. Also, you have arguments in api.search that are not in the Tweepy API (http://docs.tweepy.org/en/v3.5.0/api.html#help-methods). Anyway, this worked for me:
backoff_counter = 1
while True:
try:
for my_item in tweepy.Cursor(api.search, q="test").items():
# do something with my_item
break
except tweepy.TweepError as e:
print(e.reason)
sleep(60*backoff_counter)
backoff_counter += 1
continue
Basically, when you get the error you sleep for a while, and then try again. I used an incremental backoff to make sure that the sleeping time was enough for re-establishing the connection.
I am using beaneater/beanstalk in my app for maintaining the job queues.
https://github.com/nesquena/backburner
My global config file for backburner look like -
Backburner.configure do |config|
config.beanstalk_url = ["beanstalk://#{CONFIG['beanstalk']['host']}:#{CONFIG['beanstalk']['port']}"]
config.tube_namespace = CONFIG['beanstalk']['tube_name']
config.on_error = lambda { |e| puts e }
config.max_job_retries = 5 # default 0 retries
config.retry_delay = 30 # default 5 seconds
config.default_priority = 65536
config.respond_timeout = 120
config.default_worker = Backburner::Workers::Simple
config.logger = Logger.new('log/backburner.log')
config.priority_labels = { :custom => 50, :useless => 1000 }
config.reserve_timeout = nil
end
I want to set different retry limit and retry delay for different jobs.
I was looking at rubydoc for corresponding variable/function. As per this rubydoc link, I tried configuring retry_limit locally in a worker as:
One specific worker look like -
class AbcJob
include Backburner::Queue
queue "abc_job" # defaults to 'backburner-jobs' tube
queue_priority 10 # most urgent priority is 0
queue_respond_timeout 300 # number of seconds before job times out
queue_retry_limit 2
def self.perform(abc_id)
.....Task to be done.....
end
end
However, it is still picking up the retry limit from global config file and retrying it 5 times instead of 2. Any thing that I am missing here?
How can I over write retry limit and retry delay locally?
I could not find right way to do it but I found a solution.
I am putting entire body of perform in begin-rescue block and in case of failure I am re-enqueing it with custom delay. Also, to keep the track of number of retries I made it an argument which I am enqueueing.
class AbcJob
include Backburner::Queue
queue "abc_job" # defaults to 'backburner-jobs' tube
queue_priority 10 # most urgent priority is 0
queue_respond_timeout 300 # number of seconds before job times out
def self.perform(abc_id, attempt = 1)
begin
.....Task to be done.....
rescue StandardError => e
# Any notification method so that you can know about failure reason and fix it before next retry
# I am using NotificationMailer with e.message as body to debug
# Any function you want your retry delay to be, I am using quadratic
delay = attempt * attempt
if attempt + 1 < GlobalConstant::MaxRetryCount
Backburner::Worker.enqueue(AbcJob, [abc_id, attempt + 1], delay: delay.minute)
else
raise # if you want your jobs to be buried eventually
end
end
end
I have kept the default value of attempt to 1 so that magic nuber 1 do not appear in code which might raise question about why are we passing a constant. For enqueueing from other places in code you can use simple enqueue
Backburner::Worker.enqueue(AbcJob, abc_id)
I am looking for a better way to retrieve a systems uptime. My current method works, but I feel like things can be done better.
def uptime_range
days_up = `uptime | awk {'print$3'}`.chomp.to_i
hours_up = `uptime | awk {'print$5'}`.delete(',').chomp
seconds_up = time_to_seconds(days_up,hours_up)
started = Time.now - seconds_up
"#{started.strftime('%Y.%m.%d.%H.%M')}-#{Time.now.strftime('%Y.%m.%d.%H.%M')}"
end
On linux systems you can open /proc/uptime and read seconds since the machine booted.
def started
Time.now - IO.read('/proc/uptime').split[0].to_f
end
Edit: changed to_i to to_f. With to_i, the value of started when displayed as a string or integer is more likely to vary, especially if the boot time was close to the middle of a second rather than at the beginning or end of a second.
You could check the sysinfo gem. It should give system-independent access to the uptime data.
A rubygem called linux_stat can do that pretty well:
require 'linux_stat'
LinuxStat::OS.uptime
=> {:hour=>40, :minute=>46, :second=>19.75}
Code copied from here.
But if you don't feel like using another gem for this trivial task, you can do this:
uptime = IO.read('/proc/uptime').to_f
uptime_i = uptime.to_i
puts({
hour: uptime_i / 3600,
minute: uptime_i % 3600 / 60,
second: uptime.%(3600).%(60).round(2)
})
Which prints:
{:hour=>0, :minute=>39, :second=>54.89}