I am building an application where at some point I need to sync a bunch of data from fb with my database, so I am (attemtping) to use Delayed Job to push this into the background. Here is what part of my Delayed Job class looks like.
class FbSyncJob < Struct.new(:user_id)
require 'RsvpHelper'
def perform
user = User.find(user_id)
FbSyncJob.sync_user(user)
end
def FbSyncJob.sync_user(user)
friends = HTTParty.get(
"https://graph.facebook.com/me/friends?access_token=#{user.fb['token']}"
)
friends_list = friends["data"].map { |friend| friend["id"] }
user.fb["friends"] = friends_list
user.fb["sync"]["friends"] = Time.now
user.save!
FbSyncJob.friend_crawl(user)
end
end
With the RsvpHelper class living in lib/RsvpHelper.rb. So at some point in my application I call Delayed::Job.enqueue(FbSyncJob.new(user.id)) with a known valid user. The worker I set up even tells me that the job has been completed successfully:
1 jobs processed at 37.1777 j/s, 0 failed
However when I check the user in the database he has not had his friends list updated. Am I doing something wrong or what? Thanks so much for the help this has been driving me crazy.
Delayed::Job.enqueue will put a record in the delayed job table, but you need to run a seperate process to execute the job code (perform method)
typically in development this would be bundle exec rake jobs:work (NOTE: you must restart this rake task anytime you make code changes, it will not auto load changes)
see https://github.com/collectiveidea/delayed_job#running-jobs
I usually put the following into my delayed configuration while in development - this never puts a record in the delayed job table and runs all background code synchronously (in development) and by default rails will reload changes to your code
Delayed::Worker.delay_jobs = !(Rails.env.test? || Rails.env.development?)
https://github.com/collectiveidea/delayed_job#gory-details (see config/initializers/delayed_job_config.rb example section)
Related
I am trying to implement an API endpoint that would queue a request and return immediately.
I am using the gem https://rubygems.org/gems/activejob/versions/5.2.0 (I am on an old version for historical reasons).
I have defined a job that looks something like:
class Service::ExportBooks::Job < ActiveJob::Base
def perform
## ... Do the job
rescue StandardError
binding.pry
raise
end
end
In the controller, I am calling:
Service::ExportBooks::Job.perform_later
The job gets called synchronously and the controller gets even any errors raised by the job.
I've also tried other options such as:
job = Service::ExportBooks::Job.new
job.enqueue(wait: 5.seconds)
but it does the same, the job is not enqueued, is immediately executed.
UPDATE:
It looks like the method Resque.inline? returns true and so the execution is inline and not async. How can I make sure that it's async? I tried to set Resque.inline = false manually and the job was queued but it wasn't executed...
I have started a worker using the command:
QUEUE=* PIDFILE=./tmp/resque.pid bundle exec rake environment resque:work
Two things to do here.
Make sure Resque.inline = false.
Start up the resque workers in another process. See here.
This will get the job enqueued and run on the worker process.
I have some methods that works with API of third party app. To do it on button click is no problem, but it should be permanent process.
How to run them background? And how to pause the cycle for make some other works with same API and resume the cycle after the job is done.
Now I read about ActiveJob, but its has time dependences only...
UPDATE
I've tried to make it with whenever and sidekiq, task runs, but it do nothing. Where to look for logs I can't understand.
**schedule.rb**
every 1.minute do
runner "UpdateWorker.perform_async"
end
**update_worker.rb**
class UpdateWorker
include Sidekiq::Worker
include CommonMods
def perform
logger.info "Things are happening."
logger.debug "Here's some info: #{hash.inspect}"
myMethod
end
def myMethod
....
....
....
end
end
It's not exactly what I need, but better then nothing. Can somebody explain me with examples?
UPDATE 2 After manipulating with code it's absolutely necessary to restart sidekiq . With this problem is solved, but I'm not sure that this is the best way.
You can define a job which enqueues itself:
class MyJob < ActiveJob::Base
def perform(*args)
# Do something unless some flag is raised
ensure
self.class.set(wait: 1.hour).perform_later(*args)
end
end
There are several libraries to schedule jobs on a regular basis. For example you could use to sidekiq-cron to run a job every minute.
If you want to pause it for some time, you could set a flag somewhere (Redis/database/file) and skip execution as long it is detected.
On a somewhat related note: don't use sidetiq. It was really great but it's not maintained anymore and has incompatibilities to current Sidekiq versions.
Just enqueue next execution in ensure section after job completes after checking some flag that indicates that it should.
Also i recommend adding some delay there so that you don't end up with dead loop on some error inside job
I dont know ActiveJobs, but I can recommend the whenever gem to create cron (periodic background) jobs. Basically you end up writing a rake tasks. Like this:
desc 'send digest email'
task send_digest_email: :environment do
# ... set options if any
UserMailer.digest_email_update(options).deliver!
end
I never added a rake task to itself but for repeated processing you could do somehow like this (from answers to this specific question)
Rake::Task["send_digest_email"].execute
I'm trying to understand how to use whenever properly, or if I'm even using it for the right thing. I've created a job:
class ScheduleSendNotificationsJob < ActiveJob::Base
queue_as :notification_emails
def perform(*args)
user_ids = User.
joins(:receipts).
where(receipts: {is_read: false}).
select('DISTINCT users.id').
map(&:id)
user_ids.each do |user_id|
SendNotificationsJob.create(id: user_id)
Rails.logger.info "Scheduled a job to send notifications to user #{user_id}"
end
end
end
I'd like to perform this job ever day at a set time. The job polls to see if there are any outstanding notifications, batches them, and then sends them to users so that a user can get one email with a bunch of notifications instead of a bunch of emails with one notification per email. I tried doing this with Delayed Job, but it seems it's not designed to schedule something on a recurring basis. So now I'm trying to do it with the whenever gem, but I can't seem to figure out how to set it up properly.
This is what I have in my config/schedule.rb file:
every 1.minute do
runner ScheduleSendNotifications.create
end
When I run whenever -i in the console I get the following:
Lorenzs-MacBook-Pro:Heartbeat-pods lorenzsell$ whenever -i
config/schedule.rb:13:in `block in initialize': uninitialized constant Whenever::JobList::ScheduleSendNotifications (NameError)
What am I doing wrong here? Should I be using something else? I'm just learning ruby and rails so any help is very much appreciated. Thank you.
The whenever gem takes a string as the argument to the runner function. Whenever doesn't actually load the Rails environment so it doesn't know about your ScheduleSendNotifications class.
The code below should get the crontab set up correctly to run your job.
every 1.minute do
runner "ScheduleSendNotifications.create"
end
From your project directory run whenever -w to set up the crontab file. Run crontab -l to view the written crontab file. Every minute the system will execute your Rails runner. From there you may need to debug your ScheduleSendNotifications.create code if something isn't working.
I am facing a very interesting problem. I have tested the Delay job gem 4 times. I doubt it is the design problem of the gem or a bug. I use command rake jobs:work to create worker to do delayed job.
Once I create a LongTask record, i also make a delayed job which will change the attributeminutes_delayed to 2.
The gem works perfectly if I don't update the attributes. But once I edited the description, the gem will not work properly, which means it will not execute the delayed job, but the related delayed job record will be removed in the database.
Interesting final result:
It Seems to reference a object with attribute that is exactly the same, this picture was captured before the running time have gone over.
This one was captured after all tests have been gone though. You can see the delayed job record for test4 have been removed even this delayed job did't have any effect.
terminal results (only 2 jobs are executed)
[Worker(host:Jasonteki-MacBook-Air.local pid:1726)] Starting job worker
[Worker(host:Jasonteki-MacBook-Air.local pid:1726)] LongTask#set_delay_time_without_delay completed after 0.0343
[Worker(host:Jasonteki-MacBook-Air.local pid:1726)] 1 jobs processed at 16.6270 j/s, 0 failed ...
[Worker(host:Jasonteki-MacBook-Air.local pid:1726)] LongTask#set_delay_time_without_delay completed after 0.0105
[Worker(host:Jasonteki-MacBook-Air.local pid:1726)] 1 jobs processed at 51.4774 j/s, 0 failed ...
Code in model:
def set_delay_time(time)
self.minutes_delayed = time
# very important for this, otherwise cannot write the change into the database
self.save
end
handle_asynchronously :set_delay_time, :run_at => Proc.new { 2.minutes.from_now }
Code in controller:
def create
#long_task = LongTask.new(params[:long_task])
respond_to do |format|
if #long_task.save
#long_task.set_delay_time(2)
Without seeing your code, it's impossible to tell for sure, but it's likely that both of your delayed jobs are working on serialized copies of your object, rather than reloading them from the database.
I'm having big issues trying to get delayed_job working with Amazon S3 and Paperclip. There are a few posts around about how to do it, but for whatever reason it's simply not working for me. I've removed a couple of things to how others are doing it - originally I had a save(validations => false) in regenerate_styles, but that seemed to cause an infinite loop (due to the after save catch), and didn't seem to be necessary (since the URLs have been saved, just the images not uploaded). Here's the relevant code from my model file, submission.rb:
class Submission < ActiveRecord::Base
has_attached_file :photo ...
...
before_photo_post_process do |submission|
if photo_changed?
false
end
end
after_save do |submission|
if submission.photo_changed?
Delayed::Job.enqueue ImageJob.new(submission.id)
end
end
def regenerate_styles!
puts "Processing photo"
self.photo.reprocess!
end
def photo_changed?
self.photo_file_size_changed? ||
self.photo_file_name_changed? ||
self.photo_content_type_changed? ||
self.photo_updated_at_changed?
end
end
And my little ImageJob class that sites at the bottom of the submission.rb file:
class ImageJob < Struct.new(:submission_id)
def perform
Submission.find(self.submission_id).regenerate_styles!
end
end
As far as I can tell, the job itself gets created correctly (as I'm able to pull it out of the database via a query).
The problem arises when:
$ rake jobs:work
WARNING: Nokogiri was built against LibXML version 2.7.8, but has dynamically loaded 2.7.3
[Worker(host:Jarrod-Robins-MacBook.local pid:21738)] New Relic Ruby Agent Monitoring DJ worker host:MacBook.local pid:21738
[Worker(host:MacBook.local pid:21738)] Starting job worker
Processing photo
[Worker(host:MacBook.local pid:21738)] ImageJob completed after 9.5223
[Worker(host:MacBook.local pid:21738)] 1 jobs processed at 0.1045 j/s, 0 failed ...
The rake task then gets stuck and never exits, and the images themselves don't appear to have been reprocessed.
Any ideas?
EDIT: just another point; the same thing happens on heroku, not just locally.
Delayed job is capturing a stack trace for all failed jobs. It’s saved in the last_error column of the delayed_jobs table. Use a database gui too see whats going on.
If you should be using Collective Ideas fork with ActiveRecord as backend you can query the model as usual. To fetch an array of all stack traces for example do
Delayed::Job.where('failed_at IS NOT NULL').map(&:last_error)
By default failed jobs are deleted after 25 failed attempts. It may be that there are no jobs anymore. Prevent deletion for debugging purposes by setting
Delayed::Worker.destroy_failed_jobs = false
in your config/initializers/delayed_job_config.rb