I've been reading django-celery to try to find a native way of cleaning up celery_taskmeta, so this table won't grow up too much when in production environment.
Can'f find nothing. Should i do a command to deal with this?
I'm not using django-celery so my answer could be incomplete.
There is a method in the Backend that allow you to force cleanup of celery_taskmeta : in celery.backends.base.BaseBackend, you can find method cleanup()
So, when I need to force cleanup I have 3 ways :
using the AsyncResult on producer side
using the task on the consumer side
celerybeat task
on producer side
# Launch a task
my_task = add_a_task() # use the Task.delay() method for example
# my_task is a celery.result.AsyncResult instance
# Retrieve backend and call cleanup
my_task.backend.cleanup()
on the consumer side
from celery import Task
from .celery_app import my_celery_app
# Class for a task
#my_celery_app.task()
class CopyTask(Task):
# A Task instance got a backend property
def run(self,**kwargs):
""" Running method of the task """
# Do something
pass
def force_cleanup(self):
""" Force cleanup. """
self.backend.cleanup()
Note that calling a cleanup in the task itself seems quiet awfull.
with celerybeat task
A third option, is using a celerybeat task that cleanup backend celery_taskmeta.
This is the one I probably choose.
celery beat is basically a scheduler and you can run the task you want.
So you can have a dedicated task, like this one :
from celery import Task
from .celery_app import my_celery_app
# Class for a task
#my_celery_app.task()
class CleanupTask(Task):
# A Task instance got a backend property
def run(self,**kwargs):
""" Running method of the task """
# Cleanup here
self.backend.cleanup()
Setting a celery beat, like a crontab job is well documented
I have a cronjob running a Django command that executes this SQL statement:
"truncate table celery_taskmeta"
deleted 50.000 records in 0.47ms.
Related
I'm trying to send email campaigns in a rails app with the Heya gem and GoodJob. The example in the Heya readme as well as the Heya example app uses Sidekiq as the Active Job backend.
I'm confused about how to actually send the Heya campaigns with GoodJob.
The docs for Heya show this example of starting Sidekick: bundle exec sidekiq -q default -q heya
I assume that there is a Job queue somewhere in the gem called "Heya", but I can't find this in the source code. Do I need to create one?
Do I need to create a job that runs the Heya scheduler? While the example app uses Sidekiq, I also don't see any custom jobs in that app.
I have the following setup for GoodJob and it appears to be running fine with good_job start which should run all of the jobs and queues, but I've also tried good_job start --queues=heya,default.
Here is the relevant code:
Profile.dev
web: bin/rails server -p 3000
css: bin/rails tailwindcss:watch
worker: bundle exec good_job start
config/initializers/heya.rb
Heya.configure do |config|
config.user_type = "User"
config.campaigns.priority = [
"WelcomeCampaign",
]
end
app/jobs/application_job.rb
class ApplicationJob < ActiveJob::Base
# Automatically retry jobs that encountered a deadlock
# retry_on ActiveRecord::Deadlocked
# Most jobs are safe to ignore if the underlying records are no longer available
# discard_on ActiveJob::DeserializationError
end
app/campaigns/application_campaign.rb
class ApplicationCampaign < Heya::Campaigns::Base
segment :email_subscriber?
default from: "#{I18n.t('settings.site_name')} <#{I18n.t('settings.newsletter_email')}>"
end
app/campaigns/welcome_campaign.rb
class WelcomeCampaign < ApplicationCampaign
default wait: 5.minutes,
layout: "newsletter"
step :intro, wait: 0.minutes,
subject: "Welcome to #{I18n.t('settings.site_name')}"
end
I also have a layout and views for the campaign similar to the Heya example app, and I'm using Mailcatcher to see if any email is being sent.
What am I missing to send these emails with Heya and GoodJob?
Note that I'm subscribing the users on signups like this:
class User < ApplicationRecord
after_create_commit :add_user_to_newsletters
private
def add_user_to_newsletters
WelcomeCampaign.add(self)
EvergreenCampaign.add(self)
self.update(email_subscriber: true)
end
end
And the default segment in campaigns/application_campaign.rb is segment :email_subscriber?
If I run User.last.email_subscriber? in the console to check this it returns true.
I feel like I'm missing something about how Heya connects to Active Job that is not obvious in the Heya docs.
Also, not sure if this is related, but I added this to config/puma.rb
# https://github.com/bensheldon/good_job#execute-jobs-async--in-process
before_fork do
GoodJob.shutdown
end
on_worker_boot do
GoodJob.restart
end
on_worker_shutdown do
GoodJob.shutdown
end
MAIN_PID = Process.pid
at_exit do
GoodJob.shutdown if Process.pid == MAIN_PID
end
preload_app!
Are you running the heya scheduler periodically? $ rails heya:scheduler
It looks like you could create your own background job to be run using GoodJob Cron, by executing Heya::Campaigns::Scheduler.new.run to run the scheduler and enqueue the emails.
Reading the "Running the Scheduler" part of the README explains what's happening:
To start queuing emails, run the scheduler task periodically:
rails heya:scheduler
Heya uses ActiveJob to send emails in the background. Make sure your
ActiveJob backend is configured to process the heya queue. By default, GoodJob runs from all queues "*".
You can change Heya's default queue using the queue option:
# app/campaigns/application_campaign.rb
class ApplicationCampaign < Heya::Campaigns::Base
default queue: "custom"
end
I am trying to add couple of scheduled workers to my rails application. These workers will be crawling different sites in given intervals.
I want to test these workers but not able to do it. I am starting redis and my application. What should I do to see whether my scheduled jobs are working or not?
Here is my crawler class:
class AyedasCrawler
include Sidekiq::Worker
and my sidekiq.yml is:
:schedule:
ayedas_crawler:
cron: '0 * * * * *' # Runs once per minute
class: AyedasCrawler
start the sidekiq worker and the scheduler processes by running
bundle exec sidekiq or sidekiq from your app root in the command-line.
sidekiq-scheduler provides an extension to the Sidekiq web interface that adds a Recurring Jobs page.
There are two ways to do this:
In your routes.rb file, just below the require 'sidekiq/web', add require 'sidekiq-scheduler/web'
In your config.ru, just below the require 'sidekiq/web', add
require 'sidekiq-scheduler/web'
run Sidekiq::Web
On the browser, goto ==> http://localhost:{port}/sidekiq/recurring-jobs. where {port} is the port your application is running in.
You will see the list of scheduled jobs for your application and some other details about it.
Read more in the official documentation
You need to run Sidekiq process as well.
bundle exec sidekiq
It will start both worker/s and the scheduler
If you wish to test it using rspec, you can to the following:
it 'spawns scheduled workers' do
Sidekiq::Cron::Job.load_from_hash YAML.load_file('config/sidekiq.yml')[: schedule]
Sidekiq::Cron::Job.all.each(&:enque!)
expect(AyedasCrawler.jobs.size).to be(1)
end
It loads the YAML configuration, enqueues all the jobs, and asserts if the job has been enqued.
Using this method you can validate if your schedule YAML is correct. It will NOT test CRON syntax and scheduled intervals.
I'm also using https://github.com/philostler/rspec-sidekiq to allow sidekiq testing without jobs actually being executed.
I'm trying to use cron in my application to send mails every week but I think it doesn't work on Windows.
Does anybody knows any equivalent to cron solution that works on Windows?
Windows equivalent of Unix's cron is a "Task Scheduler". You can configure your periodical task there.
Purely Ruby solution
If you want a purely Ruby solution look into:
rufus-scheduler - it's Windows cron gem.
crono - it's a in-Rails cron scheduler, so it should work anywhere.
Web services - there are plenty of free online services that would make a request to a given URL in specific time periods. This is basically a poor man's cronjob.
I recommend taking a look at Resque and the extension Resque-scheduler gems. You will need to have a resque scheduler process running with bundle exec rake resque:scheduler and at least one worker process running with QUEUE=* bundle exec rake resque:work.
If you want these services to run in the background as a windows service, you can do it with srvany.exe as described in this SO question.
The above assumes you are ok with installing Redis - a key-value store that is very popular among the Rails community as it can be easily used to support other Rails components such as caching and ActionCable, and it is awesome by itself for many multi-process use cases.
Resque is a queue system on top of Redis that allows you to define jobs that can be executed asynchronously in the background. When you run QUEUE=* bundle exec rake resque:work, a worker process runs constantly and polls the queue. Once a job is enqueued, an available worker pops it from the queue and starts working on it. This architecture is quite scalable, as you can have multiple workers listening to the queues if you'd like.
To define a job, you do this:
class MyWeeklyEmailSenderJob
def self.perform
# Your code to send weekly emails
end
end
While you can enqueue this job to the queue yourself from anywhere (e.g. from a controller as a response to an action), in your case you want it to automatically be placed into the queue once a week. This is what Resque-scheduler is for. It allows you to configure a file such as app/config/resque_schedule.yml in which you can define which jobs should be enqueued in which time interval. For example:
send_weekly_emails:
cron: 0 8 * * Mon
class: MyWeeklyEmailSenderJob
queue: email_sender_queue
description: "Send weekly emails"
Remember that a scheduling process has to run in order for this to work with bundle exec rake resque:scheduler.
thanks guys , actually i tried rufus scheduler gem and it worked for me , i guess it's the best and easier solution
Task:
Delete an item in Async manner [HomeWork]
I have already configured Active jobs with delayed_jobs in my rails application but I am still confused about performing Async task in rails project.
Let's take an example:
I have some item to delete from the database but I want to do it in Async manner. I also read about perform_later or perform_now method in delayed_job blogs. Here is my code which is working fine:
Controller class
def destroy
PostJob.perform_now(params[:id])
respond_to do |format|
format.xml { head :ok }
format.js { render 'posts.js.erb' }
end
end
Job class
class PostJob < ActiveJob::Base
queue_as :default
def perform(id)
#post = Post.find(id)
#post.destroy
end
end
According to official doc of delayed_jobs I can add handle_asynchronously in the end of method to run in async manner. How can I implement in this case?
My Question:
When I am looking at destroy method it is not deleting the element in Async way. However every steps written in destroy method is in Synchronous. Am I wrong?
If it's not then How can I implement destroy method to delete post in async manner?
Backgrounding task and cron job are same thing?
Edit -1
After giving suggestion by A Fader Darkly, I changed perform_now to perform_later which is working perfectly for Async process but it is not deleting the entry from table (code is fine because it works when i user perform_now).
Also when I am running job manually by following command, Everything works fine:
rake jobs:work
Is there any way to execute delay_job task as soon as the queue get some new data?
If you change your destroy method to call:
PostJob.perform_later(params[:id])
it should happen asynchronously. If not, you have some more set-up to do.
For your questions:
Yes you are right, but what you say is a tautology. Everything in that method is synchronous - the job queue isn't used because of the perform_now. Thus destroy isn't deleting in an async way.
See above.
Cron jobs work on the operating system level and are scheduled regularly for particular times. You could have a cron job working every minute, for example, or every day, or week (on a particular day at a particular time). They run from a schedule file called a crontab.
'Backgrounding' a task simply stops it from taking over the IO of your terminal session. So you can carry on using the terminal while the process runs in the background. Generally this is done on an ad-hoc basis, so you don't have to wait for a heavy operation to complete before going on to do different tasks.
EDIT
Based on edits to the question, it sounds like Delayed Job daemon needs to be started. From the instructions:
Note: For Rails 4 replace script/delayed_job with bin/delayed_job
When running a queue locally, omit the 'RAILS_ENV=production' part of commands.
Running Jobs
script/delayed_job can be used to manage a background process which will start working off jobs.
To do so, add gem "daemons" to your Gemfile and make sure you've run rails generate delayed_job.
You can then do the following:
RAILS_ENV=production script/delayed_job start
RAILS_ENV=production script/delayed_job stop
Runs two workers in separate processes.
RAILS_ENV=production script/delayed_job -n 2 start
RAILS_ENV=production script/delayed_job stop
Set the --queue or --queues option to work from a particular queue.
RAILS_ENV=production script/delayed_job --queue=tracking start
RAILS_ENV=production script/delayed_job --queues=mailers,tasks start
Use the --pool option to specify a worker pool. You can use this option multiple times to start different numbers of workers for different queues.
The following command will start 1 worker for the tracking queue,
2 workers for the mailers and tasks queues, and 2 workers for any jobs:
RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start
Runs all available jobs and then exits
RAILS_ENV=production script/delayed_job start --exit-on-complete
or to run in the foreground
RAILS_ENV=production script/delayed_job run --exit-on-complete
I want to start workes for the job directly after some certain method. So, I start the application with usual rails s. Upload some stuff, so the create method is invoked. After create method the :perform_analysis -method is delayed. Some data is inserted into delayed_jobs table. Normally I start the workers to work typing "script/delayed_job start" in the command line. But I would like to start the workers work automatically, so I will type nothing.
model:
after_create :perform_analysis
def perform_analysis
bla
end
handle_asynchronously :perform_analysis, :run_at => Proc.new { 5.minutes.from_now }
So, I run an application with rails s. I log in in my wep-page. Upload some files, after 5 min the jobs are delayed. Then the worker should start to work.
I have found this page that does almost what I want but somehow the workers do not start at all. So the schedule.rb is not run. Should I do something more that is not told on that webpage?
Is there any other possibility do it?
I recommend you take a look at Foreman (http://ddollar.github.com/foreman/) and have your procfile declare a worker process:
web: bundle exec rails s
worker: bundle exec rake jobs:work
This way, a single command foreman start will start both the server and worker. The output will be presented in the same window for both.