I've got pretty old App where I have to create rake task to find all users over 18 and update flags from adult: false to adult: true. I'm wondering what I should use in a rather old version of Rails (I have Rails 5 and Ruby 2.4 on board) to keep the highest performance?
What I have for now is a sidekiq worker with, I think, some syntax error:
class MinorsWorker
include Sidekiq::Worker
def perform
adults = User.where(adults: false).where('date_of_birth >= 18, ?', ((Time.zone.now - date_of_birth.to_time) / 1.year.seconds))
adults.update(adult: true)
end
end
But this code gives me an error:
NameError: undefined local variable or method `date_of_birth' for main:Object
you can do the following. This would update all the matched records in 1 update statement.
If you are concern with db IO, you can batch it.
# in user.rb
class User
scope :adult, -> { where('date_of_birth <= ?', 18.years.ago) }
end
# in your worker file
class MinorsWorker
include Sidekiq::Worker
def perform
update_all
# for update in batches, use #update_all_in_batches
end
private
def update_all
User.adult.where(adult: false).update_all(adult: true)
end
def update_all_in_batches
User.adult.where(adult: false).in_batches(each: 1000) do |users|
users.update_all(adult: true)
sleep 2
end
end
end
Related
I'm using the Mongoid in a Rails project. To improve the performance of large queries, I'm using the includes method to eager load the relationships.
I would like to know if there is an easy way to count the real number of queries performed by a block of code so that I can check if my includes really reduced the number of DB accesses as expected. Something like:
# It will perform a large query to gather data from companies and their relationships
count = Mongoid.count_queries do
Company.to_csv
end
puts count # Number of DB access
I want to use this feature to add Rspec tests to prove that my query remains efficient after changes (e.g; when adding data from a new relationship). In python's Django framework, for instance, one may use the assertNumQueries method to this end.
Checking on rubygems.org didn't yield anything that seems to do what you want.
You might be better off looking into app performance tools like New Relic, Scout, or DataDog. You may be able to get some out of the gate benchmarking specs with
https://github.com/piotrmurach/rspec-benchmark
I just implemented this feature to count mongo queries in my rspec suite in a small module using mongo Command Monitoring.
It can be used like this:
expect { code }.to change { finds("users") }.by(3)
expect { code }.to change { updates("contents") }.by(1)
expect { code }.not_to change { inserts }
Or:
MongoSpy.flush
# ..code..
expect(MongoSpy.queries).to match(
"find" => { "users" => 1, "contents" => 1 },
"update" => { "users" => 1 }
)
Here is the Gist (ready to copy) for the last up-to-date version: https://gist.github.com/jarthod/ab712e8a31798799841c5677cea3d1a0
And here is the current version:
module MongoSpy
module Helpers
%w(find delete insert update).each do |op|
define_method(op.pluralize) { |ns = nil|
ns ? MongoSpy.queries[op][ns] : MongoSpy.queries[op].values.sum
}
end
end
class << self
def queries
#queries ||= Hash.new { |h, k| h[k] = Hash.new(0) }
end
def flush
#queries = nil
end
def started(event)
op = event.command.keys.first # find, update, delete, createIndexes, etc.
ns = event.command[op] # collection name
return unless ns.is_a?(String)
queries[op][ns] += 1
end
def succeeded(_); end
def failed(_); end
end
end
Mongo::Monitoring::Global.subscribe(Mongo::Monitoring::COMMAND, MongoSpy)
RSpec.configure do |config|
config.include MongoSpy::Helpers
end
What you're looking for is command monitoring. With Mongoid and the Ruby Driver, you can create a custom command monitoring class that you can use to subscribe to all commands made to the server.
I've adapted this from the Command Monitoring Guide for the Mongo Ruby Driver.
For this particular example, make sure that your Rails app has the log level set to debug. You can read more about the Rails logger here.
The first thing you want to do is define a subscriber class. This is the class that tells your application what to do when the Mongo::Client performs commands against the database. Here is the example class from the documentation:
class CommandLogSubscriber
include Mongo::Loggable
# called when a command is started
def started(event)
log_debug("#{prefix(event)} | STARTED | #{format_command(event.command)}")
end
# called when a command finishes successfully
def succeeded(event)
log_debug("#{prefix(event)} | SUCCEEDED | #{event.duration}s")
end
# called when a command terminates with a failure
def failed(event)
log_debug("#{prefix(event)} | FAILED | #{event.message} | #{event.duration}s")
end
private
def logger
Mongo::Logger.logger
end
def format_command(args)
begin
args.inspect
rescue Exception
'<Unable to inspect arguments>'
end
end
def format_message(message)
format("COMMAND | %s".freeze, message)
end
def prefix(event)
"#{event.address.to_s} | #{event.database_name}.#{event.command_name}"
end
end
(Make sure this class is auto-loaded in your Rails application.)
Next, you want to attach this subscriber to the client you use to perform commands.
subscriber = CommandLogSubscriber.new
Mongo::Monitoring::Global.subscribe(Mongo::Monitoring::COMMAND, subscriber)
# This is the name of the default client, but it's possible you've defined
# a client with a custom name in config/mongoid.yml
client = Mongoid::Clients.from_name('default')
client.subscribe( Mongo::Monitoring::COMMAND, subscriber)
Now, when Mongoid executes any commands against the database, those commands will be logged to your console.
# For example, if you have a model called Book
Book.create(title: "Narnia")
# => D, [2020-03-27T10:29:07.426209 #43656] DEBUG -- : COMMAND | localhost:27017 | mongoid_test_development.insert | STARTED | {"insert"=>"books", "ordered"=>true, "documents"=>[{"_id"=>BSON::ObjectId('5e7e0db3f8f498aa88b26e5d'), "title"=>"Narnia", "updated_at"=>2020-03-27 14:29:07.42239 UTC, "created_at"=>2020-03-27 14:29:07.42239 UTC}], "lsid"=>{"id"=><BSON::Binary:0x10600 type=uuid data=0xfff8a93b6c964acb...>}}
# => ...
You can modify the CommandLogSubscriber class to do something other than logging (such as incrementing a global counter).
I am trying to run a method that adds the response from an API call to Cache, I decided to use the simple_scheduler gem
Below are snippets of code that I am running
# update_cache_job.rb
class UpdateCacheJob < ActiveJob::Base
def perform
return QueuedJobs.new.update_cache
end
end
And
# simple_scheduler.yml
# Global configuration options. The `queue_ahead` and `tz` options can also be set on each task.
queue_ahead: 120 # Number of minutes to queue jobs into the future
queue_name: "default" # The Sidekiq queue name used by SimpleScheduler::FutureJob
tz: "nil" # The application time zone will be used by default if not set
# Runs once every 2 minutes
simple_task:
class: "UpdateCacheJob"
every: "2.minutes"
And the method I have scheduled to run every 2.minutes
class QueuedJobs
include VariableHelper
def initialize; end
def update_cache
#variables = obtain_software_development_table
# First refresh the project Reviews
puts 'Updating reviews...'
#records = Dashboard.new.obtain_projects_reviews.pluck(
obtain_project_reviews_student_variable,
obtain_project_reviews_id_variable,
'Project'
).map { |student, id, project| { 'Student': student, 'ID': id,
'Project': project } }
Rails.cache.write(
:reviews,
#records,
expires_in: 15.minutes
)
#grouped_reviews = Rails.cache.read(
:reviews
).group_by do |review|
review[:Student]&.first
end
puts 'reviews refreshed.'
# Then refresh the coding challenges submissions
puts "Updating challenges submissions.."
#all_required_submissions_columns = Dashboard.new.all_coding_challenges_submissions.all.map do |submission|
{
id: submission.id,
'Student': submission[obtain_coding_chall_subm_student_var],
'Challenge': submission[obtain_coding_chall_subm_challenge_var]
}
end
#all_grouped_submissions = #all_required_submissions_columns.group_by { |challenge| challenge[:Student]&.first }
Rails.cache.write(
:challenges_submissions,
#all_grouped_submissions,
expires_in: 15.minutes
)
puts "challenges submissions refreshed."
end
end
I have been able to reach these methods from the rails console but when ever I run rake simple_scheduler It just logs the first puts and sometimes it does nothing at all.
What do I need to do here?
I haven't had a lot of experience with deadlocking issues in the past, but the more I try to work with ActiveJob and concurrently processing those jobs, I'm running into this problem. An example of one Job that is creating it is shown below. The way it operates is I start ImportGameParticipationsJob and it queues up a bunch of CreateOrUpdateGameParticipationJobs.
When attempting to prevent my SQL Server from alerting me to a ton of deadlock errors, where is the cause likely happening below? Can I get a deadlock from simply selecting records to populate an object? Or can it really only happen when I'm attempting to save/update the record within my process_records method below when saving?
ImportGameParticipationsJob
class ImportGameParticipationsJob < ActiveJob::Base
queue_as :default
def perform(*args)
import_participations(args.first.presence)
end
def import_participations(*args)
games = Game.where(season: 2016)
games.each do |extract_record|
CreateOrUpdateGameParticipationJob.perform_later(extract_record.game_key)
end
end
end
CreateOrUpdateGameParticipationJob
class CreateOrUpdateGameParticipationJob < ActiveJob::Base
queue_as :import_queue
def perform(*args)
if args.first.present?
game_key = args.first
# get all particpations for a given game
game_participations = GameRoster.where(game_key: game_key)
process_records(game_participations)
end
end
def process_records(participations)
# Loop through participations and build record for saving...
participations.each do |participation|
if participation.try(:player_id)
record = create_or_find(participation)
record = update_record(record, participation)
end
begin
if record.valid?
record.save
else
end
rescue Exception => e
end
end
end
def create_or_find(participation)
participation_record = GameParticipation.where(
game_id: participation.game.try(:id),
player_id: participation.player.try(:id))
.first_or_initialize do |record|
record.game = Game.find_by(game_key: participation.game_key)
record.player = Player.find_by(id: participation.player_id)
record.club = Club.find_by(club_id: participation.club_id)
record.status = parse_status(participation.player_status)
end
return participation_record
end
def update_record(record, record)
old_status = record.status
new_status = parse_status(record.player_status)
if old_status != new_status
record.new_status = record.player_status
record.comment = "status was updated via participations import job"
end
return record
end
end
They recently updated and added an additional option you can set that should help with the deadlocking. I had the same issue and was on 4.1, moving to 4.1.1 fixed this issue for me.
https://github.com/collectiveidea/delayed_job_active_record
https://rubygems.org/gems/delayed_job_active_record
Problems locking jobs
You can try using the legacy locking code. It is usually slower but works better for certain people.
Delayed::Backend::ActiveRecord.configuration.reserve_sql_strategy = :default_sql
i am trying to work out how to write a rake tasks that will run daily and find where the days remaining is 0 to update the column amount to zero.
I have the following methods defined in my model, though they don't exactly appear to be working as I am getting the following error in the view
undefined method `-#' for Mon, 27 Jun 2016:Date
def remaining_days
expired? ? 0 : (self.expire_at - Date.today).to_i
end
def expired?
(self.expire_at - Date.today).to_i <= 0
end
def expire_credits
if expired?
self.update(:expire_at => Date.today + 6.months, :amount => 0)
end
end
with the rake tasks i have never written of these and i thought i would be able to call a method of StoreCredit that would expire the points if certain conditions are met but i am not sure how this all works
task :expire_credits => :environment do
puts 'Expiring unused credits...'
StoreCredit.expire_credits
puts "done."
end
# model/store_credit.rb
# get all store_credits that are expired on given date, default to today
scope :expire_on, -> (date = Date.current) { where("expire_at <= ?", date.beginning_of_day) }
class << self
def expire_credits!(date = Date.current)
# find all the expired credits on particular date, and update all together
self.expire_on(date).update_all(amount: 0)
end
end
Since it's a rake task, I think it's more efficient to update all expired ones together
#rake file
result = StoreCredit.expire_credits!
puts "#{result} records updated"
Retrieve Record Count Update
class << self
def expire_credits!(date = Date.current)
# find all the expired credits on particular date, and update all together
records = self.expire_on(date)
records.update_all(amount: 0)
records.length
end
end
You call class method but define instance method. You will need to define class method:
def self.expire_credits
I'm migrating my old blog posts into my new Rails blog, and I want their updated_at attribute to match the corresponding value on my old blog (not the date they were migrated into my new Rails blog).
How can I do this? When I set updated_at manually it gets overridden by the before_save callback.
Note: This question is only valid for Rails < 3.2.11. Newer versions of Rails allow you to manually set timestamps without them being overwritten.
If it's a one time thing you can turn record_timestamps on or off.
ActiveRecord::Base.record_timestamps = false
#set timestamps manually
ActiveRecord::Base.record_timestamps = true
When I ran into this issue with my app, I searched around for a bit and this seemed like it made the most sense to me. It's an initializer that I can call where I need to:
module ActiveRecord
class Base
def update_record_without_timestamping
class << self
def record_timestamps; false; end
end
save!
class << self
def record_timestamps; super ; end
end
end
end
end
As of recent versions of Rails (3.2.11 as per iGELs comment) you can set the updated_at property in code and the change will be honoured when saving.
I assume rails is keeping track of 'dirty' properties that have been manually changed and not overwriting on save.
> note = Note.last
Note Load (1.4ms) SELECT "notes".* FROM "notes" ORDER BY "notes"."id" DESC LIMIT 1
=> #<Note id: 39, content: "A wee note", created_at: "2015-06-09 11:06:01", updated_at: "2015-06-09 11:06:01">
> note.updated_at = 2.years.ago
=> Sun, 07 Jul 2013 21:20:47 UTC +00:00
> note.save
(0.4ms) BEGIN
(0.8ms) UPDATE "notes" SET "updated_at" = '2013-07-07 21:20:47.972990' WHERE "notes"."id" = 39
(0.8ms) COMMIT
=> true
> note
=> #<Note id: 39, content: "A wee note", created_at: "2015-06-09 11:06:01", updated_at: "2013-07-07 21:20:47">
So short answer, workarounds are not needed any longer in recent versions of rails.
I see two ways to accomplish this easily:
touch (Rails >=5)
In Rails 5 you can use the touch method and give a named parameter time like described in the documentation of touch
foo.touch(time: old_timestamp)
update_column (Rails >=4)
If you want it in Rails 4 and lower or want to avoid all callbacks you could use one of the update_column or update_columns methods which bypass all safe or touch callbacks and validations
foo.update_column(updated_at, old_timestamp)
or
foo.update_columns(updated_at: old_timestamp)
I took Andy's answer and modified it to accept blocks:
module ActiveRecord
class Base
def without_timestamping
class << self
def record_timestamps; false; end
end
yield
class << self
remove_method :record_timestamps
end
end
end
end
This is riffing off of Andy Gaskell's answer:
class ActiveRecord::Base
class_inheritable_writer :record_timestamps
def do_without_changing_timestamps
self.class.record_timestamps = false
yield
ensure
self.class.record_timestamps = true
end
end
The solution is to temporarily set ActiveRecord::Base.record_timestamps to false:
ActiveRecord::Base.record_timestamps = false
# Make whatever changes you want to the timestamps here
ActiveRecord::Base.record_timestamps = true
If you want a somewhat more robust solution, you may want to try something like what mrm suggested:
module ActiveRecord
class Base
def self.without_timestamping
timestamping = self.record_timestamps
begin
self.record_timestamps = false
yield
ensure
self.record_timestamps = timestamping
end
end
end
end
Then you can easily make changes to models without their timestamps being automatically updated:
ActiveRecord::Base.without_timestamping do
foo = Foo.first
bar = Bar.first
foo.updated_at = 1.month.ago
bar.updated_at = foo.updated_at + 1.week
foo.save!
bar.save!
end
Or, if you only want to update records from a specific class without timestamping:
module ActiveRecord
class Base
# Don't delete Rail's ActiveRecord::Base#inherited method
self.singleton_class.send(:alias_method, :__inherited__, :inherited)
def self.inherited(subclass)
__inherited__
# Adding class methods to `subclass`
class << subclass
def without_timestamping
# Temporarily override record_timestamps for this class
class << self
def record_timestamps; false; end
end
yield
ensure
class << self
remove_method :record_timestamps
end
end
end
end
end
end
E.g:
Foo.without_timestamping do
foo = Foo.first
bar = Bar.new(foo: foo)
foo.updated_at = 1.month.ago
foo.save! # Timestamps not automatically updated
bar.save! # Timestamps updated normally
end
Or you could use an approach similar to what Venkat D. suggested, which works on a per-instance basis:
module ActiveRecord
class Base
def without_timestamping
class << self
def record_timestamps; false; end
end
yield
ensure
class << self
remove_method :record_timestamps
end
end
end
end
E.g:
foo = Foo.first
foo.without_timestamping do
foo2 = Foo.new(parent: foo)
foo.updated_at = 1.month.ago
foo.save! # Timestamps not automatically updated
foo2.save! # Timestamps updated normally
end