I'm using raw sql bulk updates (for performance reasons) in the context of a rake task. Something like the following:
update_sql = Book.connection.execute("UPDATE books AS b SET
stock = vs.stock,
promotion = vs.promotion,
sales = vs.sales
FROM (values #{values_string}) AS vs
(stock, promotion, sales) WHERE b.id = vs.id;")
While everything is "transparent" in local development, if this SQL fails in production during the execution of the rails task (for example because the promotion column is nil and the statement becomes invalid), no error is logged.
I can manually log this with catching the exception, like below, however some option that would allow for automatic logging would be better.
begin
...
rescue ActiveRecord::StatementInvalid => e
Rails.logger.fatal "Books update: ActiveRecord::StatementInvalid: "+ e.to_s
end
You can make your own custom class in your model folder:
app/models/custom_sql_logger.rb :
class CustomSqlLogger
def self.debug(msg=nil)
#custom_log ||= Logger.new("#{Rails.root}/log/custom_sql.log")
#custom_log.debug(msg) unless msg.nil?
end
end
Then go to the rake task where you would like to debug updated fields for example lib/task/calculate_avarages.rake and call your custom debugger:
CustomSqlLogger.debug "The field was successfully updated into DB"
Example from my project:
require 'rake'
task :calculate_averages => :environment do
products = Product.all
products.each do |product|
puts "Calculating average rating for #{product.name}..."
product.update_attribute(:average_rating, product.reviews.average("rating"))
CustomSqlLogger.debug "#{product.name} was susscefully updated into DB"
end
end
Custom debugger will create the new file custom_sql.log into log folder: log/custom_sql.log and saved all information there. Beware of a log file size after a while.
Related
In Rails 6.1, ActiveStorage creates database records for all variants when they're loaded for the first time: https://github.com/rails/rails/pull/37901
I'd like to enable this, but since I have tens of thousands of files in my production Rails app, it'd be problematic (and presumably slow) to have users creating so many database records as they browse the site. Is there a way to write a Rake task that'll iterate through every attachment in my database, and generate the variants and save them in the database?
I'd run that once, after enabling the new active_storage.track_variants config, and then any newly-uploaded files would be saved when they're loaded for the first time.
Thanks for the help!
This is the Rake task I ended up creating for this. The Parallel stuff can be removed if you have a smaller dataset, but I found that with 70k+ variants it was intolerably slow when doing it without any parallelization. You can also ignore the progress bar-related code :)
Essentially, I just take all the models that have an attachment (I do this manually, you could do it in a more dynamic way if you have a ton of attachments), and then filter the ones that are not variable. Then I go through each attachment and generate a variant for each size I've defined, and then call process on it to force it to be saved to the database.
Make sure to catch MiniMagick (or vips, if you prefer) errors in the task so that a bad image file doesn't break everything.
# Rails 6.1 changes the way ActiveStorage works so that variants are
# tracked in the database. The intent of this task is to create the
# necessary variants for all game covers and user avatars in our database.
# This way, the user isn't creating dozens of variant records as they
# browse the site. We want to create them ahead-of-time, when we deploy
# the change to track variants.
namespace 'active_storage:vglist:variants' do
require 'ruby-progressbar'
require 'parallel'
desc "Create all variants for covers and avatars in the database."
task create: :environment do
games = Game.joins(:cover_attachment)
# Only attempt to create variants if the cover is able to have variants.
games = games.filter { |game| game.cover.variable? }
puts 'Creating game cover variants...'
# Use the configured max number of threads, with 2 leftover for web requests.
# Clamp it to 1 if the configured max threads is 2 or less for whatever reason.
thread_count = [(ENV.fetch('RAILS_MAX_THREADS', 5).to_i - 2), 1].max
games_progress_bar = ProgressBar.create(
total: games.count,
format: "\e[0;32m%c/%C |%b>%i| %e\e[0m"
)
# Disable logging in production to prevent log spam.
Rails.logger.level = 2 if Rails.env.production?
Parallel.each(games, in_threads: thread_count) do |game|
ActiveRecord::Base.connection_pool.with_connection do
begin
[:small, :medium, :large].each do |size|
game.sized_cover(size).process
end
# Rescue MiniMagick errors if they occur so that they don't block the
# task from continuing.
rescue MiniMagick::Error => e
games_progress_bar.log "ERROR: #{e.message}"
games_progress_bar.log "Failed on game ID: #{game.id}"
end
games_progress_bar.increment
end
end
games_progress_bar.finish unless games_progress_bar.finished?
users = User.joins(:avatar_attachment)
# Only attempt to create variants if the avatar is able to have variants.
users = users.filter { |user| user.avatar.variable? }
puts 'Creating user avatar variants...'
users_progress_bar = ProgressBar.create(
total: users.count,
format: "\e[0;32m%c/%C |%b>%i| %e\e[0m"
)
Parallel.each(users, in_threads: thread_count) do |user|
ActiveRecord::Base.connection_pool.with_connection do
begin
[:small, :medium, :large].each do |size|
user.sized_avatar(size).process
end
# Rescue MiniMagick errors if they occur so that they don't block the
# task from continuing.
rescue MiniMagick::Error => e
users_progress_bar.log "ERROR: #{e.message}"
users_progress_bar.log "Failed on user ID: #{user.id}"
end
users_progress_bar.increment
end
end
users_progress_bar.finish unless users_progress_bar.finished?
end
end
This is what the sized_cover looks like in game.rb:
def sized_cover(size)
width, height = COVER_SIZES[size]
cover&.variant(
resize_to_limit: [width, height]
)
end
sized_avatar is pretty much the same thing.
This is my class:
class Plan < ActiveRecord::Base
def testing
self.with_lock do
update_columns(lock: true)
byebug
end
end
def testing2
self.lock!
byebug
end
end
I opened two rails consoles.
In first console:
p = Plan.create
=> (basically success)
p.id
=> 12
p.testing2
(byebug) # simulation of halting the execution,
(BYEBUG) # I just leave the rails console open and wait at here. I expect others won't be able to update p because I still got the lock.
On second console:
p = Plan.find(12)
=> (basically said found)
p.name = 'should not be able to be stored in database'
=> "should not be able to be stored in database"
p.save!
=> true # what????? Why can it update my object? It's lock in the other console!
lock! in testing2 doesn't lock while with_lock in testing does work. Can anybody explain why lock! doesn't work?
#lock! uses SELECT … FOR UPDATE to acquire a lock.
According to PostgreSQL doc.
FOR UPDATE causes the rows retrieved by the SELECT statement to be locked as though for update. This prevents them from being locked, modified or deleted by other transactions until the current transaction ends.
You need a transaction to keep holding a lock of a certain row.
Try
console1:
Plan.transaction{Plan.find(12).lock!; sleep 100.days}
console2:
p = Plan.find(12)
p.name = 'should not be able to be stored in database'
p.save
#with_lock acquire a transaction for you, so you don't need explicit transaction.
(This is PostgreSQL document. But I think other databases implement similar logic. )
Im using subscriptions and plans in my application together with Stripe. My Plan entity has a lot of "visual" data on the model and is related with the payment gateway sharing a identifier.
I have already a migration that generates my basic plan data. Like this:
class CreateBasicPlanData < ActiveRecord::Migration[5.1]
def change
Plan.create(name:'Hobby',
visibility: 'Low Visibility',
card_description:'Free portfolio listing good for beginners.',
features_description:'<ul><li>5 Portfolio Images</li><li>Messaging to other Talent</li><li>Basic Search Ranking</li></ul>',
price: 0,
css:'plan-hobby',
number_albums: 1,
number_photos_per_album: 5,
payment_gateway_plan_identifier: 'hobby'
)
Plan.create(name:'Professional', card_description:'Solid portfolio for those wanting more exposure and booking opportunities.',
visibility: 'High Visibility',
features_description:'<strong>Everything included in Hobby <em>PLUS:</em></strong><ul><li>25 Portfolio Images</li><li>Intermediate Search Ranking</li><li>Multi-state portfolio listing</li></ul>',
price: 4.99,
css:'plan-professional',
number_albums: 5,
number_photos_per_album: 25,
payment_gateway_plan_identifier: 'professional'
)
I want to create a Job that, when the system is ok, get all the data from my local database, and create the Stripe Plans. My code is something like that:
class SyncLocalPlansWithStripe < ActiveJob::Base
def perform
plans = Plan.all
#delete all the plans on stripe
Plan.transaction do
begin
puts 'Start deleting'
Stripe::Plan.list.each do |plan_to_delete|
plan_to_delete.delete
end
puts 'End deleting'
end
end
Plan.transaction do
begin
plans.each do |plan|
PaymentGateway::CreatePlanService.new(plan).run
end
rescue PaymentGateway::CreatePlanServiceError => e
puts "Error message: #{e.message}"
puts "Exception message: #{e.exception_message}"
end
end
end
My question is. How can I run this Job, when I want, only once, from the console?
Something like rake:job run sync_local_plans_with_stripe
I think you are confusing rake tasks with ActiveJob.
If you want to run a job from within rails console you can just execute SyncLocalPlansWithStripe.perform_now. See https://guides.rubyonrails.org/active_job_basics.html#enqueue-the-job
As suggested in the comments you can also run the job directly from the command line using Rails runner: rails runner "SyncLocalPlansWithStripe.perform_now"
Or if you'd rather run this as a rake task then you'll need to create one for this instead. See https://guides.rubyonrails.org/command_line.html#custom-rake-tasks
I am in a situation where I have to update more than 100k records in the database with best efficient way Please see below my code:
namespace :order do
desc "update confirmed at field for Payments::Order"
task set_confirmed_at: :environment do
puts "==> Updating confirmed_at for orders starts ...".blue
Payments::Order.find_each(batch_size: 10000) do |orders|
order_action = orders.actions.where("sender LIKE ?", "%ConfirmJob%").first if orders.actions
if !order_action.blank?
orders.update_attribute(:confirmed_at, order_action.created_at)
puts "order id = #{orders.id} has been updated.".green
end
end
puts "== completed ==".blue
end
end
Here I am breaking records into 10000 of each batch size and then try to update the record on the basis of some conditions so could anyone suggest me a more efficient way to do the same task.
Thank you in advance!
You can try update_all:
Payments::Order.joins(:actions).where(Payment::OrderAction.arel_table[:sender].matches("%ConfirmJob%")).update_all("confirmed_at = actions.created_at")
So your code will look like this:
namespace :order do
desc "update confirmed at field for Payments::Order"
task set_confirmed_at: :environment do
puts "==> Updating confirmed_at for orders starts ...".blue
Payments::Order.joins(:actions).where(Payments::OrderAction.arel_table[:sender].matches("%ConfirmJob%")).update_all("confirmed_at = actions.created_at")
puts "== completed ==".blue
end
end
Update:
I've investigated an issue and found out that bulk update with joined table is a long term issue in rails
As set part uses string parameter as it is I suggest to add from clause there.
namespace :order do
desc "update confirmed at field for Payments::Order"
task set_confirmed_at: :environment do
puts "==> Updating confirmed_at for orders starts ...".blue
Payments::Order.joins(:actions).
where(Order::Action.arel_table[:sender].matches("%ConfirmJob%")).
update_all("confirmed_at = actions.created_at FROM actions")
puts "== completed ==".blue
end
end
You are doing Payments::Order.find_each so your solution will loop for each Payment::Order when you only want to loop for the ones having actions.server like '%ConfirmJob%', so I will go with this solution:
Payments::Order
.includes(:actions)
.joins(:actions)
.where("actions.server like '%?%'", "ConfirmJob")
.find_each do |order|
order_action = order.actions.first
order.update!(confirmed_at: order_action.created_at)
end
AllegroAPI is a class in the /models directory that calls an external API. It works as I wish when I test in somewhere else not by running rake task.
Example working code:
require "./AllegroAPI"
allegro = AllegroAPI.new(login: 'LOGIN',
password: File.read('XXXX.txt'),
webapikey: File.read('XXX.txt')
)
puts allegro.do_search({"search-string"=>"nokia",
"search-price-from"=>300.0,
"search-price-to"=>500.0,
"search-limit"=>50}).to_s
As I've said it works correctly. It calls the API and prints out the result.
File allegro.rb is also in the models directory and it's a file I'm executing by running this task:
namespace :data do
desc "Update auctions table in database"
task update_auctions: :environment do
Allegro.check_for_new_auctions
end
end
allegro.rb:
module Allegro
require 'AllegroAPI'
def self.check_for_new_auctions
allegro = AllegroAPI.new(login: 'LOGIN',
password: File.read('app/models/ignore/XXXX.txt'),
webapikey: File.read('app/models/ignore/XXX.txt')
)
looks = Look.all
looks.each do |l|
hash_to_ask = ActiveSupport::JSON.decode(l[:look_query]).symbolize_keys
hash_to_ask = hash_to_ask.each_with_object({}) do |(k,v), h|
if v.is_number?
h[k.to_s.split('_').join('-')] = v.to_f
else
h[k.to_s.split('_').join('-')] = v
end
end
results = allegro.do_search(hash_to_ask)
#do something with data
end
end
end
The problem is that it doesn't return anything. var result is not nil, but it does not hold anything.
When I'm trying to debug it and call API from the inside do_search function it's calling API, doesn't raise a error but response is nothing. AllegroAPI works correctly. There is no problem with var "hash_to_ask", it's exactly the same hash as in working example.
EDIT:
I've commented out check_for_new_auctions and used "puts", it works fine when I run it by executing rake task. Then I've used exactly the same code which I used in normal file which have ran properly:
class Allegro
def self.check_for_new_auctions
allegro = AllegroAPI.new(login: 'LOGIN',
password: File.read('app/models/ignore/XXXX.txt'),
webapikey: File.read('app/models/ignore/XXXX.txt')
)
hash_to_ask = {"search-string"=>"nokia",
"search-price-from"=>300.0,
"search-price-to"=>500.0,
"search-limit"=>50}
allegro.do_search(hash_to_ask).to_s
end
end
It have not worked;/ The returned value from allegro.do_search(hash_to_ask) is hash, not empty, not nil but when I try to print it, it's nothing, empty place.
EDIT:
Everything have worked properly, waste like 15 hours total debugging the problem which have not existed. I'm not sure why it have not worked but it couldn't print to the console after converting to string, so I tried writing it down to file blindly. What I have found in the text file? Data.
I don't know why it couldn't print out everything in the console.
In the IRB script that you show, you have some puts statement that is not in your rake task. So for debugging, I would add puts ... to your Rake task, e.g.:
namespace :data do
desc "Update auctions table in database"
task update_auctions: :environment do
puts "Start Auctions..."
results = Allegro.check_for_new_auctions
puts "Results: #{results}"
end
end
Now, when you run:
rake data:update_auctions
You should get some output. Otherwise rinse-and-repeat by adding puts statements in the method that you are calling.