Increase speed of Insert/Update/Delete using ActiveRecord / Rails - ruby-on-rails

I'm iterating over a collection of updates (which include additions, updates and deletions). I'm currently looping out over these updates, checking to see whether it exists. If it does, updating the information. If it doesn't, creating a record.
This seems to be taking an unpleasurable amount of time. I'd say 1 every 2 seconds, maybe slightly faster. But I've got 60,000 left to process and it's going to take 109 hours (give or take) to complete. Which isn't acceptable.
Is there anyway to perform this more efficiently?
Update.all.each do |u|
product = Product.find(:all, :conditions => ["product_codes = ?", u.product_codes])
if u.update_type == "Delete"
daap = DeletedProduct.new
daap.product_codes = u.product_codes
daap.date_deleted = timestamp
daap.save
if product.count == 1
product.first.destroy
puts "Product #{ u.product_codes } deleted"
u.destroy
#delete = #delete + 1
elsif product.count > 1
product.each do |pro|
pro.destroy
puts "Product #{ u.product_codes } deleted (UPDATE #{ u.id})"
#delete = #delete + 1
end
u.destroy
else
#notfound.push(u.product_codes)
u.destroy
#delete = #delete + 1
end
elsif u.update_type == "AddOrUpdate"
if product.count == 1
p = Product.first
p.data = u.data
p.last_updated = timestamp
p.save
puts "Product Updated #{ p.product_codes }"
u.destroy
#update = #update + 1
else
p = Product.new
p.product_codes = u.product_codes
p.data = u.data
p.last_updated = timestamp
p.save
puts "Product Added #{ p.product_codes }"
u.destroy
#delete = #delete + 1
end
end
end
EDIT: It's the product lookup that takes so long. Is there anyway to speed up this part of the process? 'product = Product.find(...'

Related

How does recalling a variable work in a method?

I have the following method:
vendor_orders = VendorOrder.where(id: params[:vendor_order_ids])
orders = Order.find(vendor_orders.pluck(:order_id))
products = Product.joins(:vendor_product).where(vendor_products:{vendor_id: current_user.id }).ids #get all vendor_products that match current_user.vendor
line_items = LineItem.joins(:shop_product).where(cart_id: orders.pluck(:cart_id), fulfillment_status: "processing", shop_products: {product_id: products}).where.not(fulfillment_status: "canceled")
messages = []
n = 0
puts "line items: #{line_items.count}" #puts out 1
line_items.map do |li|
if li.update_attribute(:fulfillment_status, params[:mass][:fulfillment_status])
n+=1
else
messages << "#{vendor_order.vendor_order_token}"
end
end
puts "line items2: #{line_items.count}" #puts out 0
if n == line_items.count
flash.keep[:notice] = "Update for #{vendor_orders.count} order(s) and #{n} product(s) successful"
else
flash.keep[:notice] = "Failed update for Order: #{messages.join if messages.any?}"
end
puts "line item3 #{line_items.count}" #puts out 0
respond_to do |format|
format.html { redirect_to vendor_orders_path }
end
The question i have is about the puts
When calling line_items.count after I update the line_items to then not match the variable, does it recall line_items from above?
Is this true? I always assumed once something was defined and passed, it would stay at the rate, unless redefined.
Marek solved the issue by letting me know .count is a call to the database which will then call what you defined previously.
To solve this, I just use line_items_count = line_items.count and use that above any alterations to check against it.

Scraping data in rails using thread

I am doing scraping to fetch the data from the website to my database in rails.I am fetching the 32000 record with this script there isn't any issue but i want to fetch the data faster so i apply the thread in my rake task but then there is a issue while running the rake task some of the data is fetching then the rake task getting aborted.
I am not aware of what to do task if any help can be done i am really grateful . Here is my rake task code for the scraping.
task scratch_to_database: :environment do
time2 = Time.now
puts "Current Time : " + time2.inspect
client = Mechanize.new
giftcard_types=Giftcard.card_types
find_all_merchant=Merchant.all.pluck(:id, :name).to_h
#first index page of the merchant
index_page = client.get('https://www.twitter.com//')
document_page_index = Nokogiri::HTML::Document.parse(index_page.body)
#set all merchant is deteled true
# set_merchant_as_deleted = Merchant.update_all(is_deleted: true) if Merchant.exists?
# set_giftcard_as_deleted = Giftcard.update_all(is_deleted: true) if Giftcard.exists?
update_all_merchant_record = []
update_all_giftcard_record = []
threads = []
#Merchant inner page pagination loop
page_no_merchant = document_page_index.css('.pagination.pagination-centered ul li:nth-last-child(2) a').text.to_i
1.upto(page_no_merchant) do |page_number|
threads << Thread.new do
client.get("https://www.twitter.com/buy-gift-cards?page=#{page_number}") do |page|
document = Nokogiri::HTML::Document.parse(page.body)
#Generate the name of the merchant and image of the merchant loop
document.css('.product-source').each do |item|
merchant_name= item.children.css('.name').text.gsub("Gift Cards", "")
href = item.css('a').first.attr('href')
image_url=item.children.css('.img img').attr('data-src').text.strip
#image url to parse the url of the image
image_url=URI.parse(image_url)
#saving the record of the merchant
# #merchant=Merchant.create(name: merchant_name , image_url:image_url)
if find_all_merchant.has_value?(merchant_name)
puts "this if"
merchant_id=find_all_merchant.key(merchant_name)
puts merchant_id
else
#merchant= Merchant.create(name: merchant_name , image_url:image_url)
update_all_merchant_record << #merchant.id
merchant_id=#merchant.id
end
# #merchant.update_attribute(:is_deleted, false)
#set all giftcard is deteled true
# set_giftcard_as_deleted = Giftcard.where(merchant_id: #merchant.id).update_all(is_deleted: true) if Giftcard.where(merchant_id: #merchant.id).exists?
#first page of the giftcard details page
first_page = client.get("https://www.twitter.com#{href}")
document_page = Nokogiri::HTML::Document.parse(first_page.body)
page_no = document_page.css('.pagination.pagination-centered ul li:nth-last-child(2) a').text.to_i
hrefextra =document_page.css('.dropdown-menu li a').last.attr('href')
#generate the giftcard details loop with the pagination
# update_all_record = []
find_all_giftcard=Giftcard.where(merchant_id:merchant_id).pluck(:row_id)
puts merchant_name
# puts find_all_giftcard.inspect
card_page = client.get("https://www.twitter.com#{hrefextra}")
document_page = Nokogiri::HTML::Document.parse(card_page.body)
#table details to generate the details of the giftcard with price ,per_off and final value of the giftcard
document_page.xpath('//table/tbody/tr[#class="toggle-details"]').collect do |row|
type1=[]
row_id = row.attr("id").to_i
row.at("td[2] ul").children.each do |typeli|
type = typeli.text.strip if typeli.text.strip.length != 0
type1 << type if typeli.text.strip.length != 0
end
value = row.at('td[3]').text.strip
value = value.to_s.tr('$', '').to_f
per_discount = row.at('td[4]').text.strip
per_discount = per_discount.to_s.tr('%', '').to_f
final_price = row.at('td[5] strong').text.strip
final_price = final_price.to_s.tr('$', '').to_f
type1.each do |type|
if find_all_giftcard.include?(row_id)
update_all_giftcard_record<<row_id
puts "exists"
else
puts "new"
#giftcard= Giftcard.create(card_type: giftcard_types.values_at(type.to_sym)[0], card_value:value, per_off:per_discount, card_price: final_price, merchant_id: merchant_id , row_id: row_id )
update_all_giftcard_record << #giftcard.row_id
end
end
#saving the record of the giftcard
# #giftcard=Giftcard.create(card_type:1, card_value:value, per_off:per_discount, card_price: final_price, merchant_id: #merchant.id , gift_card_type: type1)
end
# Giftcard.where(:id =>update_all_record).update_all(:is_deleted => false)
#delete all giftcard which is not present
# giftcard_deleted = Giftcard.where(:is_deleted => true,:merchant_id => #merchant.id).destroy_all if Giftcard.where(merchant_id: #merchant.id).exists?
time2 = Time.now
puts "Current Time : " + time2.inspect
end
end
end
end
threads.each(&:join)
puts "-------"
puts threads
# merchant_deleted = Merchant.where(:is_deleted => true).destroy_all if Merchant.exists?
merchant_deleted = Merchant.where('id NOT IN (?)',update_all_merchant_record).destroy_all if Merchant.exists?
giftcard_deleted = Giftcard.where('row_id NOT IN (?)',update_all_giftcard_record).destroy_all if Giftcard.exists?
end
end
Error i am receiving:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use
Each thread requires a separate connection to your database. You need to increase the connection pool size that your application can use in your database.yml file.
But your database should also be capable of handling the incoming connections. If you are using mysql you can check this by running select ##MAX_CONNECTIONS on your console.

Cant found model with out an ID in rails 3.2.12

i ve this method. I m not at all able to understand the error which is
Couldn't find Company without an ID
in ActiveRecord::RecordNotFound in CustomersController#bulk_create
This method is written to create customers for a company in bulk by taking their name and numbers in format name:number.
The method is as follows:
def bulk_create
res = ""
comp_id = params[:customer][:selected_companies].delete_if{|a| a.blank?}.first
comp = Company.find(comp_id)
s = SentSmsMessage.new
s.set_defaults
s.data = tmpl("command_signup_ok", customer, comp) unless params[:customer][:email].length > 0
s.data = params[:customer][:email] if params[:customer][:email].length > 0
s.company = comp if !comp.nil?
s.save
unless comp_id.blank?
params[:customer][:name].lines.each do |line|
(name, phone) = line.split(/\t/) unless line.include?(":")
(name, phone) = line.split(":") if line.include?(":")
phone = phone.gsub("\"", "")
phone = phone.strip if phone.strip.to_i > 0
name = name.gsub("\"", "")
name = name.gsub("+", "")
phone = "47#{phone}" if params[:customer][:active].to_i == 1
customer = Customer.first(:conditions => ["phone_number = ?", phone])
if customer.nil?
customer = Customer.new
customer.name = name
# customer.email
# customer.login
# customer.password
customer.accepted_agreement = DateTime.now
customer.phone_number = phone
customer.active = true
customer.accepted_agreement = DateTime.now
customer.max_msg_week = params[:customer][:max_msg_week]
customer.max_msg_day = params[:customer][:max_msg_day]
customer.selected_companies = params[:customer][:selected_companies].delete_if{|a| a.blank?}
res += "#{name} - #{phone}: Create OK<br />" if customer.save
res += "#{name} - #{phone}: Create failed<br />" unless customer.save
else
params[:customer][:selected_companies].each do |cid|
new_company = Company.find(cid) unless cid.blank?
if !new_company.nil?
if !customer.companies.include?(new_company)
customer.companies << new_company
if customer.save
res += "#{name} - #{phone}: Customer exists and the customer was added to the firm #{new_company.name}<br />"
else
res += "#{name} - #{phone}: Customer exist, but something went wrong during storage. Check if the client is in the firm.<br />"
end
else
res += "#{name} - #{phone}: Customer exists and is already on firm #{new_company.name}<br />"
end
end
end
end
s.sms_recipients.create(:phone_number => customer.phone_number)
end
s.save
s.send_as_sms
#result = res
respond_to do |format|
format.html { render "bulk_create"}
end
else
#result = "You have not selected any firm to add these users. Press the back button and try again."
respond_to do |format|
format.html { render "bulk_create"}
end
end
end
I want to update one situation here. That when i submit the form blank then it gives this error. Also if i filled the form with the values then its show the situation which the method is returning in case of fail.
res += "#{name} - #{phone}: Create failed <br />"
The tmpl method
private
def tmpl(setting_name, customer, company = nil)
text = ""
if customer.companies.count > 0
sn = "#{setting_name}_#{#customer.companies.first.company_category.suffix}".downcase rescue setting_name
text = Setting.value_by(sn) rescue ""
end
textlenth = text.length rescue 0
if textlenth < 3
text = Setting.value_by(setting_name) rescue Setting.value_by("command_error")
end
return fill_template(text, customer, company)
end
From the model customer.rb
def selected_companies=(cmps)
cmps.delete("")
# Check the old ones. Make a note if they are not in the list. If the existing ones are not in the new list, just remove them
self.companies.each do |c|
self.offer_subscriptions.find(:first, ["customer_id = ?", c]).destroy unless cmps.include? c.id.to_s
cmps.delete c.id.to_s if cmps.include? c.id.to_s
end
# Then create the new ones
cmps.each do |c2|
cmp = Company.find(:first, ["id = ?", c2])
if cmp && !c2.blank?
offerSubs = offer_subscriptions.new
offerSubs.company_id = c2
offerSubs.save
end
end
end
def selected_companies
return self.companies.collect{|c| c.id}
end
The association of customer is as follows:
has_many :offer_subscriptions
has_many :companies, :through => :offer_subscriptions
This code is written by the some one else. I m trying to understand this method but so far not being able to understand this code.
Please help.
Thanks in advance.
You are getting 'Couldn't find Company without an ID' error because your Company table doesn't contain record with id = comp_id
Change comp = Company.find(comp_id) to comp = Company.find_by_id(comp_id).
This will return nil instead of an error.
Add comp is not nil condition is already handled in your code.
Your comp_id line is returning nil.
comp_id = params[:customer][:selected_companies].delete_if{|a| a.blank?}.first
Post the params that get passed to this function and we could hopefully find out why. In the meantime you could enclose the block in a begin - rescue block to catch these errors:
begin
<all your code>
rescue ActiveRecord::RecordNotFound
return 'Unable to find a matching record'
end
try this:
comp = ""
comp = Company.find(comp_id) unless comp_id.nil?
instead of comp = Company.find(comp_id)
further nil checking present in your code.
Reason being
params[:customer][:selected_companies].delete_if{|a| a.blank?} = []
so [].first = nil
therefor, params[:customer][:selected_companies].delete_if{|a| a.blank?}.first = nil
and comp_id is nil
So check the log file and check what is coming in the parameter "selected_companies"
when you will find the parameter, everything will be understood well....

Rspec instance variables and controller testing

I'm writing a bowling score calculator, and I'm trying to set up RSpec tests, but for some reason I can't get my tests to work correctly.
players_controller_spec.rb:
require 'spec_helper'
describe PlayersController do
let(:player_names) { ["player1",
"player2",
"player3",
"player4"] }
describe "POST bowl" do
before(:each) do
#game = Game.create!
player_names.each do |name|
Player.create!(:name => name)
end
#game.players = Player.all
Player.all.each do |player|
(0..9).each do |number|
player.frames << Frame.create(:number => number)
end
end
end
describe "for the player's third bowl" do
before(:each) do
#game.players[#game.current_player].frames[9].update_attributes({:number => 9, :first_bowl => "X", :second_bowl => "X", :score => 20})
#game.update_attributes({:current_player => 0, :current_frame => 9})
end
describe "if the bowl is a number score" do
before(:each) do
post :bowl, {:score => "5", :id => #game.id}
end
it "should update the player's score" do
#game.players[#game.current_player].frames[#game.current_frame].score.should == 25
end
end
end
end
end
players_controller.rb
def bowl
#game = Game.find(params[:id])
#score = params[:score]
#current_player = #game.current_player
#current_frame = #game.current_frame
#player = #game.players[#current_player]
#frame = #player.frames[#current_frame]
if #frame.first_bowl.nil?
#frame.first_bowl = #score
if #score == "/"
raise "Error"
end
if #score == "X" && #frame.number == 9
#frame.bonus = 2
end
#frame.score = (/\A[0-9]\z/ === #score ? #score.to_i : 10)
elsif #frame.second_bowl.nil?
#frame.second_bowl = #score
if #frame.score + #score.to_i > 10
raise "Error"
end
if #score == "X"
if #frame.number != 9 || (#frame.number == 9 && #frame.first_bowl != "X") # can't be a spare has to be number or strike
raise "Error"
end
end
if #score == "/" && #frame.number == 9
#frame.bonus = 1
end
if /\A[0-9]\z/ === #score
#frame.score += #score.to_i
elsif #score == "/"
#frame.score = 10
elsif #score == "X"
#frame.score = 20
end
elsif #frame.third_bowl.nil?
#frame.third_bowl = #score
if #frame.number != 9
raise "Error"
end
#frame.bonus = nil
#frame.update_attributes({:score => (/\A[0-9]\z/ === #score ? #frame.score + #score.to_i : #frame.score + 10)})
else
raise "Error"
end
#frame.save
if #game.current_frame > 0
#prev_frame = #player.frames[#frame.number-1]
if #prev_frame.nil?
#prev_frame = Frame.create(:number => #game.current_frame-1)
#player.frames << #prev_frame
#player.frames = #player.frames.sort_by { |f| f.number }
end
update_scores
end
The spec in question is players_controller_spec.rb and at the start of the tests I'm creating a new game with 4 players and each player with 10 frames. Before each test, I'm setting a certain frame's values to be fit what I'm trying to test. The test above is an example where I want to make sure that bowling a score of 5 on the third bowl on the last frame correctly updates the score. But, even though in the debugger I see that the score is updated in the frame (when I debug in the controller method), once I return to the Rspec test, it doesn't work. It expects 25 but gets nil. Is there something I'm missing about how instance variables are transferred between specs and controllers?
So first off there is no 'transferring'. The controller and the example are 2 completely independent objects, each with their own instance variables (You can use the assigns spec helper to retrieve the value of a controller instance variable though).
That's not the root cause of your issue. You do, even before the controller executes, have an #game instance variable that is the game you are interested in. However with activerecord, every time you do Game.find you'll receive separate ruby objects (corresponding to the same database row). Once the row has been loaded from the database it doesn't notice changes made to the database behind its back.
You can reload the object with #game.reload
As a side note this sort of stuff is easier to work with if most of that logic was pushed down into one of your models rather than sitting in the controller.

Ruby on rails rake task not returning up-to-date model info

While building a billing system I encountered the following problem:
In scheduler.rake :
#users.each do |user|
begin
puts "CALCULATING COSTS =========="
costs = user.total_billable_price_per_month(Time.now)
lead_count = user.total_billable_leads_count(Time.now)
click_count = user.total_billable_clicks_count(Time.now)
puts "CREATING NEW INVOICES ======="
#invoice = user.invoices.new
# if user.free_credits
# invoice.total_price_in_cents = 0
# else
# invoice.total_price_in_cents = costs.cents
# end
#invoice.total_leads = lead_count
#invoice.total_clicks = click_count
#invoice.total_price_in_cents = costs.cents
# Als de vorige factuur onder de 10 euro was, dan geld bij huidige factuur optellen
if user.invoices.last && user.invoices.last.total_price_in_cents < 1000
puts "Bedrag onder 10 euro"
puts "Last invoice = #{user.invoices.last.total_price_in_cents}"
#invoice.total_price_in_cents += user.invoices.last.total_price_in_cents
end
puts "SAVING INVOICE FOR #{user.name} with ID = #{user.id}"
# Factuur saven
#invoice.save
#Als de factuur hoger of gelijk is als 10euro, dan factuur aanmaken
if #invoice.total_price_in_cents >= 1000
#Moneybird factuur versturen
puts "COSTS ARE HIGHER THAN 10 EURO, CREATING MONEYBIRD INVOICE"
moneybird_invoice = MoneybirdInvoice.new
sleep(15)
moneybird_invoice.contact_id = MoneybirdContact.find_by_customer_id(user.id).id
sleep(15)
moneybird_invoice.details_attributes = [
{ :description => "Aantal leads", :amount => lead_count, :price => user.lead_price.cents.to_f/100},
{ :description => "Aantal clicks", :amount => click_count, :price => user.click_price.cents.to_f/100}
]
puts "TRYING TO SAVE MONEYBIRD INVOICE"
if moneybird_invoice.save && moneybird_invoice.put(:send_invoice)
puts "SUCCESFULLY SAVED INVOICE"
#GET UPDATED PAY URL
#sent_mb_invoice = MoneybirdInvoice.get(moneybird_invoice.id)
sleep(15)
#invoice.update_attributes(:moneybird_invoice_id => #sent_mb_invoice['invoice_id'], :moneybird_pay_url => #sent_mb_invoice['pay_url'])
else
puts "MONEYBIRD INVOICE FAILED"
puts moneybird_invoice.errors.inspect
end
else
# GEEN MONEYBIRD FACTUUR AANMAKEN
end
rescue
puts "ER IS IETS FOUT GEGAAN MET FACTUREREN USER ID = #{user.id} & NAME = #{user.name}"
puts #invoice.errors.inspect
end
end
This piece of code should constantly increment each time the rake task is run, except when the total amount reaches > 1000.
if user.invoices.last && user.invoices.last.total_price_in_cents < 1000
puts "Bedrag onder 10 euro"
puts "Last invoice = #{user.invoices.last.total_price_in_cents}"
#invoice.total_price_in_cents += user.invoices.last.total_price_in_cents
end
The code above always puts "Last invoice = 100" => This should increment each time the rake tasks is run
Every new invoice still has the same total_price_in_cents (when I'm expecting that it should increment).
What is going on ?
EDIT: Added after code upadte:
In your updated code, it looks like you were calling user.invoices.last after you called user.invoices.new, this is why it always returned the same value.
Create a variable #last_invoice = user.invoices.last before call user.invoices.new.
ORIGINAL ANSWER:
In your original code posting, it looks like your save call on #invoice happened outside the loop -- I believe you're only saving it once.
task :create_invoices => :environment do
# Begin the loop for each use
User.all.each do |user|
#invoice = user.invoices.build
#If last bill < 1000
if user.invoices.last && user.invoices.last.total_price_in_cents < 1000
puts "Last invoice = #{user.invoices.last.total_price_in_cents}"
#invoice.total_price_in_cents += user.invoices.last.total_price_in_cents
#invoice.save
end # end 'if' statement
end # end loop for all users
end # end task definition
So you loop though the users table, but never save updates -- except for the very last time after you exit the loop

Resources