I am writing a loop with switch case , instead of querying database three time , I am getting all lookup in one query and looping with conditions to assign to different instance variable ( which I will use in simple form as collections) . I am not sure how to append active record relations to instance variable, pls help me.
class QuestionsController < ApplicationController
def index
#question = Question.new
#question_lookups = Lookup.where({:look_up_for => "question"})
#question_lookups.each do |lk|
case lk.look_up_type
when 'mode'
#question_mode = lk #How can i do this here..?
when 'status'
#question_status = lk
else
#question_type = lk
end
end
session[:lk] = #question_mode
# #question_mode = Lookup.where({:look_up_for => "question", :look_up_type => "mode"})
# #question_status = Lookup.where({:look_up_for => "question", :look_up_type => "status"})
# #question_type = Lookup.where({:look_up_for => "question", :look_up_type => "type"})
end
end
Convert question_mode and so on into arrays and push lk to it
#question_mode, #question_status, #question_type = []
#question_lookups.each do |lk|
case lk.look_up_type
when 'mode'
#question_mode << lk #How can i do this here..?
when 'status'
#question_status << lk
else
#question_type << lk
end
end
Related
In Rails, i have an object called values that could be 1 of 20 kinds of ActiveRecord, and in only 1 of them there's a method(may be the wrong term, rails newbie) that can add a customized field in returned JSON object where the method name is the field name and method returned value is the field value. For example
class XXXController < ApplicationController
..
if a
values = A
elsif b
values = B
elseif c
values = C
..
end
render :json => values.to_json(:methods => :type_needed)
and you will see response like
{
..
"type_needed": true,
..
}
I only have type_needed defined in A which will return true in some cases. For others like B, C, D... which in total 19, i want them to all have type_needed returned as false, is there a way i can do that in one place instead of add type_needed method in the rest 19?
I will do it as follows:
json = values.to_json(:methods => :type_needed)
# => "[{\"id\":1,\"name\":\"Aaa\"},{\"id\":\"2\",\"name\":\"Bbb\"}]" # => Representational value only
ary = JSON.parse(json)
# => [{"id"=>1, "name"=>"Aaa"}, {"id"=>2, "name"=>"Bbb"}]
ary.map! { |hash| hash[:type_needed] = false unless hash.key?(:type_needed); hash }
# => [{"id"=>1, "name"=>"Aaa", :type_needed=>false}, {"id"=>2, "name"=>"Bbb", :type_needed=>false}]
ary.to_json
# => "[{\"id\":1,\"name\":\"Aaa\",\"type_needed\":false},{\"id\":\"2\",\"name\":\"Bbb\",\"type_needed\":false}]"
If I am understanding your question correctly then you want to define type_needed method once and have it included on all your 20 models. If yes, then you can define a concern and include it in all your 20 models.
app/models/concerns/my_model_concern.rb
module MyModelConcern
extend ActiveSupport::Concern
def type_needed?
self.respond_to?(:some_method)
end
end
app/models/a.rb
class A < ApplicationRecord
include MyModelConcern
def some_method
end
end
app/models/b.rb
class B < ApplicationRecord
include MyModelConcern
end
app/models/c.rb
class C < ApplicationRecord
include MyModelConcern
end
With the above
a = A.new
a.type_needed?
=> true
b = B.new
b.type_needed?
=> false
c = C.new
c.type_needed?
=> false
See if this helps.
In our Rails 3.2.13 app (Ruby 2.0.0 + Postgres on Heroku), we are often retreiving a large amount of Order data from an API, and then we need to update or create each order in our database, as well as the associations. A single order creates/updates itself plus approx. 10-15 associcated objects, and we are importing up to 500 orders at a time.
The below code works, but the problem is it's not at all efficient in terms of speed. Creating/updating 500 records takes approx. 1 minute and generates 6500+ db queries!
def add_details(shop, shopify_orders)
shopify_orders.each do |shopify_order|
order = Order.where(:order_id => shopify_order.id.to_s, :shop_id => shop.id).first_or_create
order.update_details(order,shopify_order,shop) #This calls update_attributes for the Order
ShippingLine.add_details(order, shopify_order.shipping_lines)
LineItem.add_details(order, shopify_order.line_items)
Taxline.add_details(order, shopify_order.tax_lines)
Fulfillment.add_details(order, shopify_order.fulfillments)
Note.add_details(order, shopify_order.note_attributes)
Discount.add_details(order, shopify_order.discount_codes)
billing_address = shopify_order.billing_address rescue nil
if !billing_address.blank?
BillingAddress.add_details(order, billing_address)
end
shipping_address = shopify_order.shipping_address rescue nil
if !shipping_address.blank?
ShippingAddress.add_details(order, shipping_address)
end
payment_details = shopify_order.payment_details rescue nil
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end
end
def update_details(order,shopify_order,shop)
order.update_attributes(
:order_name => shopify_order.name,
:order_created_at => shopify_order.created_at,
:order_updated_at => shopify_order.updated_at,
:status => Order.get_status(shopify_order),
:payment_status => shopify_order.financial_status,
:fulfillment_status => Order.get_fulfillment_status(shopify_order),
:payment_method => shopify_order.processing_method,
:gateway => shopify_order.gateway,
:currency => shopify_order.currency,
:subtotal_price => shopify_order.subtotal_price,
:subtotal_tax => shopify_order.total_tax,
:total_discounts => shopify_order.total_discounts,
:total_line_items_price => shopify_order.total_line_items_price,
:total_price => shopify_order.total_price,
:total_tax => shopify_order.total_tax,
:total_weight => shopify_order.total_weight,
:taxes_included => shopify_order.taxes_included,
:shop_id => shop.id,
:email => shopify_order.email,
:order_note => shopify_order.note
)
end
So as you can see, we are looping through each order, finding out if it exists or not (then either loading the existing Order or creating the new Order), and then calling update_attributes to pass in the details for the Order. After that we create or update each of the associations. Each associated model looks very similar to this:
class << self
def add_details(order, tax_lines)
tax_lines.each do |shopify_tax_line|
taxline = Taxline.find_or_create_by_order_id(:order_id => order.id)
taxline.update_details(shopify_tax_line)
end
end
end
def update_details(tax_line)
self.update_attributes(:price => tax_line.price, :rate => tax_line.rate, :title => tax_line.title)
end
I've looked into the activerecord-import gem but unfortunately it seems to be more geared towards creation of records in bulk and not update as we also require.
What is the best way that this can be improved for performance?
Many many thanks in advance.
UPDATE:
I came up with this slight improvement, which essentialy removes the call to update the newly created Orders (one query less per order).
def add_details(shop, shopify_orders)
shopify_orders.each do |shopify_order|
values = {:order_id => shopify_order.id.to_s, :shop_id => shop.id,
:order_name => shopify_order.name,
:order_created_at => shopify_order.created_at,
:order_updated_at => shopify_order.updated_at,
:status => Order.get_status(shopify_order),
:payment_status => shopify_order.financial_status,
:fulfillment_status => Order.get_fulfillment_status(shopify_order),
:payment_method => shopify_order.processing_method,
:gateway => shopify_order.gateway,
:currency => shopify_order.currency,
:subtotal_price => shopify_order.subtotal_price,
:subtotal_tax => shopify_order.total_tax,
:total_discounts => shopify_order.total_discounts,
:total_line_items_price => shopify_order.total_line_items_price,
:total_price => shopify_order.total_price,
:total_tax => shopify_order.total_tax,
:total_weight => shopify_order.total_weight,
:taxes_included => shopify_order.taxes_included,
:email => shopify_order.email,
:order_note => shopify_order.note}
get_order = Order.where(:order_id => shopify_order.id.to_s, :shop_id => shop.id)
if get_order.blank?
order = Order.create(values)
else
order = get_order.first
order.update_attributes(values)
end
ShippingLine.add_details(order, shopify_order.shipping_lines)
LineItem.add_details(order, shopify_order.line_items)
Taxline.add_details(order, shopify_order.tax_lines)
Fulfillment.add_details(order, shopify_order.fulfillments)
Note.add_details(order, shopify_order.note_attributes)
Discount.add_details(order, shopify_order.discount_codes)
billing_address = shopify_order.billing_address rescue nil
if !billing_address.blank?
BillingAddress.add_details(order, billing_address)
end
shipping_address = shopify_order.shipping_address rescue nil
if !shipping_address.blank?
ShippingAddress.add_details(order, shipping_address)
end
payment_details = shopify_order.payment_details rescue nil
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end
end
and for the associated objects:
class << self
def add_details(order, tax_lines)
tax_lines.each do |shopify_tax_line|
values = {:order_id => order.id,
:price => tax_line.price,
:rate => tax_line.rate,
:title => tax_line.title}
get_taxline = Taxline.where(:order_id => order.id)
if get_taxline.blank?
taxline = Taxline.create(values)
else
taxline = get_taxline.first
taxline.update_attributes(values)
end
end
end
end
Any better suggestions?
Try wrapping your entire code into a single database transaction. Since you're on Heroku it'll be a Postgres bottom-end. With that many update statements, you can probably benefit greatly by transacting them all at once, so your code executes quicker and basically just leaves a "queue" of 6500 statements to run on Postgres side as the server is able to dequeue them. Depending on the bottom end, you might have to transact into smaller chunks - but even transacting 100 at a time (and then close and re-open the transaction) would greatly improve throughput into Pg.
http://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html
http://www.postgresql.org/docs/9.2/static/sql-set-transaction.html
So before line 2 you'd add something like:
def add_details(shop, shopify_orders)
Order.transaction do
shopify_orders.each do |shopify_order|
And then at the very end of your method add another end:
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end //shopify_orders.each..
end //Order.transaction..
end //method
You can monkey-patch ActiveRecord like this:
class ActiveRecord::Base
#http://stackoverflow.com/questions/15317837/bulk-insert-records-into-active-record-table?lq=1
#https://gist.github.com/jackrg/76ade1724bd816292e4e
# "UPDATE THIS SET <list_of_column_assignments> FROM <table_name> THIS JOIN (VALUES (<csv1>, <csv2>,...) VALS ( <column_names> ) ON <list_of_primary_keys_comparison>"
def self.bulk_update(record_list)
pk = self.primary_key
raise "primary_key not found" unless pk.present?
raise "record_list not an Array of Hashes" unless record_list.is_a?(Array) && record_list.all? {|rec| rec.is_a? Hash }
return nil if record_list.empty?
result = nil
#test if every hash has primary keys, so we can JOIN
record_list.each { |r| raise "Primary Keys '#{self.primary_key.to_s}' not found on record: #{r}" unless hasAllPKs?(r) }
#list of primary keys comparison
pk_comparison_array = []
if (pk).is_a?(Array)
pk.each {|thiskey| pk_comparison_array << "THIS.#{thiskey} = VALS.#{thiskey}" }
else
pk_comparison_array << "THIS.#{pk} = VALS.#{pk}"
end
pk_comparison = pk_comparison_array.join(' AND ')
#SQL
(1..record_list.count).step(1000).each do |start|
key_list, value_list = convert_record_list(record_list[start-1..start+999])
#csv values
csv_vals = value_list.map {|v| "(#{v.join(", ")})" }.join(", ")
#column names
column_names = key_list.join(", ")
#list of columns assignments
columns_assign_array = []
key_list.each {|col|
unless inPK?(col)
columns_assign_array << "THIS.#{col} = VALS.#{col}"
end }
columns_assign = columns_assign_array.join(', ')
sql = "UPDATE THIS SET #{columns_assign} FROM #{self.table_name} THIS JOIN ( VALUES #{csv_vals} ) VALS ( #{column_names} ) ON ( #{pk_comparison} )"
result = self.connection.execute(sql)
return result if result<0
end
return result
end
def self.inPK?(str)
pk = self.primary_key
test = str.to_s
if pk.is_a?(Array)
(pk.include?(test))
else
(pk==test)
end
end
#test if given hash has primary keys included as hash keys and those keys are not empty
def self.hasAllPKs?(hash)
h = hash.stringify_keys
pk = self.primary_key
if pk.is_a?(Array)
(pk.all? {|k| h.key?(k) and h[k].present? })
else
h.key?(pk) and h[pk].present?
end
end
def self.convert_record_list(record_list)
# Build the list of keys
key_list = record_list.map(&:keys).flatten.map(&:to_s).uniq.sort
value_list = record_list.map do |rec|
list = []
key_list.each {|key| list << ActiveRecord::Base.connection.quote(rec[key] || rec[key.to_sym]) }
list
end
# If table has standard timestamps and they're not in the record list then add them to the record list
time = ActiveRecord::Base.connection.quote(Time.now)
for field_name in %w(created_at updated_at)
if self.column_names.include?(field_name) && !(key_list.include?(field_name))
key_list << field_name
value_list.each {|rec| rec << time }
end
end
return [key_list, value_list]
end
end
Then, you can generate a array of hashes containing your models attributes (including theirs primary keys) and do something like:
ActiveRecord::Base.transaction do
Model.bulk_update [ {attr1: val1, attr2: val2,...}, {attr1: val1, attr2: val2,...}, ... ]
end
It will be a single SQL command without Rails callbacks and validations.
For PostgreSQL, there are several issues that the above approach does not address:
You must specify an actual table, not just an alias, in the update target table.
You cannot repeat the target table in the FROM phrase. Since you are joining the target table to a VALUES table (hence there is only one table in the FROM phrase, you won't be able to use JOIN, you must instead use "WHERE ".
You don't get the same "free" casts in a VALUES table that you do in a simple "UPDATE" command, so you must cast date/timestamp values as such (#val_cast does this).
class ActiveRecord::Base
def self.update!(record_list)
raise ArgumentError "record_list not an Array of Hashes" unless record_list.is_a?(Array) && record_list.all? {|rec| rec.is_a? Hash }
return record_list if record_list.empty?
(1..record_list.count).step(1000).each do |start|
field_list, value_list = convert_record_list(record_list[start-1..start+999])
key_field = self.primary_key
non_key_fields = field_list - [%Q["#{self.primary_key}"], %Q["created_at"]]
columns_assign = non_key_fields.map {|field| "#{field} = #{val_cast(field)}"}.join(",")
value_table = value_list.map {|row| "(#{row.join(", ")})" }.join(", ")
sql = "UPDATE #{table_name} AS this SET #{columns_assign} FROM (VALUES #{value_table}) vals (#{field_list.join(", ")}) WHERE this.#{key_field} = vals.#{key_field}"
self.connection.update_sql(sql)
end
return record_list
end
def self.val_cast(field)
field = field.gsub('"', '')
if (column = columns.find{|c| c.name == field }).sql_type =~ /time|date/
"cast (vals.#{field} as #{column.sql_type})"
else
"vals.#{field}"
end
end
def self.convert_record_list(record_list)
# Build the list of fields
field_list = record_list.map(&:keys).flatten.map(&:to_s).uniq.sort
value_list = record_list.map do |rec|
list = []
field_list.each {|field| list << ActiveRecord::Base.connection.quote(rec[field] || rec[field.to_sym]) }
list
end
# If table has standard timestamps and they're not in the record list then add them to the record list
time = ActiveRecord::Base.connection.quote(Time.now)
for field_name in %w(created_at updated_at)
if self.column_names.include?(field_name) && !(field_list.include?(field_name))
field_list << field_name
value_list.each {|rec| rec << time }
end
end
field_list.map! {|field| %Q["#{field}"] }
return [field_list, value_list]
end
end
Lets say I have a model (an ActiveRecord class):
class Sample < ActiveRecord::Base
attr_accessor :x1
end
I know that
Sample.last.x1 == 1 #true
If I set Sample.last.x1 = 3 then Sample.last.x1_was == 1 #true.
But when I set the value of x1 again: Sample.last.x1 = 8 then Sample.last.x1_was == 3 #false, but Sample.last.x1 == 1 #true
I can guess why it happens (Sample.last wasn't saved since the change), but I want to find a way to retrieve the former value (not the db value) of x1. Can you suggest a way to do it?
I can't think of a reason to do that, but if you really need to, you could override the setter to store the various changes as you go.
def x1=( value )
#previous_x1_value = x1
super
end
def previous_x1_value
#previous_x1_value || x1_was
end
IT's all built in to rails. See the documentation for [ActiveRecord#dirty][1]
person.name = 'Bob'
person.changed? # => true
person.name_changed? # => true
person.name_was # => 'uncle bob'
person.name_change # => ['uncle bob', 'Bob']
person.name = 'Bill'
person.name_change # => ['uncle bob', 'Bill']
Here is my Lesson model:
#encoding: utf-8
class Lesson < ActiveRecord::Base
attr_accessible :content, :title, :parsed_content, :html_content, :user_id
serialize :parsed_content, Array
serialize :html_content, Array
serialize :pinyin_content, Array
serialize :defined_content, Array
serialize :literal_content, Array
validates :title, :presence => true
validates :content, :presence => true
belongs_to :user
before_update do |lesson|
lesson.makesandwich
end
before_save do |lesson|
lesson.delay.makesandwich
end
def makesandwich
require 'rmmseg'
#require 'to_lang'
require 'bing_translator'
require 'ruby-pinyin'
self.parsed_content = []
RMMSeg::Dictionary.load_dictionaries
content = self.content
paragraphs = content.split(/\r\n\r\n/) #convert to array of paragraphs
self.parsed_content = paragraphs
paragraphs.each_with_index do |text, ti|
text = text.gsub("。", "^^.")
text = text.gsub("？", "~~?")
text = text.gsub("！", "||!")
text = text.gsub(":", "：") #fix missing colons
text = text.split(/[.?!]/u) #convert to an array
text.each do |s|
s.gsub!("^^", "。")
s.gsub!("~~", "？")
s.gsub!("||", "！")
#s.gsub!("———————————",":")
end
text.each_with_index do |val, index|
algor = RMMSeg::Algorithm.new(text[index])
splittext = []
loop do
tok = algor.next_token
break if tok.nil?
tex = tok.text.force_encoding('UTF-8')
splittext << tex
text[index] = splittext
end
paragraphs[ti] = text
end
end
bing = BingTranslator.new(BING_API)
self.parsed_content = paragraphs
textarray = Marshal.load(Marshal.dump(paragraphs))
self.defined_content = Marshal.load(Marshal.dump(paragraphs))
self.literal_content = Marshal.load(Marshal.dump(paragraphs))
self.pinyin_content = Marshal.load(Marshal.dump(paragraphs))
textarray.each_with_index do |paragraph, pi|
paragraph.each_with_index do |sentence, si|
sentence.each_with_index do |word, wi|
if DictionaryEntry.find_by_simplified(word) != nil
self.defined_content[pi][si][wi] = DictionaryEntry.find_by_simplified(word).definition
#self.literal_content is down below
self.pinyin_content[pi][si][wi] = DictionaryEntry.find_by_simplified(word).pinyin
else
self.defined_content[pi][si][wi] = bing.translate(word, :from => 'zh-CHS', :to => 'en')
#self.defined_content[pi][si][wi] = word
#self.literal_content is down below
if PinYin.of_string(word, true).length > 1 #for punctuation
self.pinyin_content[pi][si][wi] = PinYin.of_string(word, true).join(" ").downcase
else
self.pinyin_content[pi][si][wi] = word
end
end
end
end
end
#Literal
literalarray = Marshal.load(Marshal.dump(paragraphs))
literalarray.each_with_index do |paragraph, pi|
paragraph.each_with_index do |sentence, si| #iterate array of sentence
literalarray[pi][si] = []
sentence.each_with_index do |word, wi| #iterate sentence's array of words
entrytobesliced = DictionaryEntry.find_by_simplified(word)
slicedentry = []
if entrytobesliced == nil
if word.length > 1 && word !~ /\w/ #/^\s*\w\d+\s*$/ #number regex #for cases where there is no DictionaryEntry
split = []
wordarray = word.split("").each_with_index() do |ws, wsi|
split << [DictionaryEntry.find_by_simplified(ws).definition]
end
literalarray[pi][si] << split
else
literalarray[pi][si] << [word] #in case none of the above work
end
else
entrytobesliced.simplified.each_char do |w|
singlechar = DictionaryEntry.find_by_simplified(w)
slicedentry << singlechar.definition.split("\", \"")
end
literalarray[pi][si] << slicedentry
end
self.literal_content = literalarray #slicedentry #literalarray
end
end
end
end
end
When I try to create a new lesson it errors like this: Jobs cannot be created for records before they've been persisted
But if I change it to after_save instead of before_save then I can see the work run, but it doesn't update the serialized arrays in the database.
Can someone please help me implement delayed_jobs for this? It was working when I had:
before_save do |lesson|
lesson.makesandwich #no delay
end
I think you're getting these errors:
Jobs cannot be created for records before they've been persisted
because your Lesson instances won't have an id until they've been saved and without an id, DJ has no way to know which instance it should be working with. So you have to use an after_save so that your Lesson has an id and can be uniquely identified. But then your updates from the delayed job won't be saved because, well, nothing asks for them to be saved. You should be able to get around that simply by adding a self.save or self.save! call at the end of makesandwich.
Now I'm fetching data from another url...
Here is my code:
require 'rubygems'
require 'nokogiri'
html = page.body
doc = Nokogiri::HTML(html)
doc.encoding = 'utf-8'
rows = doc.search('//table[#id = "MainContent_GridView1"]//tr')
#details = rows.collect do |row|
detail = {}
[
[:car, 'td[1]/text()'],
[:article, 'td[2]/text()'],
[:group, 'td[3]/text()'],
[:price, 'td[4]/text()'],
].each do |name, xpath|
detail[name] = row.at_xpath(xpath).to_s.strip
end
detail
end
#details
I tried to do it via array, not a hash. But I get a lot of errors...
Are there any ideas?
I need it for another method...
also i set data (this result hash) to another car here:
oem_art = []
#constr_num.each do |o|
as_oem = get_from_as_oem(o.ARL_SEARCH_NUMBER)
if as_oem.present?
oem_art << as_oem
end
end
#oem_art = oem_art.to_a.uniq
Do you just want to change a hash into an array? If so, just use the to_a method on your hash.
hash = {:a => "something", :b => "something else"}
array = hash.to_a
array.inspect #=> [[:a, "something"], [:b, "something else"]]
It looks like you're looking for something like hash['key'] to hash.key in Ruby
The Hash Class doesn't support .key notation by default, OpenStruct creates an Object from the Hash so you can use dot notation to access the properties. Overall it's basically just syntactic sugar with overhead.
Suggested code (from linked answer)
>> require 'ostruct'
=> []
>> foo = {'bar'=>'baz'}
=> {"bar"=>"baz"}
>> foo_obj = OpenStruct.new foo
=> #<OpenStruct bar="baz">
>> foo_obj.bar
=> "baz"
So in your example, you could do:
# Initialised somewhere
require 'ostruct'
DETAIL_INDICES = {
:car => 1,
:article => 2,
:group => 3,
:price => 4,
}
# ** SNIP **
#details = rows.map do |row|
DETAIL_INDICES.inject({}) do |h,(k,v)|
h.merge(k => row.at_xpath("td[#{v}]/text()").to_s.strip)
end
end.collect { |hash| OpenStruct.new hash }
#details.each do |item|
puts item.car
end
Of course if performance is a concern you can merge your map&collect (They are the same), but this is just a minor separation for basic semantic differences, although I usually only use map for consistency, so feel free to choose yourself :)
EDIT -- Additional code from your edit simplified
#oem_art = #constr_num.select do |item|
as_oem = get_from_as_oem(item.ARL_SEARCH_NUMBER)
as_oem.present?
end
puts #oem_art.uniq