I have a Convo table and a GroupMeeting table that both are associated with a Msg table.
I want to find all the instances where the current_user has convos or group_meetings with msgs, combine the two, and then show both together to the user in order of the last msg.created_at
Here I have defined both:
#convos = Convo.includes(:msgs).where("sender_id = ? OR recipient_id = ?", current_user, current_user).where.not(:msgs => { :id => nil }).merge(Msg.order(created_at: :desc))
#group_meetings = current_user.group_meetings.includes(:msgs).where.not(:msgs => { :id => nil }).merge(Msg.order(created_at: :desc))
And then combined them together:
#convos = #convos + #group_meetings
What I can't figure out is how to now sort them by msg.created_at
I have tried the following:
#convos = (#convos + #group_meetings).sort_by(&:"#{msg.created_at}")
#convos.order('msg.created_at DESC')
These all seem to be server-side sorting though. How can I sort these based off the join table, after the array has been created?
Please let me know if I need to supply any other details. Thank you!!
You can try the following:
(#convos + #group_meetings).sort_by { |item| item.msgs.minimum(:created_at) }
I'm trying to cache a lot of data (100 000) that i took with an SQL query, but the caching is not working (take about 30sec to write into the cache and the same amount of time to read it) What I'm I doing wrong? my config variable is already set to true
query = "SELECT inscriptions.`id`, banners.`id`, banners.`name`, inscriptions.`registered_at`,
inscriptions.`synched_at`, inscriptions.`state`
FROM inscriptions
JOIN firm_offices
ON inscriptions.`firm_office_1_id` = firm_offices.`id`
JOIN firms
ON firm_offices.`firm_id` = firms.`id`
JOIN banners
ON firms.`banner_id` = banners.`id`
GROUP BY inscriptions.`id`"
result = ActiveRecord::Base.connection.execute(query)
Rails.cache.fetch 'huge-array' do
data = []
result.each do |r|
data.push({ :id => r[0],
:banner_id => r[1],
:banner_name => r[2],
:registered_at => r[3],
:synched_at => r[4],
:state => r[5]})
end
data
end
#data = Rails.cache.read("huge-array")
Move it all inside your fetch block:
#data ||= Rails.cache.fetch 'huge-array' do
query = "SELECT inscriptions.`id`, banners.`id`, banners.`name`, inscriptions.`registered_at`, inscriptions.`synched_at`, inscriptions.`state`
FROM inscriptions
JOIN firm_offices
ON inscriptions.`firm_office_1_id` = firm_offices.`id`
JOIN firms
ON firm_offices.`firm_id` = firms.`id`
JOIN banners
ON firms.`banner_id` = banners.`id`
GROUP BY inscriptions.`id`"
result = ActiveRecord::Base.connection.execute(query)
data = []
result.each do |r|
data.push({ :id => r[0],
:banner_id => r[1],
:banner_name => r[2],
:registered_at => r[3],
:synched_at => r[4],
:state => r[5]})
end
data
end
Notes:
You don't need to actually move all of it into the block, just the expensive parts (e.g., execute(query)).
Your big SQL query looks like it could translate pretty easily into an AR query. You might want to translate it into an AR query, and then use to_sql if that turns out to be more efficient.
There's no need to run the query and the fetch block each time this function is called. Try something like
#data = Rails.cache.read("huge-array")
if #data.empty?
result = ActiveRecord::Base.connection.execute(query)
#data = []
result.each do |r|
#data.push({ :id => r[0],
:banner_id => r[1],
:banner_name => r[2],
:registered_at => r[3],
:synched_at => r[4],
:state => r[5]})
end
Rails.cache.write("huge-array", #data)
end
return #data
This way you only have to do the expensive query + array creation if the data does not already exist in cache.
I have the following query in my rails app but its result does not have attributes name so I am not able to use it in amcharts
EmployeeDepartment.joins(:states).group
("employee_departments.name").count
the result is : {"Academic Support":1}
how to make it like this {"department_name":"Academic Support","department_count":1}
Let's say you have a hash like this:
hash = { "Academic Support" => 1, "Another Department" => 3, "Something Else" => 4 }
You can just use map to transform it into an array of hashes containing what you need.
hash.map { |k, v| { "department_name" => k, "department_count" => v } }
=> [{"department_name"=>"Academic Support", "department_count"=>1},
{"department_name"=>"Another Department", "department_count"=>3},
{"department_name"=>"Something Else", "department_count"=>4}]
If your hash only ever contains one key/value pair and you just want another hash, you could try this:
Hash[[["department_name", "department_count"], hash.first].transpose]
Or even simpler...
{ "department_name" => hash.keys.first, "department_count" => hash.values.first }
I solved my problem by this steps:
the following code:
EmployeeDepartment.joins(:states).group
("employee_departments.name").count
generates this in rails console
SELECT COUNT(*) AS count_all, employee_departments.name AS employee_departments_name FROM "employee_departments" INNER JOIN "tickets" ON "tickets"."employee_department_id" = "employee_departments"."id" INNER JOIN "states" ON "states"."id" = "tickets"."state_id" GROUP BY employee_departments.name
i used what was generated in the console in the following:
#department_count = EmployeeDepartment.find(:all,
:select => 'employee_departments.name AS employee_departments_name, COUNT(*) AS department_counter',
:joins => 'INNER JOIN tickets ON tickets.employee_department_id = employee_departments.id INNER JOIN states ON states.id = tickets.state_id',
:group => 'employee_departments.name')
Now the result in amcharts is:
var chartData = [{"department_counter":1,"employee_departments_name":"Academic Support"}];
In our Rails 3.2.13 app (Ruby 2.0.0 + Postgres on Heroku), we are often retreiving a large amount of Order data from an API, and then we need to update or create each order in our database, as well as the associations. A single order creates/updates itself plus approx. 10-15 associcated objects, and we are importing up to 500 orders at a time.
The below code works, but the problem is it's not at all efficient in terms of speed. Creating/updating 500 records takes approx. 1 minute and generates 6500+ db queries!
def add_details(shop, shopify_orders)
shopify_orders.each do |shopify_order|
order = Order.where(:order_id => shopify_order.id.to_s, :shop_id => shop.id).first_or_create
order.update_details(order,shopify_order,shop) #This calls update_attributes for the Order
ShippingLine.add_details(order, shopify_order.shipping_lines)
LineItem.add_details(order, shopify_order.line_items)
Taxline.add_details(order, shopify_order.tax_lines)
Fulfillment.add_details(order, shopify_order.fulfillments)
Note.add_details(order, shopify_order.note_attributes)
Discount.add_details(order, shopify_order.discount_codes)
billing_address = shopify_order.billing_address rescue nil
if !billing_address.blank?
BillingAddress.add_details(order, billing_address)
end
shipping_address = shopify_order.shipping_address rescue nil
if !shipping_address.blank?
ShippingAddress.add_details(order, shipping_address)
end
payment_details = shopify_order.payment_details rescue nil
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end
end
def update_details(order,shopify_order,shop)
order.update_attributes(
:order_name => shopify_order.name,
:order_created_at => shopify_order.created_at,
:order_updated_at => shopify_order.updated_at,
:status => Order.get_status(shopify_order),
:payment_status => shopify_order.financial_status,
:fulfillment_status => Order.get_fulfillment_status(shopify_order),
:payment_method => shopify_order.processing_method,
:gateway => shopify_order.gateway,
:currency => shopify_order.currency,
:subtotal_price => shopify_order.subtotal_price,
:subtotal_tax => shopify_order.total_tax,
:total_discounts => shopify_order.total_discounts,
:total_line_items_price => shopify_order.total_line_items_price,
:total_price => shopify_order.total_price,
:total_tax => shopify_order.total_tax,
:total_weight => shopify_order.total_weight,
:taxes_included => shopify_order.taxes_included,
:shop_id => shop.id,
:email => shopify_order.email,
:order_note => shopify_order.note
)
end
So as you can see, we are looping through each order, finding out if it exists or not (then either loading the existing Order or creating the new Order), and then calling update_attributes to pass in the details for the Order. After that we create or update each of the associations. Each associated model looks very similar to this:
class << self
def add_details(order, tax_lines)
tax_lines.each do |shopify_tax_line|
taxline = Taxline.find_or_create_by_order_id(:order_id => order.id)
taxline.update_details(shopify_tax_line)
end
end
end
def update_details(tax_line)
self.update_attributes(:price => tax_line.price, :rate => tax_line.rate, :title => tax_line.title)
end
I've looked into the activerecord-import gem but unfortunately it seems to be more geared towards creation of records in bulk and not update as we also require.
What is the best way that this can be improved for performance?
Many many thanks in advance.
UPDATE:
I came up with this slight improvement, which essentialy removes the call to update the newly created Orders (one query less per order).
def add_details(shop, shopify_orders)
shopify_orders.each do |shopify_order|
values = {:order_id => shopify_order.id.to_s, :shop_id => shop.id,
:order_name => shopify_order.name,
:order_created_at => shopify_order.created_at,
:order_updated_at => shopify_order.updated_at,
:status => Order.get_status(shopify_order),
:payment_status => shopify_order.financial_status,
:fulfillment_status => Order.get_fulfillment_status(shopify_order),
:payment_method => shopify_order.processing_method,
:gateway => shopify_order.gateway,
:currency => shopify_order.currency,
:subtotal_price => shopify_order.subtotal_price,
:subtotal_tax => shopify_order.total_tax,
:total_discounts => shopify_order.total_discounts,
:total_line_items_price => shopify_order.total_line_items_price,
:total_price => shopify_order.total_price,
:total_tax => shopify_order.total_tax,
:total_weight => shopify_order.total_weight,
:taxes_included => shopify_order.taxes_included,
:email => shopify_order.email,
:order_note => shopify_order.note}
get_order = Order.where(:order_id => shopify_order.id.to_s, :shop_id => shop.id)
if get_order.blank?
order = Order.create(values)
else
order = get_order.first
order.update_attributes(values)
end
ShippingLine.add_details(order, shopify_order.shipping_lines)
LineItem.add_details(order, shopify_order.line_items)
Taxline.add_details(order, shopify_order.tax_lines)
Fulfillment.add_details(order, shopify_order.fulfillments)
Note.add_details(order, shopify_order.note_attributes)
Discount.add_details(order, shopify_order.discount_codes)
billing_address = shopify_order.billing_address rescue nil
if !billing_address.blank?
BillingAddress.add_details(order, billing_address)
end
shipping_address = shopify_order.shipping_address rescue nil
if !shipping_address.blank?
ShippingAddress.add_details(order, shipping_address)
end
payment_details = shopify_order.payment_details rescue nil
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end
end
and for the associated objects:
class << self
def add_details(order, tax_lines)
tax_lines.each do |shopify_tax_line|
values = {:order_id => order.id,
:price => tax_line.price,
:rate => tax_line.rate,
:title => tax_line.title}
get_taxline = Taxline.where(:order_id => order.id)
if get_taxline.blank?
taxline = Taxline.create(values)
else
taxline = get_taxline.first
taxline.update_attributes(values)
end
end
end
end
Any better suggestions?
Try wrapping your entire code into a single database transaction. Since you're on Heroku it'll be a Postgres bottom-end. With that many update statements, you can probably benefit greatly by transacting them all at once, so your code executes quicker and basically just leaves a "queue" of 6500 statements to run on Postgres side as the server is able to dequeue them. Depending on the bottom end, you might have to transact into smaller chunks - but even transacting 100 at a time (and then close and re-open the transaction) would greatly improve throughput into Pg.
http://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html
http://www.postgresql.org/docs/9.2/static/sql-set-transaction.html
So before line 2 you'd add something like:
def add_details(shop, shopify_orders)
Order.transaction do
shopify_orders.each do |shopify_order|
And then at the very end of your method add another end:
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end //shopify_orders.each..
end //Order.transaction..
end //method
You can monkey-patch ActiveRecord like this:
class ActiveRecord::Base
#http://stackoverflow.com/questions/15317837/bulk-insert-records-into-active-record-table?lq=1
#https://gist.github.com/jackrg/76ade1724bd816292e4e
# "UPDATE THIS SET <list_of_column_assignments> FROM <table_name> THIS JOIN (VALUES (<csv1>, <csv2>,...) VALS ( <column_names> ) ON <list_of_primary_keys_comparison>"
def self.bulk_update(record_list)
pk = self.primary_key
raise "primary_key not found" unless pk.present?
raise "record_list not an Array of Hashes" unless record_list.is_a?(Array) && record_list.all? {|rec| rec.is_a? Hash }
return nil if record_list.empty?
result = nil
#test if every hash has primary keys, so we can JOIN
record_list.each { |r| raise "Primary Keys '#{self.primary_key.to_s}' not found on record: #{r}" unless hasAllPKs?(r) }
#list of primary keys comparison
pk_comparison_array = []
if (pk).is_a?(Array)
pk.each {|thiskey| pk_comparison_array << "THIS.#{thiskey} = VALS.#{thiskey}" }
else
pk_comparison_array << "THIS.#{pk} = VALS.#{pk}"
end
pk_comparison = pk_comparison_array.join(' AND ')
#SQL
(1..record_list.count).step(1000).each do |start|
key_list, value_list = convert_record_list(record_list[start-1..start+999])
#csv values
csv_vals = value_list.map {|v| "(#{v.join(", ")})" }.join(", ")
#column names
column_names = key_list.join(", ")
#list of columns assignments
columns_assign_array = []
key_list.each {|col|
unless inPK?(col)
columns_assign_array << "THIS.#{col} = VALS.#{col}"
end }
columns_assign = columns_assign_array.join(', ')
sql = "UPDATE THIS SET #{columns_assign} FROM #{self.table_name} THIS JOIN ( VALUES #{csv_vals} ) VALS ( #{column_names} ) ON ( #{pk_comparison} )"
result = self.connection.execute(sql)
return result if result<0
end
return result
end
def self.inPK?(str)
pk = self.primary_key
test = str.to_s
if pk.is_a?(Array)
(pk.include?(test))
else
(pk==test)
end
end
#test if given hash has primary keys included as hash keys and those keys are not empty
def self.hasAllPKs?(hash)
h = hash.stringify_keys
pk = self.primary_key
if pk.is_a?(Array)
(pk.all? {|k| h.key?(k) and h[k].present? })
else
h.key?(pk) and h[pk].present?
end
end
def self.convert_record_list(record_list)
# Build the list of keys
key_list = record_list.map(&:keys).flatten.map(&:to_s).uniq.sort
value_list = record_list.map do |rec|
list = []
key_list.each {|key| list << ActiveRecord::Base.connection.quote(rec[key] || rec[key.to_sym]) }
list
end
# If table has standard timestamps and they're not in the record list then add them to the record list
time = ActiveRecord::Base.connection.quote(Time.now)
for field_name in %w(created_at updated_at)
if self.column_names.include?(field_name) && !(key_list.include?(field_name))
key_list << field_name
value_list.each {|rec| rec << time }
end
end
return [key_list, value_list]
end
end
Then, you can generate a array of hashes containing your models attributes (including theirs primary keys) and do something like:
ActiveRecord::Base.transaction do
Model.bulk_update [ {attr1: val1, attr2: val2,...}, {attr1: val1, attr2: val2,...}, ... ]
end
It will be a single SQL command without Rails callbacks and validations.
For PostgreSQL, there are several issues that the above approach does not address:
You must specify an actual table, not just an alias, in the update target table.
You cannot repeat the target table in the FROM phrase. Since you are joining the target table to a VALUES table (hence there is only one table in the FROM phrase, you won't be able to use JOIN, you must instead use "WHERE ".
You don't get the same "free" casts in a VALUES table that you do in a simple "UPDATE" command, so you must cast date/timestamp values as such (#val_cast does this).
class ActiveRecord::Base
def self.update!(record_list)
raise ArgumentError "record_list not an Array of Hashes" unless record_list.is_a?(Array) && record_list.all? {|rec| rec.is_a? Hash }
return record_list if record_list.empty?
(1..record_list.count).step(1000).each do |start|
field_list, value_list = convert_record_list(record_list[start-1..start+999])
key_field = self.primary_key
non_key_fields = field_list - [%Q["#{self.primary_key}"], %Q["created_at"]]
columns_assign = non_key_fields.map {|field| "#{field} = #{val_cast(field)}"}.join(",")
value_table = value_list.map {|row| "(#{row.join(", ")})" }.join(", ")
sql = "UPDATE #{table_name} AS this SET #{columns_assign} FROM (VALUES #{value_table}) vals (#{field_list.join(", ")}) WHERE this.#{key_field} = vals.#{key_field}"
self.connection.update_sql(sql)
end
return record_list
end
def self.val_cast(field)
field = field.gsub('"', '')
if (column = columns.find{|c| c.name == field }).sql_type =~ /time|date/
"cast (vals.#{field} as #{column.sql_type})"
else
"vals.#{field}"
end
end
def self.convert_record_list(record_list)
# Build the list of fields
field_list = record_list.map(&:keys).flatten.map(&:to_s).uniq.sort
value_list = record_list.map do |rec|
list = []
field_list.each {|field| list << ActiveRecord::Base.connection.quote(rec[field] || rec[field.to_sym]) }
list
end
# If table has standard timestamps and they're not in the record list then add them to the record list
time = ActiveRecord::Base.connection.quote(Time.now)
for field_name in %w(created_at updated_at)
if self.column_names.include?(field_name) && !(field_list.include?(field_name))
field_list << field_name
value_list.each {|rec| rec << time }
end
end
field_list.map! {|field| %Q["#{field}"] }
return [field_list, value_list]
end
end
I am parsing HTML into an array as shown below. I can get the data into the array and then push the data into the database. One row of the array data is:
{:address=>"6222 Lodgepole Dr", :members=>["Diana L Dillard", "Kemberly J Williams", "George S Williams Iii"]}
The database result is:
Master Neighbor Name
7545 XYZ Dr --- - 8283 Southern Watch Pl --- - - Diana L Dillard - Kemberly J Williams - George S Williams Iii
These are my questions:
How do I get the Address.create loop to only insert the first member from the array, in this example Diana L Dillard, into the "name" field instead of all the names? Ideally I want to create one field for the first and then another for all others.
Why do I get these dashes "---" inserted with my data? I do not want them.
This is my current code:
url = "XYZ"
doc = Nokogiri::HTML(open(url))
results = []
# parse .single tags
doc.css('.single').each do |single_div|
res = {}
res[:address] = single_div.at_css('span.address').text
res[:members] = single_div.css('li.basic_info').collect{|el| el.text.strip}
results << res
end
results.each do|address|
puts "#{address}: #{members}"
Address.create(:neighboradd => "#{address}", :master => '7545 XYZ Dr', :name => "{members}")
end
This
results.each do |address|
puts "#{address}: #{members}"
Address.create(
:neighboradd => "#{address}",
:master => '7545 XYZ Dr',
:name => "{members}"
)
end
Should be something like
results.each do |res|
puts "#{res[:address]}: #{res[:members]}"
Address.create(
:neighboradd => res[:address],
:master => '7545 XYZ Dr',
:name => res[:members].join(', ') # this should be "#{res[members]}"? (with "#")
)
end
The --- is because you were using the whole hash for :neighboradd.
If you want only the first member:
:name => res[:members].first