Related
I am working on requirements we have data in hash around 100+ keys. we need to generate CSV file as per user-defined header with some transformation, we may end up having 100+ template
Main changes will be
1) Change column name such as Fname –> First name
2) Data transformation like Full name – > First name + Last name (adding 2 column)
3) Fixing the position of a column – Fname should be at 35 positions etc.
please suggest is it possible to define declarative way or any gem available. Can you let me know any design pattern we can apply here?
Some sample scenarios
I have input like this with many columns (100+)
[ {:employee_id=>"001", :first_name=>"John",:last_name=>"Dee" :date_of_birth=>"10/10/1983", :salary=>"100000",:bounus =>"50000",......},
{:employee_id=>"002", :first_name=>"Alex",:last_name=>"Peck" :date_of_birth=>"11/01/1988", :salary=>"120000",:bounus =>"70000", .........},
]
Some customer need CSV as
Employee ID, First Name, Last Name, Date of birth, Salary, Bonus
001,John,Dee,10/10/1983,100000,50000,...
002,Alex,Peck,11/01/1988,120000,70000,...
Others (only header change)
ID, FName, LName, Dob, Salary, Bounus
001,John,Dee,10/10/1983,100000,50000,...
002,Alex,Peck,11/01/1988,120000,70000,...
another (merge of colum FName, LName -> Fullname)
ID, Fullname, Dob, Salary, Bounus
001,John Dee,10/10/1983,100000,50000,...
002,Alex Peck,11/01/1988,120000,70000,...
anothers (merge of column Salary, Bonus -> Salary+ Bonus)
ID, FName, LName, Dob, Salary
001,John,Dee,10/10/1983,150000,...
002,Alex,Peck,11/01/1988,190000,...
anothers ( column order changed also insted of Dob need age)
FName, LName, ID, age, Salary
John,Dee,001,36,150000,...
Alex,Peck,003,32,190000,...
Like many variations with the same input
Thanks for help
What you need is the presenter design pattern.
Your controller will request the data and store it in a local variable, and then your will have to load a presenter for your client passing it the data variable.
In response you'll get the final CSV to return to the client.
Let's say you clients have uniq codes, so that a Client model instance has a code attribute which is a string.
So your controller would looks like this:
app/controllers/exports_controller.rb
class ExportsController < ApplicationController
def export
data = MyService.fetchData # <== data contains the data you gave as an example
# Gets the right presenter, initialise it, and build the CSV
csv = PresenterFactory.for(current_client).new(data).present
respond_to do |format|
format.html
format.csv { send_data csv, filename: "export-name-for-#{current_client.code}.csv" }
end
end
end
The PresenterFactory class would be something like that:
app/models/presenter_factory.rb
class PresenterFactory
def self.for(client)
# For client with code "ABCD" it will return Presenters::Abcd class
"Presenters::#{client.code.capitalize}".constantize
end
end
The factory return the client's presenter class
And here is an example for a client's presenter class, for a client having the code ABCD:
app/models/presenters/abcd.rb
module Presenters
class Abcd
def initialize(data)
#data = data
end
def present
CSV.generate(headers: true) do |csv|
# Here is the client's specific CSV header
csv << [
'Employee ID',
'First Name',
# ...
]
#data.each do |row|
# Here is the client's specific CSV row
csv << [
row[:employee_id],
row[:first_name],
# ...
]
end
end
end
end
end
You can achieve your objective by constructing a transformation hash whose keys are the names of the columns in the desired CSV file, in order, and whose values are procs, which when called with an argument equal to an element of the given array of hashes, returns an element to be written in a row of the CSV file in the column corresponding to the key.
Code
require 'csv'
def construct_csv(fname, arr, transform)
CSV.open(fname, "wb") do |csv|
keys = transform.keys
csv << keys
arr.each { |h| csv << keys.map { |k| transform[k].call(h) } }
end
end
Examples
I will now illustrate how this method is used with various transformations.
Common data
arr = [{:employee_id=>"001", :first_name=>"John", :last_name=>"Dee",
:date_of_birth=>"10/10/1983", :salary=>"100000", :bonus=>"50000" },
{:employee_id=>"002", :first_name=>"Alex", :last_name=>"Peck",
:date_of_birth=>"11/01/1988", :salary=>"120000", :bonus=>"70000" }]
FName = 'temp.csv'
Write a CSV file with the same keys, in the same order, and the same values
keys = arr.first.keys
#=> [:employee_id, :first_name, :last_name, :date_of_birth, :salary, :bonus]
transform = keys.each_with_object({}) { |k,g| g[k] = ->(h) { h[k] } }
#=> {:employee_id=>#<Proc:0x00005bd270a0e710#(irb):451 (lambda)>,
# :first_name=>#<Proc:0x00005bd270a13260#(irb):451 (lambda)>,
# ...
# :bonus=>#<Proc:0x00005bd270a19cc8#(irb):451 (lambda)>}
construct_csv(FName, arr, transform)
Let's see what was written.
puts File.read(FName)
employee_id,first_name,last_name,date_of_birth,salary,bonus
001,John,Dee,10/10/1983,100000,50000
002,Alex,Peck,11/01/1988,120000,70000
Write a CSV file with the columns reordered1
col_order = [:last_name, :first_name, :employee_id, :salary, :bonus,
:date_of_birth]
keys = arr.first.keys
order_map = col_order.each_with_object({}) { |k,h| h[k] = keys.index(k) }
#=> {:last_name=>2, :first_name=>1, :employee_id=>0, :salary=>4,
# :bonus=>5, :date_of_birth=>3}
transform = col_order.each_with_object({}) { |k,g|
g[k] = ->(h) { h[keys[order_map[k]]] } }
#=> {:last_name=>#<Proc:0x00005bd270f8e5a0#(irb):511 (lambda)>,
# :first_name=>#<Proc:0x00005bd270f8e550#(irb):511 (lambda)>,
# ...
# :date_of_birth=>#<Proc:0x00005bd270f8e3c0#(irb):511 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
last_name,first_name,employee_id,salary,bonus,date_of_birth
Dee,John,001,100000,50000,10/10/1983
Peck,Alex,002,120000,70000,11/01/1988
Write a CSV file with a subset of keys, renamed and reordered
keymap = { :FirstName=>:first_name, :LastName=>:last_name, :ID=>:employee_id,
:Salary=>:salary, :Bonus=>:bonus }
transform = keymap.each_with_object({}) { |(new,old),g| g[new] = ->(h) { h[old] } }
#=> {:FirstName=>#<Proc:0x00005bd270d50298#(irb):391 (lambda)>,
# :LastName=>#<Proc:0x00005bd270d50220#(irb):391 (lambda)>,
# ...
# :Bonus=>#<Proc:0x00005bd270d830f8#(irb):391 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
FirstName,LastName,ID,Salary,Bonus
John,Dee,001,100000,50000
Alex,Peck,002,120000,70000
Write a CSV file after removing keys and adding keys whose values are computed
keys_to_remove = [:first_name, :last_name]
keys_to_add = [:full_name, :compensation]
keys = arr.first.keys + keys_to_add - keys_to_remove
#=> [:employee_id, :date_of_birth, :salary, :bonus, :full_name,
# :compensation]
transform = keys.each_with_object({}) do |k,h|
h[k] =
case k
when :full_name
->(h) { h[:first_name] + " " + h[:last_name] }
when :compensation
->(h) { h[:salary].to_i + h[:bonus].to_i }
else
->(h) { h[k] }
end
end
#=> {:employee_id=>#<Proc:0x00005bd271001000#(irb):501 (lambda)>,
# :date_of_birth=>#<Proc:0x00005bd271000f88#(irb):501 (lambda)>,
# :salary=>#<Proc:0x00005bd271000f10#(irb):501 (lambda)>,
# :bonus=>#<Proc:0x00005bd271000ec0#(irb):501 (lambda)>,
# :full_name=>#<Proc:0x00005bd271000e20#(irb):497 (lambda)>,
# :compensation=>#<Proc:0x00005bd271000dd0#(irb):499 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
employee_id,date_of_birth,salary,bonus,full_name,compensation
001,10/10/1983,100000,50000,John Dee,150000
002,11/01/1988,120000,70000,Alex Peck,190000
1. I don't understand the reason for doing this but it was mentioned as a possible requirement.
I have processed a log file and created a data structure of below format
values = [ {:VM=>"VM_US_OLA_1"}
{:id=>"OLFTB51", :datum=>{"LAP"=>"6.93817", "YCC_OWER"=>"1.0391"}}
{:id=>"OLFTB10", :datum=>{"LAP_2"=>"2.72646", "CFG_ON"=>"15.9489746", "YCC_ON"=>".401794"}}
{:VM=>"VM_ASIA_FLO_1"}
{:id=>"LOPMLAP", :datum=>{"LAP"=>"1.81048584", "FM_ON"=>".00"} ]
values is an array.
I'm trying to create a spreadsheet of below format where only the VM column gets highlighted in green and for every VM set, a blank line highlighted in 'yellow' should be inserted. I tried multiple approaches and went thru the axlsx documentation too but unable to get the desired format.
My excel snippet below:
require 'axlsx'
p = Axlsx::Package.new
p.workbook.add_worksheet(:name => "Statistics") do |sheet|
style1 = sheet.styles.add_style(:bg_color => "EF0920", :fg_color => "FFFFFF", b:true)
style2 = sheet.styles.add_style(:bg_color => "00FF00", :fg_color => "FFFFFF", b:true)
sheet.add_row ["VM", "NAME", "DETAILS", "OCC"], :style => style1
values.each do |val|
sheet.add_row [ val[:VM], val[:id] ], :style =>[style2, nil]
val[:datum].each do |k, v|
sheet << ["", "", k, v]
end
end
sheet.add_row
end
p.serialize 'Stat.xlsx'
Any suggestions should be really helpful here, many thanks.
Expected Output
Current Output
Adding Log file and my code
Log
---- vm name ----
VM_US_OLA_1
OLFTB51
OWNER IN_GB
------------------------------ ----------
LAP 6.93817
YCC_OWER 1.0391
=========================================
---- vm name ----
OLFTB10
OWNER IN_GB
------------------------------ ----------
LAP_2 2.7266846
CFG_ON 15.9489746
YCC_ON .401794
=========================================
---- vm name ----
VM_ASIA_FLO_1
LOPMLAP
OWNER IN_GB
------------------------------ ----------
LAP 1.81048584
FM_ON .00
=========================================
---- vm name ----
INGTY_2
OWNER IN_GB
------------------------------ ----------
=========================================
so on of the same format
Code to process the logs
require 'csv'
values = []
total = File.read("final.log")
total.each_line do |line|
line.strip!
next if line.empty?
next if line.include?('selected') || line.include?('IN_GB')
next if ['-','='].include? line[0]
parts = line.split ' '
if parts.size == 1 and line.start_with?('size')
values[current += 1] = {vm: line.strip}
next
elsif parts.size == 1 and parts = /^(?!.*size_).*$/
values[current += 1] = {id: line, datum: {}}
next
end
parts.each_cons(2) do |key, value|
values[current][:datum][key] = value
end
end
puts values
The problem is with the structure of your data. Try to change to have the values of your array in this structure
{:VM=>"VM_US_OLA_1", :id=>"OLFTB51", :datum=>{"LAP"=>"6.93817", "YCC_OWER"=>"1.0391"}
What's happening is VM is one record in your array, and the other data is another record in the array, you need to merge them so you access them in the same loop iteration
EDIT
I guess how we parse the log file is a bit tricky, so I am going to leave it but I am sure that this part I am going to introduce can be done in the log file parsing
First, we can change the structure of the array like that (before p.workbook.add_worksheet)
dataset = []
values.each_with_index do |value|
if value[:VM].present? # or :vm, not sure about the key
dataset << value
dataset.last[:data] = []
else
dataset.last[:data] << value
end
end
Then this loop: values.each do |val|
Can be changed to dataset.each do |val|
Then you can continue with the rest of your logic:
sheet.add_row [ val[:VM], val[:data][0][:id] ], :style =>[style2, nil]
val[data].each do |record|
record[:datanum].each do |k, v|
The problem with your previous code is, you used to add a new row sheet.add_row at the end of each iteration, so this was messing things up as you were expecting the data you need to be just right after your VM but a new line has already been inserted
In our Rails 3.2.13 app (Ruby 2.0.0 + Postgres on Heroku), we are often retreiving a large amount of Order data from an API, and then we need to update or create each order in our database, as well as the associations. A single order creates/updates itself plus approx. 10-15 associcated objects, and we are importing up to 500 orders at a time.
The below code works, but the problem is it's not at all efficient in terms of speed. Creating/updating 500 records takes approx. 1 minute and generates 6500+ db queries!
def add_details(shop, shopify_orders)
shopify_orders.each do |shopify_order|
order = Order.where(:order_id => shopify_order.id.to_s, :shop_id => shop.id).first_or_create
order.update_details(order,shopify_order,shop) #This calls update_attributes for the Order
ShippingLine.add_details(order, shopify_order.shipping_lines)
LineItem.add_details(order, shopify_order.line_items)
Taxline.add_details(order, shopify_order.tax_lines)
Fulfillment.add_details(order, shopify_order.fulfillments)
Note.add_details(order, shopify_order.note_attributes)
Discount.add_details(order, shopify_order.discount_codes)
billing_address = shopify_order.billing_address rescue nil
if !billing_address.blank?
BillingAddress.add_details(order, billing_address)
end
shipping_address = shopify_order.shipping_address rescue nil
if !shipping_address.blank?
ShippingAddress.add_details(order, shipping_address)
end
payment_details = shopify_order.payment_details rescue nil
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end
end
def update_details(order,shopify_order,shop)
order.update_attributes(
:order_name => shopify_order.name,
:order_created_at => shopify_order.created_at,
:order_updated_at => shopify_order.updated_at,
:status => Order.get_status(shopify_order),
:payment_status => shopify_order.financial_status,
:fulfillment_status => Order.get_fulfillment_status(shopify_order),
:payment_method => shopify_order.processing_method,
:gateway => shopify_order.gateway,
:currency => shopify_order.currency,
:subtotal_price => shopify_order.subtotal_price,
:subtotal_tax => shopify_order.total_tax,
:total_discounts => shopify_order.total_discounts,
:total_line_items_price => shopify_order.total_line_items_price,
:total_price => shopify_order.total_price,
:total_tax => shopify_order.total_tax,
:total_weight => shopify_order.total_weight,
:taxes_included => shopify_order.taxes_included,
:shop_id => shop.id,
:email => shopify_order.email,
:order_note => shopify_order.note
)
end
So as you can see, we are looping through each order, finding out if it exists or not (then either loading the existing Order or creating the new Order), and then calling update_attributes to pass in the details for the Order. After that we create or update each of the associations. Each associated model looks very similar to this:
class << self
def add_details(order, tax_lines)
tax_lines.each do |shopify_tax_line|
taxline = Taxline.find_or_create_by_order_id(:order_id => order.id)
taxline.update_details(shopify_tax_line)
end
end
end
def update_details(tax_line)
self.update_attributes(:price => tax_line.price, :rate => tax_line.rate, :title => tax_line.title)
end
I've looked into the activerecord-import gem but unfortunately it seems to be more geared towards creation of records in bulk and not update as we also require.
What is the best way that this can be improved for performance?
Many many thanks in advance.
UPDATE:
I came up with this slight improvement, which essentialy removes the call to update the newly created Orders (one query less per order).
def add_details(shop, shopify_orders)
shopify_orders.each do |shopify_order|
values = {:order_id => shopify_order.id.to_s, :shop_id => shop.id,
:order_name => shopify_order.name,
:order_created_at => shopify_order.created_at,
:order_updated_at => shopify_order.updated_at,
:status => Order.get_status(shopify_order),
:payment_status => shopify_order.financial_status,
:fulfillment_status => Order.get_fulfillment_status(shopify_order),
:payment_method => shopify_order.processing_method,
:gateway => shopify_order.gateway,
:currency => shopify_order.currency,
:subtotal_price => shopify_order.subtotal_price,
:subtotal_tax => shopify_order.total_tax,
:total_discounts => shopify_order.total_discounts,
:total_line_items_price => shopify_order.total_line_items_price,
:total_price => shopify_order.total_price,
:total_tax => shopify_order.total_tax,
:total_weight => shopify_order.total_weight,
:taxes_included => shopify_order.taxes_included,
:email => shopify_order.email,
:order_note => shopify_order.note}
get_order = Order.where(:order_id => shopify_order.id.to_s, :shop_id => shop.id)
if get_order.blank?
order = Order.create(values)
else
order = get_order.first
order.update_attributes(values)
end
ShippingLine.add_details(order, shopify_order.shipping_lines)
LineItem.add_details(order, shopify_order.line_items)
Taxline.add_details(order, shopify_order.tax_lines)
Fulfillment.add_details(order, shopify_order.fulfillments)
Note.add_details(order, shopify_order.note_attributes)
Discount.add_details(order, shopify_order.discount_codes)
billing_address = shopify_order.billing_address rescue nil
if !billing_address.blank?
BillingAddress.add_details(order, billing_address)
end
shipping_address = shopify_order.shipping_address rescue nil
if !shipping_address.blank?
ShippingAddress.add_details(order, shipping_address)
end
payment_details = shopify_order.payment_details rescue nil
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end
end
and for the associated objects:
class << self
def add_details(order, tax_lines)
tax_lines.each do |shopify_tax_line|
values = {:order_id => order.id,
:price => tax_line.price,
:rate => tax_line.rate,
:title => tax_line.title}
get_taxline = Taxline.where(:order_id => order.id)
if get_taxline.blank?
taxline = Taxline.create(values)
else
taxline = get_taxline.first
taxline.update_attributes(values)
end
end
end
end
Any better suggestions?
Try wrapping your entire code into a single database transaction. Since you're on Heroku it'll be a Postgres bottom-end. With that many update statements, you can probably benefit greatly by transacting them all at once, so your code executes quicker and basically just leaves a "queue" of 6500 statements to run on Postgres side as the server is able to dequeue them. Depending on the bottom end, you might have to transact into smaller chunks - but even transacting 100 at a time (and then close and re-open the transaction) would greatly improve throughput into Pg.
http://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html
http://www.postgresql.org/docs/9.2/static/sql-set-transaction.html
So before line 2 you'd add something like:
def add_details(shop, shopify_orders)
Order.transaction do
shopify_orders.each do |shopify_order|
And then at the very end of your method add another end:
if !payment_details.blank?
PaymentDetail.add_details(order, payment_details)
end
end //shopify_orders.each..
end //Order.transaction..
end //method
You can monkey-patch ActiveRecord like this:
class ActiveRecord::Base
#http://stackoverflow.com/questions/15317837/bulk-insert-records-into-active-record-table?lq=1
#https://gist.github.com/jackrg/76ade1724bd816292e4e
# "UPDATE THIS SET <list_of_column_assignments> FROM <table_name> THIS JOIN (VALUES (<csv1>, <csv2>,...) VALS ( <column_names> ) ON <list_of_primary_keys_comparison>"
def self.bulk_update(record_list)
pk = self.primary_key
raise "primary_key not found" unless pk.present?
raise "record_list not an Array of Hashes" unless record_list.is_a?(Array) && record_list.all? {|rec| rec.is_a? Hash }
return nil if record_list.empty?
result = nil
#test if every hash has primary keys, so we can JOIN
record_list.each { |r| raise "Primary Keys '#{self.primary_key.to_s}' not found on record: #{r}" unless hasAllPKs?(r) }
#list of primary keys comparison
pk_comparison_array = []
if (pk).is_a?(Array)
pk.each {|thiskey| pk_comparison_array << "THIS.#{thiskey} = VALS.#{thiskey}" }
else
pk_comparison_array << "THIS.#{pk} = VALS.#{pk}"
end
pk_comparison = pk_comparison_array.join(' AND ')
#SQL
(1..record_list.count).step(1000).each do |start|
key_list, value_list = convert_record_list(record_list[start-1..start+999])
#csv values
csv_vals = value_list.map {|v| "(#{v.join(", ")})" }.join(", ")
#column names
column_names = key_list.join(", ")
#list of columns assignments
columns_assign_array = []
key_list.each {|col|
unless inPK?(col)
columns_assign_array << "THIS.#{col} = VALS.#{col}"
end }
columns_assign = columns_assign_array.join(', ')
sql = "UPDATE THIS SET #{columns_assign} FROM #{self.table_name} THIS JOIN ( VALUES #{csv_vals} ) VALS ( #{column_names} ) ON ( #{pk_comparison} )"
result = self.connection.execute(sql)
return result if result<0
end
return result
end
def self.inPK?(str)
pk = self.primary_key
test = str.to_s
if pk.is_a?(Array)
(pk.include?(test))
else
(pk==test)
end
end
#test if given hash has primary keys included as hash keys and those keys are not empty
def self.hasAllPKs?(hash)
h = hash.stringify_keys
pk = self.primary_key
if pk.is_a?(Array)
(pk.all? {|k| h.key?(k) and h[k].present? })
else
h.key?(pk) and h[pk].present?
end
end
def self.convert_record_list(record_list)
# Build the list of keys
key_list = record_list.map(&:keys).flatten.map(&:to_s).uniq.sort
value_list = record_list.map do |rec|
list = []
key_list.each {|key| list << ActiveRecord::Base.connection.quote(rec[key] || rec[key.to_sym]) }
list
end
# If table has standard timestamps and they're not in the record list then add them to the record list
time = ActiveRecord::Base.connection.quote(Time.now)
for field_name in %w(created_at updated_at)
if self.column_names.include?(field_name) && !(key_list.include?(field_name))
key_list << field_name
value_list.each {|rec| rec << time }
end
end
return [key_list, value_list]
end
end
Then, you can generate a array of hashes containing your models attributes (including theirs primary keys) and do something like:
ActiveRecord::Base.transaction do
Model.bulk_update [ {attr1: val1, attr2: val2,...}, {attr1: val1, attr2: val2,...}, ... ]
end
It will be a single SQL command without Rails callbacks and validations.
For PostgreSQL, there are several issues that the above approach does not address:
You must specify an actual table, not just an alias, in the update target table.
You cannot repeat the target table in the FROM phrase. Since you are joining the target table to a VALUES table (hence there is only one table in the FROM phrase, you won't be able to use JOIN, you must instead use "WHERE ".
You don't get the same "free" casts in a VALUES table that you do in a simple "UPDATE" command, so you must cast date/timestamp values as such (#val_cast does this).
class ActiveRecord::Base
def self.update!(record_list)
raise ArgumentError "record_list not an Array of Hashes" unless record_list.is_a?(Array) && record_list.all? {|rec| rec.is_a? Hash }
return record_list if record_list.empty?
(1..record_list.count).step(1000).each do |start|
field_list, value_list = convert_record_list(record_list[start-1..start+999])
key_field = self.primary_key
non_key_fields = field_list - [%Q["#{self.primary_key}"], %Q["created_at"]]
columns_assign = non_key_fields.map {|field| "#{field} = #{val_cast(field)}"}.join(",")
value_table = value_list.map {|row| "(#{row.join(", ")})" }.join(", ")
sql = "UPDATE #{table_name} AS this SET #{columns_assign} FROM (VALUES #{value_table}) vals (#{field_list.join(", ")}) WHERE this.#{key_field} = vals.#{key_field}"
self.connection.update_sql(sql)
end
return record_list
end
def self.val_cast(field)
field = field.gsub('"', '')
if (column = columns.find{|c| c.name == field }).sql_type =~ /time|date/
"cast (vals.#{field} as #{column.sql_type})"
else
"vals.#{field}"
end
end
def self.convert_record_list(record_list)
# Build the list of fields
field_list = record_list.map(&:keys).flatten.map(&:to_s).uniq.sort
value_list = record_list.map do |rec|
list = []
field_list.each {|field| list << ActiveRecord::Base.connection.quote(rec[field] || rec[field.to_sym]) }
list
end
# If table has standard timestamps and they're not in the record list then add them to the record list
time = ActiveRecord::Base.connection.quote(Time.now)
for field_name in %w(created_at updated_at)
if self.column_names.include?(field_name) && !(field_list.include?(field_name))
field_list << field_name
value_list.each {|rec| rec << time }
end
end
field_list.map! {|field| %Q["#{field}"] }
return [field_list, value_list]
end
end
What is the best way to achieve the following, I have following array of actions under ABC
ABC:-
ABC:Actions,
ABC:Actions:ADD-DATA,
ABC:Actions:TRANSFER-DATA,
ABC:Actions:EXPORT,
ABC:Actions:PRINT,
ABC:Detail,
ABC:Detail:OVERVIEW,
ABC:Detail:PRODUCT-DETAIL,
ABC:Detail:EVENT-LOG,
ABC:Detail:ORDERS
I want to format this as:
ABC =>{Actions=> [ADD-DATA,TRANSFER-DATA,EXPORT,PRINT], Detail => [Overview, Product-detail, event-log,orders]}
There's probably a ton of ways to do it but here's one:
a = ["ABC:Actions",
"ABC:Actions:ADD-DATA",
"ABC:Actions:TRANSFER-DATA",
"ABC:Actions:EXPORT",
"ABC:Actions:PRINT",
"ABC:Detail",
"ABC:Detail:OVERVIEW",
"ABC:Detail:PRODUCT-DETAIL",
"ABC:Detail:EVENT-LOG",
"ABC:Detail:ORDERS"]
a.map { |action| action.split(":") }.inject({}) do |m, s|
m[s.at(0)] ||= {}
m[s.at(0)][s.at(1)] ||= [] if s.at(1)
m[s.at(0)][s.at(1)] << s.at(2) if s.at(2)
m
end
The map call returns an array where each of the strings in the original array have been split into an array of elements that were separated by :. For example [["ABC","Actions","ADD-DATA"] ... ]
The inject call then builds up a hash by going through each of these "split" arrays. It creates a mapping for the first element, if one doesn't already exist, to an empty hash, e.g. "ABC" => {}. Then it creates a mapping in that hash for the second element, if one doesn't already exist, to an empty array, e.g. "ABC" => { "Detail" => [] }. Then it adds the third element to that array to give something like "ABC" => { "Detail" => ["OVERVIEW"] }. Then it goes onto the next "split" array and adds that to the hash too in the same way.
I will do this as below :
a = ["ABC:Actions",
"ABC:Actions:ADD-DATA",
"ABC:Actions:TRANSFER-DATA",
"ABC:Actions:EXPORT",
"ABC:Actions:PRINT",
"ABC:Detail",
"ABC:Detail:OVERVIEW",
"ABC:Detail:PRODUCT-DETAIL",
"ABC:Detail:EVENT-LOG",
"ABC:Detail:ORDERS"]
m = a.map{|i| i.split(":")[1..-1]}
# => [["Actions"],
# ["Actions", "ADD-DATA"],
# ["Actions", "TRANSFER-DATA"],
# ["Actions", "EXPORT"],
# ["Actions", "PRINT"],
# ["Detail"],
# ["Detail", "OVERVIEW"],
# ["Detail", "PRODUCT-DETAIL"],
# ["Detail", "EVENT-LOG"],
# ["Detail", "ORDERS"]]
m.each_with_object(Hash.new([])){|(i,j),ob| ob[i] = ob[i] + [j] unless j.nil? }
# => {"Actions"=>["ADD-DATA", "TRANSFER-DATA", "EXPORT", "PRINT"],
# "Detail"=>["OVERVIEW", "PRODUCT-DETAIL", "EVENT-LOG", "ORDERS"]}
It was just interesting to do it with group_by :)
a = ['ABC:Actions',
'ABC:Actions:ADD-DATA',
'ABC:Actions:TRANSFER-DATA',
'ABC:Actions:EXPORT',
'ABC:Actions:PRINT',
'ABC:Detail',
'ABC:Detail:OVERVIEW',
'ABC:Detail:PRODUCT-DETAIL',
'ABC:Detail:EVENT-LOG',
'ABC:Detail:ORDERS']
result = a.map { |action| action.split(":") }.group_by(&:shift)
result.each do |k1,v1|
result[k1] = v1.group_by(&:shift)
result[k1].each { |k2,v2| result[k1][k2] = v2.flatten }
end
p result
{"ABC"=>{"Actions"=>["ADD-DATA", "TRANSFER-DATA", "EXPORT", "PRINT"], "Detail"=>["OVERVIEW", "PRODUCT-DETAIL", "EVENT-LOG", "ORDERS"]}}
Now I'm fetching data from another url...
Here is my code:
require 'rubygems'
require 'nokogiri'
html = page.body
doc = Nokogiri::HTML(html)
doc.encoding = 'utf-8'
rows = doc.search('//table[#id = "MainContent_GridView1"]//tr')
#details = rows.collect do |row|
detail = {}
[
[:car, 'td[1]/text()'],
[:article, 'td[2]/text()'],
[:group, 'td[3]/text()'],
[:price, 'td[4]/text()'],
].each do |name, xpath|
detail[name] = row.at_xpath(xpath).to_s.strip
end
detail
end
#details
I tried to do it via array, not a hash. But I get a lot of errors...
Are there any ideas?
I need it for another method...
also i set data (this result hash) to another car here:
oem_art = []
#constr_num.each do |o|
as_oem = get_from_as_oem(o.ARL_SEARCH_NUMBER)
if as_oem.present?
oem_art << as_oem
end
end
#oem_art = oem_art.to_a.uniq
Do you just want to change a hash into an array? If so, just use the to_a method on your hash.
hash = {:a => "something", :b => "something else"}
array = hash.to_a
array.inspect #=> [[:a, "something"], [:b, "something else"]]
It looks like you're looking for something like hash['key'] to hash.key in Ruby
The Hash Class doesn't support .key notation by default, OpenStruct creates an Object from the Hash so you can use dot notation to access the properties. Overall it's basically just syntactic sugar with overhead.
Suggested code (from linked answer)
>> require 'ostruct'
=> []
>> foo = {'bar'=>'baz'}
=> {"bar"=>"baz"}
>> foo_obj = OpenStruct.new foo
=> #<OpenStruct bar="baz">
>> foo_obj.bar
=> "baz"
So in your example, you could do:
# Initialised somewhere
require 'ostruct'
DETAIL_INDICES = {
:car => 1,
:article => 2,
:group => 3,
:price => 4,
}
# ** SNIP **
#details = rows.map do |row|
DETAIL_INDICES.inject({}) do |h,(k,v)|
h.merge(k => row.at_xpath("td[#{v}]/text()").to_s.strip)
end
end.collect { |hash| OpenStruct.new hash }
#details.each do |item|
puts item.car
end
Of course if performance is a concern you can merge your map&collect (They are the same), but this is just a minor separation for basic semantic differences, although I usually only use map for consistency, so feel free to choose yourself :)
EDIT -- Additional code from your edit simplified
#oem_art = #constr_num.select do |item|
as_oem = get_from_as_oem(item.ARL_SEARCH_NUMBER)
as_oem.present?
end
puts #oem_art.uniq