I am working on requirements we have data in hash around 100+ keys. we need to generate CSV file as per user-defined header with some transformation, we may end up having 100+ template
Main changes will be
1) Change column name such as Fname –> First name
2) Data transformation like Full name – > First name + Last name (adding 2 column)
3) Fixing the position of a column – Fname should be at 35 positions etc.
please suggest is it possible to define declarative way or any gem available. Can you let me know any design pattern we can apply here?
Some sample scenarios
I have input like this with many columns (100+)
[ {:employee_id=>"001", :first_name=>"John",:last_name=>"Dee" :date_of_birth=>"10/10/1983", :salary=>"100000",:bounus =>"50000",......},
{:employee_id=>"002", :first_name=>"Alex",:last_name=>"Peck" :date_of_birth=>"11/01/1988", :salary=>"120000",:bounus =>"70000", .........},
]
Some customer need CSV as
Employee ID, First Name, Last Name, Date of birth, Salary, Bonus
001,John,Dee,10/10/1983,100000,50000,...
002,Alex,Peck,11/01/1988,120000,70000,...
Others (only header change)
ID, FName, LName, Dob, Salary, Bounus
001,John,Dee,10/10/1983,100000,50000,...
002,Alex,Peck,11/01/1988,120000,70000,...
another (merge of colum FName, LName -> Fullname)
ID, Fullname, Dob, Salary, Bounus
001,John Dee,10/10/1983,100000,50000,...
002,Alex Peck,11/01/1988,120000,70000,...
anothers (merge of column Salary, Bonus -> Salary+ Bonus)
ID, FName, LName, Dob, Salary
001,John,Dee,10/10/1983,150000,...
002,Alex,Peck,11/01/1988,190000,...
anothers ( column order changed also insted of Dob need age)
FName, LName, ID, age, Salary
John,Dee,001,36,150000,...
Alex,Peck,003,32,190000,...
Like many variations with the same input
Thanks for help
What you need is the presenter design pattern.
Your controller will request the data and store it in a local variable, and then your will have to load a presenter for your client passing it the data variable.
In response you'll get the final CSV to return to the client.
Let's say you clients have uniq codes, so that a Client model instance has a code attribute which is a string.
So your controller would looks like this:
app/controllers/exports_controller.rb
class ExportsController < ApplicationController
def export
data = MyService.fetchData # <== data contains the data you gave as an example
# Gets the right presenter, initialise it, and build the CSV
csv = PresenterFactory.for(current_client).new(data).present
respond_to do |format|
format.html
format.csv { send_data csv, filename: "export-name-for-#{current_client.code}.csv" }
end
end
end
The PresenterFactory class would be something like that:
app/models/presenter_factory.rb
class PresenterFactory
def self.for(client)
# For client with code "ABCD" it will return Presenters::Abcd class
"Presenters::#{client.code.capitalize}".constantize
end
end
The factory return the client's presenter class
And here is an example for a client's presenter class, for a client having the code ABCD:
app/models/presenters/abcd.rb
module Presenters
class Abcd
def initialize(data)
#data = data
end
def present
CSV.generate(headers: true) do |csv|
# Here is the client's specific CSV header
csv << [
'Employee ID',
'First Name',
# ...
]
#data.each do |row|
# Here is the client's specific CSV row
csv << [
row[:employee_id],
row[:first_name],
# ...
]
end
end
end
end
end
You can achieve your objective by constructing a transformation hash whose keys are the names of the columns in the desired CSV file, in order, and whose values are procs, which when called with an argument equal to an element of the given array of hashes, returns an element to be written in a row of the CSV file in the column corresponding to the key.
Code
require 'csv'
def construct_csv(fname, arr, transform)
CSV.open(fname, "wb") do |csv|
keys = transform.keys
csv << keys
arr.each { |h| csv << keys.map { |k| transform[k].call(h) } }
end
end
Examples
I will now illustrate how this method is used with various transformations.
Common data
arr = [{:employee_id=>"001", :first_name=>"John", :last_name=>"Dee",
:date_of_birth=>"10/10/1983", :salary=>"100000", :bonus=>"50000" },
{:employee_id=>"002", :first_name=>"Alex", :last_name=>"Peck",
:date_of_birth=>"11/01/1988", :salary=>"120000", :bonus=>"70000" }]
FName = 'temp.csv'
Write a CSV file with the same keys, in the same order, and the same values
keys = arr.first.keys
#=> [:employee_id, :first_name, :last_name, :date_of_birth, :salary, :bonus]
transform = keys.each_with_object({}) { |k,g| g[k] = ->(h) { h[k] } }
#=> {:employee_id=>#<Proc:0x00005bd270a0e710#(irb):451 (lambda)>,
# :first_name=>#<Proc:0x00005bd270a13260#(irb):451 (lambda)>,
# ...
# :bonus=>#<Proc:0x00005bd270a19cc8#(irb):451 (lambda)>}
construct_csv(FName, arr, transform)
Let's see what was written.
puts File.read(FName)
employee_id,first_name,last_name,date_of_birth,salary,bonus
001,John,Dee,10/10/1983,100000,50000
002,Alex,Peck,11/01/1988,120000,70000
Write a CSV file with the columns reordered1
col_order = [:last_name, :first_name, :employee_id, :salary, :bonus,
:date_of_birth]
keys = arr.first.keys
order_map = col_order.each_with_object({}) { |k,h| h[k] = keys.index(k) }
#=> {:last_name=>2, :first_name=>1, :employee_id=>0, :salary=>4,
# :bonus=>5, :date_of_birth=>3}
transform = col_order.each_with_object({}) { |k,g|
g[k] = ->(h) { h[keys[order_map[k]]] } }
#=> {:last_name=>#<Proc:0x00005bd270f8e5a0#(irb):511 (lambda)>,
# :first_name=>#<Proc:0x00005bd270f8e550#(irb):511 (lambda)>,
# ...
# :date_of_birth=>#<Proc:0x00005bd270f8e3c0#(irb):511 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
last_name,first_name,employee_id,salary,bonus,date_of_birth
Dee,John,001,100000,50000,10/10/1983
Peck,Alex,002,120000,70000,11/01/1988
Write a CSV file with a subset of keys, renamed and reordered
keymap = { :FirstName=>:first_name, :LastName=>:last_name, :ID=>:employee_id,
:Salary=>:salary, :Bonus=>:bonus }
transform = keymap.each_with_object({}) { |(new,old),g| g[new] = ->(h) { h[old] } }
#=> {:FirstName=>#<Proc:0x00005bd270d50298#(irb):391 (lambda)>,
# :LastName=>#<Proc:0x00005bd270d50220#(irb):391 (lambda)>,
# ...
# :Bonus=>#<Proc:0x00005bd270d830f8#(irb):391 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
FirstName,LastName,ID,Salary,Bonus
John,Dee,001,100000,50000
Alex,Peck,002,120000,70000
Write a CSV file after removing keys and adding keys whose values are computed
keys_to_remove = [:first_name, :last_name]
keys_to_add = [:full_name, :compensation]
keys = arr.first.keys + keys_to_add - keys_to_remove
#=> [:employee_id, :date_of_birth, :salary, :bonus, :full_name,
# :compensation]
transform = keys.each_with_object({}) do |k,h|
h[k] =
case k
when :full_name
->(h) { h[:first_name] + " " + h[:last_name] }
when :compensation
->(h) { h[:salary].to_i + h[:bonus].to_i }
else
->(h) { h[k] }
end
end
#=> {:employee_id=>#<Proc:0x00005bd271001000#(irb):501 (lambda)>,
# :date_of_birth=>#<Proc:0x00005bd271000f88#(irb):501 (lambda)>,
# :salary=>#<Proc:0x00005bd271000f10#(irb):501 (lambda)>,
# :bonus=>#<Proc:0x00005bd271000ec0#(irb):501 (lambda)>,
# :full_name=>#<Proc:0x00005bd271000e20#(irb):497 (lambda)>,
# :compensation=>#<Proc:0x00005bd271000dd0#(irb):499 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
employee_id,date_of_birth,salary,bonus,full_name,compensation
001,10/10/1983,100000,50000,John Dee,150000
002,11/01/1988,120000,70000,Alex Peck,190000
1. I don't understand the reason for doing this but it was mentioned as a possible requirement.
I've created the following hash keys with values parsed from PDF into array:
columns = ["Sen", "P-Hire#", "Emp#", "DOH", "Last", "First"]
h = Hash[columns.map.with_index.to_h]
=> {"Sen"=>0, "P-Hire#"=>1, "Emp#"=>2, "DOH"=>3, "Last"=>4, "First"=>5}
Now I want to update the value of each key with 6 equivalent values from another parsed data array:
rows = list.text.scan(/^.+/)
row = rows[0].tr(',', '')
#data = row.split
=> ["2", "6", "239", "05/05/67", "Harp", "Erin"]
I can iterate over #data in the view and it will list each of the 6 values. When I try to do the same in the controller it sets the same value to each key:
data.each do |e|
h.update(h){|key,v1| (e) }
end
=>
{"Sen"=>"Harper", "P-Hire#"=>"Harper", "Emp#"=>"Harper", "DOH"=>"Harper", "Last"=>"Harper", "First"=>"Harper"
So it's setting the value of each key to the last value of the looped array...
I would just do:
h.keys.zip(#data).to_h
If the only purpose of h is as an interim step getting to the result, you can dispense with it and do:
columns.zip(#data).to_h
There are several ways to solve this problem but a more direct and straight forward way would be:
columns = ["Sen", "P-Hire#", "Emp#", "DOH", "Last", "First"]
...
#data = row.split
h = Hash.new
columns.each_with_index do |column, index|
h[column] = #data[index]
end
Another way:
h.each do |key, index|
h[key] = #data[index]
end
Like I said, there are several ways of solving the issue and the best is always going to depend on what you're trying to achieve.
I'm trying to figure out why I keep getting the following error:
From the following code:
def information_transfer()
file_contents = CSV.read("test.csv", col_sep: ",", encoding: "ISO8859-1")
file_contents2 = CSV.read("applicantinfo.csv", col_sep: ",", encoding:"ISO8859-1")
arraysize = file_contents.length
arraysize1 = file_contents2.length
for i in 1..arraysize
for x in 1..arraysize1
if file_contents[i][0] == file_contents2[x][0]
CSV.open("language_output.csv", "wb") do |csv|
csv << [file_contents[i][0], file_contents[i][1], file_contents[i][2],file_contents[i][3], file_contents[i][4],
file_contents[i][5], file_contents[i][6], file_contents[i][7], file_contents[i][8],file_contents[i][9],
file_contents[i][10], file_contents[i][11], file_contents[i][12], file_contents[i][13], file_contents[i][14],
file_contents[i][15], file_contents[i][16], file_contents[i][17], file_contents[i][18], file_contents2[i][24],file_contents2[i][25],
file_contents2[i][26],file_contents2[i][27], file_contents2[i][28], file_contents2[i][29], file_contents2[i][30], file_contents2[i][31], file_contents2[i][32], file_contents2[i][33]]
end
end
end
end
end
I'm basically trying to take two individual .csv files and merge certain columns together. I have two arrays (file_contents and file_contents2) that are reading the individual csv files and storing the contents in arrays. For some reason i'm getting a syntax error for my if statement. I was hoping someone could help me figure out why the if statement that I wrote isn't valid. I figured it would be. Any help is appreciated. Thanks!
Seems like one of file_contents or file_contents2 is empty.
You can skip the loop if you don't want to raise the error on that specific line.
next if file_contents[i].blank? || file_contents2[i].blank?
if file_contents[i][0] == file_contents2[x][0]
One of your arrays file_contents or file_contents2 might be empty. Output both, as well as printing file_contents[i][0] and file_contents2[x][0] before your if statement.
You can make a simple change that should work:
for i in 0..arraysize
for x in 0..arraysize1
And add an error check:
if !file_contents[i].blank? and !file_contents2[x].blank? and file_contents[i][0] == file_contents2[x][0]
for i in 1..arraysize
for x in 1..arraysize1
Array indexes run from 0 to length − 1 in Ruby; loop in 0...arraysize instead.
If file_contents2[i] can or should be written as file_contents2[x], you can just loop over the arrays’ contents directly:
for a in file_contents
for b in file_contents2
and use slices to get consecutive array elements into another array:
def information_transfer()
file_contents = CSV.read("test.csv", col_sep: ",", encoding: "ISO8859-1")
file_contents2 = CSV.read("applicantinfo.csv", col_sep: ",", encoding: "ISO8859-1")
for a in file_contents
for b in file_contents2
if a[0] == b[0]
CSV.open("language_output.csv", "wb") do |csv|
csv << a[0..18] + b[24..33]
end
end
end
end
end
and if you’re trying to join the two files one-to-one, you can do that more efficiently by putting the key into a hash. You also probably didn’t mean to reopen the output file every time.
def information_transfer()
file_contents = CSV.read("test.csv", col_sep: ",", encoding: "ISO8859-1")
file_contents2 = CSV.read("applicantinfo.csv", col_sep: ",", encoding: "ISO8859-1")
h = Hash[file_contents.collect { |row| [row[0], row] }]
CSV.open("language_output.csv", "wb") do |csv|
for b in file_contents2
a = h[b[0]]
csv << a[0..18] + b[24..33]
end
end
end
I have a Hash in the following format:
{"PPS_Id"=>["fe4d7c06-215a-48a7-966d-19ab58976548", "6e90208e-4ab2-4d44-bbaa-9a874bff095b"], "Amount"=>"[\"10000.000\", \"2374.000\"]"}
When I write this data into an excel get the following output.
I want to write the data into the Excel this way:
PPS_Id Amount
fe4d7c06-215a-48a7-966d-19ab58976548 10000.000
6e90208e-4ab2-4d44-bbaa-9a874bff095b 2374.000
How do I convert my current Hash to the below?
{PPS_Id"=>["fe4d7c06-215a-48a7-966d-19ab58976548","Amount"=>"10000.000"},{PPS_Id"=>["6e90208e-4ab2-4d44-bbaa-9a874bff095b","Amount"=>"2374.000"}
Can you please assist.
If you can modify your original hash from
hash = {"PPS_Id"=>["fe4d7c06-215a-48a7-966d-19ab58976548", "6e90208e-4ab2-4d44-bbaa-9a874bff095b"], "Amount"=>"[\"10000.000\", \"2374.000\"]"}
to
hash = {"PPS_Id"=>["fe4d7c06-215a-48a7-966d-19ab58976548", "6e90208e-4ab2-4d44-bbaa-9a874bff095b"], "Amount"=>["10000.000", "2374.000"]}
Note: the last value in the hash is an Array instead of a String.
Then, you can generate an Array of hashes on which you can iterate to fill your excel:
ary = hash.inject([]) do |r, (key, value)|
value.each_with_index do |e, i|
r[i] ||= {}
r[i][key] = e
end
r
end
ary # [{"PPS_Id"=>"fe4d7c06-215a-48a7-966d-19ab58976548", "Amount"=>"10000.000"}, {"PPS_Id"=>"6e90208e-4ab2-4d44-bbaa-9a874bff095b", "Amount"=>"2374.000"}]
If your hash really looks like this :
hash = {"PPS_Id"=>["fe4d7c06-215a-48a7-966d-19ab58976548", "6e90208e-4ab2-4d44-bbaa-9a874bff095b"],
"Amount"=>"[\"10000.000\", \"2374.000\"]"}
you can use scan to parse the floats first :
hash["Amount"] = hash["Amount"].scan(/[\d\-\.]+/)
Your hash will now look like :
{"PPS_Id"=>["fe4d7c06-215a-48a7-966d-19ab58976548", "6e90208e-4ab2-4d44-bbaa-9a874bff095b"],
"Amount"=>["10000.000", "2374.000"]}
To get the table you want, you could just transpose the hash values :
hash.values_at("PPS_Id", "Amount").transpose.each{|id, amount|
puts format("%s\t%.3f", id, amount)
}
It will output :
fe4d7c06-215a-48a7-966d-19ab58976548 10000.000
6e90208e-4ab2-4d44-bbaa-9a874bff095b 2374.000
I'm trying to upload rows from a CSV file into my database, but the spaces in the headers keep messing me up. So for example, the header will be "Order Item Id" and I want the hash key to be "order_item_id". Here's what my code looks like now:
CSV.foreach(file.path, headers:true, :header_converters => lambda { |h| h.try(:downcase) }, col_sep: ';') do |row|
product_hash = row.to_hash
product = OrderCsv.where(id: product_hash["id"])
if product.count ==1
product.first.update_attributes(product_hash)
else
user.order_csvs.create!(product_hash)
end
end
I've tried editing the product_hash with product_hash.keys.each { |k| k = "..." }
but it doesn't do anything. I've also tried creating a header converter like the one that does the downcasing, but I wasn't able to make that work either. Sorry if this is a newb question, but I've been looking everywhere for an answer and none of them have been working for me. Thanks a lot!
You can concatenate the replacement after the downcase, in the :header_converters, like this:
lambda { |h| h.try(:downcase).try(:gsub,' ', '_') }
Try this:
product_hash = { "Order Item Id" => 2 }
product_hash = product_hash.each_with_object({}) do |(k, v), h|
h[k.parameterize.underscore] = v
end
puts product_hash # {"order_item_id"=>2}
In case anyone else stumble upon this question you can make use of :symbol header_converter
https://docs.ruby-lang.org/en/2.1.0/CSV.html#HeaderConverters
The header String is downcased, spaces are replaced with underscores, non-word characters are dropped, and finally to_sym() is called.
Example:
CSV.foreach(csv_path, headers: true, header_converters: :symbol) do |row|
# do stuff
end
If you wish to convert a hash with keys containing spaces to a hash with keys containing underscores, you can do the following
hash_with_spaces = {"order item id" => '1', "some other id" => '2'}
new_hash = hash_with_spaces.inject({}) do |h, (k, v)|
h[k.gsub(' ', '_')] = v ; h
end
new_hash
#=> {"order_item_id"=>"1", "some_other_id"=>"2"}