I am working on requirements we have data in hash around 100+ keys. we need to generate CSV file as per user-defined header with some transformation, we may end up having 100+ template
Main changes will be
1) Change column name such as Fname –> First name
2) Data transformation like Full name – > First name + Last name (adding 2 column)
3) Fixing the position of a column – Fname should be at 35 positions etc.
please suggest is it possible to define declarative way or any gem available. Can you let me know any design pattern we can apply here?
Some sample scenarios
I have input like this with many columns (100+)
[ {:employee_id=>"001", :first_name=>"John",:last_name=>"Dee" :date_of_birth=>"10/10/1983", :salary=>"100000",:bounus =>"50000",......},
{:employee_id=>"002", :first_name=>"Alex",:last_name=>"Peck" :date_of_birth=>"11/01/1988", :salary=>"120000",:bounus =>"70000", .........},
]
Some customer need CSV as
Employee ID, First Name, Last Name, Date of birth, Salary, Bonus
001,John,Dee,10/10/1983,100000,50000,...
002,Alex,Peck,11/01/1988,120000,70000,...
Others (only header change)
ID, FName, LName, Dob, Salary, Bounus
001,John,Dee,10/10/1983,100000,50000,...
002,Alex,Peck,11/01/1988,120000,70000,...
another (merge of colum FName, LName -> Fullname)
ID, Fullname, Dob, Salary, Bounus
001,John Dee,10/10/1983,100000,50000,...
002,Alex Peck,11/01/1988,120000,70000,...
anothers (merge of column Salary, Bonus -> Salary+ Bonus)
ID, FName, LName, Dob, Salary
001,John,Dee,10/10/1983,150000,...
002,Alex,Peck,11/01/1988,190000,...
anothers ( column order changed also insted of Dob need age)
FName, LName, ID, age, Salary
John,Dee,001,36,150000,...
Alex,Peck,003,32,190000,...
Like many variations with the same input
Thanks for help
What you need is the presenter design pattern.
Your controller will request the data and store it in a local variable, and then your will have to load a presenter for your client passing it the data variable.
In response you'll get the final CSV to return to the client.
Let's say you clients have uniq codes, so that a Client model instance has a code attribute which is a string.
So your controller would looks like this:
app/controllers/exports_controller.rb
class ExportsController < ApplicationController
def export
data = MyService.fetchData # <== data contains the data you gave as an example
# Gets the right presenter, initialise it, and build the CSV
csv = PresenterFactory.for(current_client).new(data).present
respond_to do |format|
format.html
format.csv { send_data csv, filename: "export-name-for-#{current_client.code}.csv" }
end
end
end
The PresenterFactory class would be something like that:
app/models/presenter_factory.rb
class PresenterFactory
def self.for(client)
# For client with code "ABCD" it will return Presenters::Abcd class
"Presenters::#{client.code.capitalize}".constantize
end
end
The factory return the client's presenter class
And here is an example for a client's presenter class, for a client having the code ABCD:
app/models/presenters/abcd.rb
module Presenters
class Abcd
def initialize(data)
#data = data
end
def present
CSV.generate(headers: true) do |csv|
# Here is the client's specific CSV header
csv << [
'Employee ID',
'First Name',
# ...
]
#data.each do |row|
# Here is the client's specific CSV row
csv << [
row[:employee_id],
row[:first_name],
# ...
]
end
end
end
end
end
You can achieve your objective by constructing a transformation hash whose keys are the names of the columns in the desired CSV file, in order, and whose values are procs, which when called with an argument equal to an element of the given array of hashes, returns an element to be written in a row of the CSV file in the column corresponding to the key.
Code
require 'csv'
def construct_csv(fname, arr, transform)
CSV.open(fname, "wb") do |csv|
keys = transform.keys
csv << keys
arr.each { |h| csv << keys.map { |k| transform[k].call(h) } }
end
end
Examples
I will now illustrate how this method is used with various transformations.
Common data
arr = [{:employee_id=>"001", :first_name=>"John", :last_name=>"Dee",
:date_of_birth=>"10/10/1983", :salary=>"100000", :bonus=>"50000" },
{:employee_id=>"002", :first_name=>"Alex", :last_name=>"Peck",
:date_of_birth=>"11/01/1988", :salary=>"120000", :bonus=>"70000" }]
FName = 'temp.csv'
Write a CSV file with the same keys, in the same order, and the same values
keys = arr.first.keys
#=> [:employee_id, :first_name, :last_name, :date_of_birth, :salary, :bonus]
transform = keys.each_with_object({}) { |k,g| g[k] = ->(h) { h[k] } }
#=> {:employee_id=>#<Proc:0x00005bd270a0e710#(irb):451 (lambda)>,
# :first_name=>#<Proc:0x00005bd270a13260#(irb):451 (lambda)>,
# ...
# :bonus=>#<Proc:0x00005bd270a19cc8#(irb):451 (lambda)>}
construct_csv(FName, arr, transform)
Let's see what was written.
puts File.read(FName)
employee_id,first_name,last_name,date_of_birth,salary,bonus
001,John,Dee,10/10/1983,100000,50000
002,Alex,Peck,11/01/1988,120000,70000
Write a CSV file with the columns reordered1
col_order = [:last_name, :first_name, :employee_id, :salary, :bonus,
:date_of_birth]
keys = arr.first.keys
order_map = col_order.each_with_object({}) { |k,h| h[k] = keys.index(k) }
#=> {:last_name=>2, :first_name=>1, :employee_id=>0, :salary=>4,
# :bonus=>5, :date_of_birth=>3}
transform = col_order.each_with_object({}) { |k,g|
g[k] = ->(h) { h[keys[order_map[k]]] } }
#=> {:last_name=>#<Proc:0x00005bd270f8e5a0#(irb):511 (lambda)>,
# :first_name=>#<Proc:0x00005bd270f8e550#(irb):511 (lambda)>,
# ...
# :date_of_birth=>#<Proc:0x00005bd270f8e3c0#(irb):511 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
last_name,first_name,employee_id,salary,bonus,date_of_birth
Dee,John,001,100000,50000,10/10/1983
Peck,Alex,002,120000,70000,11/01/1988
Write a CSV file with a subset of keys, renamed and reordered
keymap = { :FirstName=>:first_name, :LastName=>:last_name, :ID=>:employee_id,
:Salary=>:salary, :Bonus=>:bonus }
transform = keymap.each_with_object({}) { |(new,old),g| g[new] = ->(h) { h[old] } }
#=> {:FirstName=>#<Proc:0x00005bd270d50298#(irb):391 (lambda)>,
# :LastName=>#<Proc:0x00005bd270d50220#(irb):391 (lambda)>,
# ...
# :Bonus=>#<Proc:0x00005bd270d830f8#(irb):391 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
FirstName,LastName,ID,Salary,Bonus
John,Dee,001,100000,50000
Alex,Peck,002,120000,70000
Write a CSV file after removing keys and adding keys whose values are computed
keys_to_remove = [:first_name, :last_name]
keys_to_add = [:full_name, :compensation]
keys = arr.first.keys + keys_to_add - keys_to_remove
#=> [:employee_id, :date_of_birth, :salary, :bonus, :full_name,
# :compensation]
transform = keys.each_with_object({}) do |k,h|
h[k] =
case k
when :full_name
->(h) { h[:first_name] + " " + h[:last_name] }
when :compensation
->(h) { h[:salary].to_i + h[:bonus].to_i }
else
->(h) { h[k] }
end
end
#=> {:employee_id=>#<Proc:0x00005bd271001000#(irb):501 (lambda)>,
# :date_of_birth=>#<Proc:0x00005bd271000f88#(irb):501 (lambda)>,
# :salary=>#<Proc:0x00005bd271000f10#(irb):501 (lambda)>,
# :bonus=>#<Proc:0x00005bd271000ec0#(irb):501 (lambda)>,
# :full_name=>#<Proc:0x00005bd271000e20#(irb):497 (lambda)>,
# :compensation=>#<Proc:0x00005bd271000dd0#(irb):499 (lambda)>}
construct_csv(FName, arr, transform)
puts File.read(FName)
employee_id,date_of_birth,salary,bonus,full_name,compensation
001,10/10/1983,100000,50000,John Dee,150000
002,11/01/1988,120000,70000,Alex Peck,190000
1. I don't understand the reason for doing this but it was mentioned as a possible requirement.
I'm attempting to copy all the contents of a CSV file over to an excel workbook using the AXLSX gem. On a second sheet, I only want 2 of the columns copied over. Below is an example.
I tried the '.map' method but that didn't work.
require 'csv'
require 'Axlsx'
p = Axlsx::Package.new
wb = p.workbook
animals = CSV.read('test.csv', headers:true)
column = ['Animals', 'Name']
headers = Array.new
headers << "Animal"
headers << "Name"
headers << "Age"
headers << "State"
wb.add_worksheet(:name => 'Copy') do |sheet|
animals.each do |row|
headers.map { |col| sheet.add_row [row[col]] }
end
end
wb.add_worksheet(:name => 'Names') do |sheet|
animals.each do |row|
column.map { |col| sheet.add_row [row[col]] }
end
end
p.serialize 'Animals.xlsx'
CSV - But also desired output on XLSX
Output from my code
At first read file using IO::readlines, then split every line (separate by the cells using Array#map and String#split).
It will produce nested array. Something like
[["Animal", "Name", "Age", "State"], ["Dog", "Rufus", "7", "CA"], ["Bird", "Magnus", "3", "FL"]]
Each subarray here is the row of your table, each item of subarray is the cell of your table.
If you need just "Name" column on second sheet you need make nested array
[["Name"], ["Rufus"], ["Magnus"]]
That's all you need to generate XLSX file.
For example you have animals.csv with data serapated by , and want to generate animals.xlsx:
require 'axlsx'
copy = File.readlines('animals.csv', chomp: true).map { |s| s.split(',') }
names = copy.map { |l| [l[1]] }
Axlsx::Package.new do |p|
p.workbook.add_worksheet(name: 'Animals') { |sheet| copy.each { |line| sheet.add_row(line) }}
p.workbook.add_worksheet(name: 'Names') { |sheet| names.each { |line| sheet.add_row(line) }}
p.serialize('animals.xlsx')
end
I have a CSV file with two columns:
PPS_Id Amount
123 100
1234 150
I read data from this file and insert in a array using the code below:
CSV.foreach("filename.CSV", headers: true) do |row|
file_details << row.inspect # hash
end
I am then trying to push the data in the file_details into a hash with PPS_Id as key and Amount as Value, I am using the code below:
file_details_hash = Hash.new
file_details.each { |x|
file_details_hash[x['PPS_Id']] = x['Amount']
}
But when I print the result I get nothing just {"PPS_Id"=>"Amount"}
Can you please help
Your code, modified to work
You need to specify the column separator for your csv, and remove inspect.
require 'csv'
file_details = []
CSV.foreach("filename.CSV", headers: true, col_sep: "\s" ) do |row|
file_details << row
end
file_details_hash = Hash.new
file_details.each { |x|
file_details_hash[x['PPS_Id']] = x['Amount']
}
p file_details_hash
#=> {"123"=>"100", "1234"=>"150"}
It now returns what you expected to get.
Shorter solution
Read the csv, drop the first line (header) and convert to a Hash :
p CSV.read("filename.CSV", col_sep: "\s").drop(1).to_h
#=> {"123"=>"100", "1234"=>"150"}
First of all, you are collecting strings into an array (see String#inspect):
file_details << row.inspect
After that you call (sic!) String#[] on that strings:
x['PPS_Id'] #⇒ "PPS_Id", because string contains this substring
That said, your code has nothing but errors. You might achieve what you want with:
csv = CSV.parse(File.read("filename.CSV"), col_sep: "\s")
csv[1..-1].to_h
#⇒ {
# "123" => "100",
# "1234" => "150"
# }
Using inspect will save your CSV rows as strings, so obviously you won't be able get what you need. Instead try this:
file_details = CSV.read("filename.csv")
Read CSV directly will create an 2D array that you can then iterate over, which will look like this: [["PPS_Id", "Amount"], ["123", "100"], ["1234", "150"]]
From there you can slightly modify your approach:
file_details.each do |key, value|
file_details_hash[key] = value
end
To receive a hash like this: {"PPS_Id"=>"Amount", "123"=>"100", "1234"=>"150"}
I like to export a dataset from my rails application as a csv file using the builtin csv library of rails. Usually a csv file is written row wise like in my example below which comes from my datasets_controller.rb:
require 'csv'
dataset = Dataset.find(6)
dataset_headers = dataset.datacolumns.collect { |dc| dc.columnheader }
csv_file = CSV.generate do |csv|
csv << dataset_headers
end
And now my question is if I could also write my csv files column wise like this?
require 'csv'
dataset_columns = Datacolumn.all(:conditions => ["dataset_id = ?", 6], :order => "columnnr ASC").uniq
csv_file = CSV.generate do |csv|
csv << "here put one after another all my data columns"
end
EDIT:
Based on Douglas suggestion I came up with the colde below.
data_columns=Datacolumn.all(:conditions => ["dataset_id = ?", dataset.id], :order => "columnnr ASC").uniq
CSV.generate do |csv|
value=Array.new
data_columns.each do |dc|
value << dc.columnheader
dc.sheetcells.each do |sc|
if sc.datatype && sc.datatype.is_category? && sc.category
value << sc.category.short
elsif sc.datatype && sc.datatype.name.match(/^date/) && sc.accepted_value
value << sc.accepted_value.to_date.to_s
elsif sc.accepted_value
value << sc.accepted_value
else
value << sc.import_value
end
end
csv << value
value = Array.new
end
end
The output is not transposed for this case and looks like this:
height,10,2,<1,na
fullauthor,Fortune,(Siebold & Zucc.) Kuntze,Fortune,(Siebold & Zucc.) Kuntze
Year,1850,1891,1850,1891
fullname,Mahonia bealei,Toxicodendron sylvestre,Mahonia bealei,Toxicodendron sylvestre
But when I change the line which writes the csv to
csv << value.transpose
I get an error which tells me that it could not convert a string to array to do that.
Anybody an Idea how to fix this?
Any help with this would be appreciated.
Best Claas
You could use Array#transpose, which will flip your rows to columns. A simple example:
> a = [['name', 'charles', 'dave'],['age', 24, 36],['height', 165, 193]]
=> [["name", "charles", "dave"], ["age", 24, 36], ["height", 165, 193]]
> a.transpose
=> [["name", "age", "height"], ["charles", 24, 165], ["dave", 36, 193]]
Thus, assuming dataset_columns is an array:
require 'csv'
dataset_columns = Datacolumn.all(:conditions => ["dataset_id = ?", 6], :order => "columnnr ASC").uniq
csv_file = CSV.generate do |csv|
csv << dataset_columns.transpose
end