I am writing my Hashmap into a csv file with headers using the below piece of code. However, what I notice is that the first data row is not available in the file. I can the headers and all other rows accurately
def self.execute_auto_pg_debtors pg_debtors_dt_list
partition_key = Date.today.prev_day.prev_day.strftime "%Y%m%d"
csvfilename = "PG_Debtors_" + partition_key + ".CSV"
pg_debtors_dt_batch = Array.new
rowid = 0
pg_debtors_dt_list.each { |x|
pg_debtors_details = Hash.new
pg_debtors_details["Store_Order_Id"] = x['Store_Order_Id']
pg_debtors_details["Transaction_Id"] = x['Transaction_Id']
pg_debtors_details["Gateway_Payment_Id"] = x['Gateway_Payment_Id']
pg_debtors_details["PPS_Id"] = x['PPS_Id']
pg_debtors_details["Event_Type"] = x['Event_Type']
pg_debtors_details["Event_Date"] = x['Event_Date']
pg_debtors_details["Gateway_Name"] = x['Gateway_Name']
pg_debtors_details["Open_Amount"] = "%f" % x['Open_Amount']
pg_debtors_details["Invoice_No"] = x['Invoice_No']
pg_debtors_dt_batch << pg_debtors_details
rowid += 1
if rowid == 1
CSV.open(csvfilename, "w") do |csv|
csv << pg_debtors_details.keys# adding header row (column labels)
end
else
CSV.open(csvfilename, "a") do |csv|
csv << pg_debtors_details.values# of if/else inside hsh
end# of hsh's (rows)
end# of csv open
}
return pg_debtors_dt_batch
end
Please help.
You are writing the headers instead of the first row!
I recommend, that you open the file and iterate through your hash inside the CSV.open do ... end
AND do not use a else after your if rowid == 1. Just execute that for EVERY values, so you do not skip data row 1
Even if you check for rowid, the .each loop is still not aware of it. So, for rowid == 1, it will write the headers, but in the next iteration, x will point to the second item in pg_debtors_dt_list.
To solve it, write your code in the following order:
Open the file, and write the headers first.
Loop through pg_debtors_dt_list, and write subsequent data to the file.
Hope it helps.
Related
Using RubyXL I want to know what row number my iteration is.
workbook = RubyXL::Parser.parse("./file.xlsx")
worksheet = workbook[0]
worksheet.each do |row|
test0 = row[0].value
line = ????
puts "Line number #{line} - Value = #{test0}"
end
You can use #each_with_index and write it like this:
workbook = RubyXL::Parser.parse("./file.xlsx")
workbook.first.each_with_index do |row, index|
puts "Line number #{index} - Value = #{row[0].value}"
end
You can use each_with_index method while looping to get the current row number of the iteration
worksheet.each_with_index do |row, index|
test0 = row[0].value
line = index
puts "Line number #{line} - Value = #{test0}"
end
So, I have the following:
twodarray = []
File.open("SG_hum50_LODG.gro", "r") do |f|
f.each_line do |line|
textarray = line.split()
textarray[0],textarray[1],textarray[2] = textarray[1],textarray[0],textarray[2].to_i
textarray[1] = textarray[1].gsub(/:/, '').to_i
twodarray << textarray
end
end
Which works pretty well. The problem I am having is I need to ignore the last line of the text file, and I need to add
["Sensor", "Timestamp", "Sensor Value"],
As the first row in the array.
I would do this
twodarray = [["Sensor", "Timestamp", "Sensor Value"]]
File.open("SG_hum50_LODG.gro", "r") do |f|
f.each_line do |line|
textarray = line.split()
textarray[0], textarray[1], textarray[2] =textarray[1], textarray[0], textarray[2].to_i
textarray[1].gsub!(/:/, '').to_i
twodarray << textarray
end
end
twodarray.pop
twodarray << textarray unless line[f.last]
Code
def read_sensor_data(fname, first_arr)
File.open(fname, "r") do |f|
f.each_line.with_object([first_arr]) do |line, arr|
next if f.eof?
sensor, time, value, *rest = line.split
arr << [time.delete(':').to_i, sensor, value.to_i, *rest]
end
end
end
*rest is not needed if every line of the file contains no more than three words.
Example
FName = "demo"
Create a file.
File.write FName,
<<-_
heat 10:21 16
sound 11:45 74
That's all, folks!
_
#=> 49
Check the file.
puts File.read FName
heat 10:21 16
sound 11:45 74
That's all, folks!
Try the method.
read_sensor_data FName, ["Sensor", "Timestamp", "Sensor Value"]
#=> [["Sensor", "Timestamp", "Sensor Value"],
# [1021, "heat", 16],
# [1145, "sound", 74]]
Be careful aggregating lines in a file into an array because that's not a scalable solution. Only do it if you can guarantee that the file being loaded, plus the needs of your script, will never grow beyond the available free memory.
Your question isn't clear about what you're trying to do, but it sounds like you're trying to modify a file on disk, which is easily done using File.foreach to read the incoming file line-by-line.
There's a couple ways to ignore the last line, but the easiest is to count the lines in the file then read all but the last line. There are many ways to count the lines in a file, the easiest is to use the *nix wc -l command, which is designed to do it:
lines_in_file = `wc -l /path/to/file`.to_i
and for a pure Ruby solution I'd do something like this:
lines_in_file = File.foreach('input.txt').inject(0){ |i, _| i + 1 }
Then it's a simple matter of counting lines_in_file - 1 lines. This is untested but it looks about right:
INPUT_FILE = 'input.txt'
lines_in_file = File.foreach(INPUT_FILE).inject(0){ |i, _| i + 1 }
File.open('output.txt', 'w') do |fo|
fo.puts '"Sensor", "Timestamp", "Sensor Value"'
File.foreach(INPUT_FILE) do |li|
fo.puts li
break if $. == (lines_in_file - 1)
end
end
We just found out that CSV export cut off text in one field. Here is the original text for CSV export (from a text field of postgres 9.3):
#payable_approved_unpaid, #payable_paid, #payable_po_unpaid = {}, {}, {}
models.each do |m|
#payable_po_unpaid[m.id.to_s] = PurchaseOrderx::Order.where(project_id: m.id).sum('po_total') - PaymentRequestx::PaymentRequest.where(project_id: m.id).where(resource_string: 'purchase_orderx/orders').sum('amount')
#payable_paid[m.id.to_s] = PaymentRequestx::PaymentRequest.where(project_id: m.id).where(wf_state: :paid).sum('amount')
#payable_approved_unpaid[m.id.to_s] = PaymentRequestx::PaymentRequest.where(project_id: m.id).where('approved = ? AND wf_state != ?', true, :paid).sum('amount')
end
Here is what we get from CSV:
#payable_approved_unpaid, #payable_paid, #payable_po_unpaid = {}, {}, {}
models.each do |m|
#payable_po_unpaid[m.id.to_s] = PurchaseOrderx::Order.where(project_id: m.id).sum('po_total') - PaymentRequestx::PaymentRequest.where(project_id: m.id).wher
In the same file, there are fields which is much long than this and there is no problem. We have been using the export for a long time and this is first time we are losing text. What could cause the cut=off of text in CSV export?
Here is the method for CSV export, argument_value is the field with cut-off text. In debug, we verified that full context of the column has been assigned to CSV before exporting:
def self.to_csv
CSV.generate do |csv|
header = ['id', 'engine_name', 'engine_version', 'argument_name', 'argument_value', 'last_updated_by_id', 'created_at', 'updated_at', 'brief_note', 'global']
csv << header
i = 1
all.each do |config|
base = OnboardDatax.engine_config_class.find_by_id(config.engine_config_id)
row = Array.new
row << i
row << (base.global ? nil : base.engine.name)
row << base.engine_version
row << base.argument_name
row << (config.custom_argument_value.present? ? config.custom_argument_value : base.argument_value)
row << config.last_updated_by_id
row << config.created_at
row << config.updated_at
row << base.brief_note
row << (base.global ? 't' : 'f')
csv << row
i += 1
end
end
If wrapping the whole text with quotation mark, then the column can be exported to CSV in its entirety.
What could cause the cut=off of text in CSV export?
puts File.read('your_cutoff_text.txt').size
--output:--
256
Rails column types:
String:
Limited to 255 characters (depending on DBMS)
My OS automatically adds a newline to the end of a file, so your cutoff text contains exactly 255 characters.
In the same file, there are fields which is much long than this and
there is no problem.
Rails column types:
Text:
Unlimited length (depending on DBMS)
I have a ruby controller
def new
counter = 1
fileW = File.new("query_output.txt", "w")
file = File.new("query_data.txt", "r")
while (line = file.gets)
puts "#{counter}: #{line}"
query = "select name,highway from planet_osm_line where name ilike '" +line+"'"
#output = PlanetOsmLine.connection.execute(query)
#output.each do |output|
fileW.write(output['highway'] + "\n")
end
counter = counter + 1
end
file.close
query = ""
#output = PlanetOsmLine.connection.execute(query)
end
So in this I am reading from a file like
%12th%main%
%100 feet%
%12th%main%
%12th%main%
%12th%main%
%100 feet%
In the ruby console I can see all the queries getting executed but in query_output.txt I only get the output of last query. What am I doing wrong here?
You use filemode w which will re-create the output file every time (so you will write into an empty file). Instead open your file as follows:
fileW = File.new("query_output.txt", "a")
a stands for append. It will open or create the file, and append at the back.
For more info concerning file-modes: http://pubs.opengroup.org/onlinepubs/009695399/functions/fopen.html
Can we use column headers to specify the column number from which we are parsing the excel sheet using roo gem? My current code is like this now:
oo = Openoffice.new("simple_spreadsheet.ods")
oo.default_sheet = oo.sheets.first
(2..oo.last_row).each do |line|
date = oo.cell(line,'A')
start_time = oo.cell(line,'B')
end_time = oo.cell(line,'C')
pause = oo.cell(line,'D')
...
end
I would like to parse from column headers instead of specifying columns as 'A' 'B' 'C' ... Can I acheive this using Roo?
You can grab the entire header row as an array and hash the entire row key'd on the header row.
oo = Openoffice.new("simple_spreadsheet.ods")
oo.default_sheet = oo.sheets.first
header = oo.row(1)
2.upto(oo.last_row) do |line|
row_data = Hash[header.zip oo.row(line)]
...
end
You could also use row_data[line] to nest the hashes for later use.
A cleaner/clearer version of the above is
oo = Openoffice.new("simple_spreadsheet.ods")
oo.default_sheet = file.sheets.first
header = oo.first_row
2.upto(oo.last_row) do |line|
row_data = Hash[*header.zip(row).flatten]
...
end
the original took me a bit to understand because especially as i thought hash was a local variable named hash instead of the class Hash
This will use the header row as the keys. The helpful parts are transpose and strip.
def self.excel_to_hash(folder_name, file_name, tab_name)
# Takes an excel file name and a tab name, and returns an array of stripped, transposed rows
# Sample call: my_data = excel_to_hash File.join(Rails.root,'db/data/data_to_import.xlsx'), 'models'
rows = []
file = File.open(File.join(folder_name, file_name), mode = 'r')
excel = Excelx.new(file.path, nil, :ignore)
excel.default_sheet = excel.sheets.index(tab_name) + 1
header = excel.row(1)
(2..excel.last_row).each do |i|
next unless excel.row(i)[0]
row = Hash[[header, excel.row(i)].transpose]
row.each_key{|x| row[x] = row[x].to_s.strip if row[x]}
rows << row
end
return rows
end
valid through Roo gem 1.10.2
This works for me
require 'roo'
# open excel file
excel_file = Roo::Spreadsheet.open(file_path)
# iterate on each sheet
excel_file.each_with_pagename do |name, sheet|
# iterate on each sheet
sheet.parse(headers: true, pad_cells: true) do |row|
# data should be access by column header if we have column header Name we can access like this
row['Name']
end
end
end