Rails Testing Contents of CSV Download - ruby-on-rails

I have a project set up to download information from a page as a csv. I'm trying to write a test that ensures the contents of this csv matches what is shown on the site, but I am not sure how to access the contents of the CSV. Here's my current attempt:
test "check_acquired_shares_contents" do
data = ["Label Amount Of Shares Share Price Total Price Occurred On From Shareholder Share Transaction Action Share Transaction Type",
"Initial 100.0 100.0 10000.0 2001-05-06 CREATE ESPP", "On Date 20.0 10.0 200.0 2012-05-06 Bill TRANSFER INDIVIDUAL",
"Label Amount Of Shares Share Price Total Price Occurred On To Shareholder Share Transaction Action Share Transaction Type",
"Individual_transfers 10.0 10.0 100.0 2010-05-06 Bill TRANSFER INDIVIDUAL"]
visit admin_shareholder_path(id: 1)
find('.action_item', :text => 'Acquired Shares CSV').click
#binding.pry
assert data == page.all('tr').map { |tr| tr.text }
end
The data array is what the CSV is meant to contain. However, I think I am not checking the contents of the CSV, because the test always fails.
Anyone have ideas on how to check the CSV contents? Thank you!

Reading CSV file itself would be good to test the exact CSV scenario here
You can do something like below:
require 'csv'
csv_data = CSV.read('csv_file_name.csv', { encoding: 'UTF-8', headers: true, converters: :all, header_converters: :symbol})
data = csv_data.map { |d| d.to_hash }
What u would get is:
[ {:Label=>"Initial", :Share=>"100.0", :Price=>"10000.0",..}, {:Label=>...}.. ]
Now you can easily write your test cases by comparing this data and the one you are expecting.

Related

Rails - Accessing JSON members

I am new to Rails, and working with some JSON, and not sure how to get to the data as the examples below:
1) If i were to use JSON.parse(response)['Response']['test']['data']['123456'], i will need to parse another response for 123457, is there a better way to loop through all the objects in data?
2) base on the membershipId, identify the top level object, ie data.
"test": {
"data": {
"123456": {
"membershipId": "321321312",
"membershipType": a,
},
"123457": {
"membershipId": "321321312",
"membershipType": a,
},
}
JSON.parse(response)['Response']['test']['data'].each do |key, object|
puts key
puts object['membershipID']
...
end
To select the data record associated with a particular membership
match_membership = '321321312'
member = JSON.parse(response)['Response']['test']['data'].select |_key, object|
object['membershipID'] == match_membership
end
puts member.key
=> 123456
For 1:
Assumption:
By you saying "need to parse another response", you were doing something like below:
# bad code: because you are parsing `response` multiple times
JSON.parse(response)['Response']['test']['data']['123456']
JSON.parse(response)['Response']['test']['data']['123457']
then simply:
Solution 1:
If you are gonna be accessing 2+ level deep hash values for just maybe 2 or 3 times,
response_hash = JSON.parse(response)
response_hash['Response']['test']['data']['123456']
response_hash['Response']['test']['data']['123457']
Solution 2:
If you are gonna be accessing 2+ level deep hash values for loooooots of times,
response_hash = JSON.parse(response)
response_hash_response_test_data = response_hash['Response']['test']['data']
response_hash_response_test_data['123456']
response_hash_response_test_data['123457']
response_hash_response_test_data['123458']
response_hash_response_test_data['123459']
response_hash_response_test_data['123460']
# ...
Solution 2 is better than Solution 1 because it saves repetitive method calls for Hash#[] which is the "getter" method each time you do like ...['test'] then ['data'] then ['123456'], and so is better-off doing Solution 2 which you store the nested-level of the hash into a variable (this does not duplicate the values in-memory!). Plus it's more readable this way.

Rails Roo gem .xlsx output contains the object not the output of method

I am using the Roo gem to output a spreadsheet from a Rails app. One of my columns is a hash (Postgres DB). I would like to format the cell contents into something more readable. I am using a method to return a human readable cell.
The column data looks like this:
Inspection.first.results
=> {"soiled"=>"oil on back",
"assigned_to"=>"Warehouse#firedatasolutions.com",
"contaminated"=>"blood on left cuff",
"inspection_date"=>"01/01/2017",
"physical_damage_seam_integrity"=>"",
"physical_damage_thermal_damage"=>"",
"physical_damage_reflective_trim"=>"",
"physical_damage_rips_tears_cuts"=>"small tear on right sleeve",
"correct_assembly_size_compatibility_of_shell_liner_and_drd"=>"",
"physical_damage_damaged_or_missing_hardware_or_closure_systems"=>""}
In my Inspections model I defined the following method:
def print_results
self.results.each do |k,v|
puts "#{k.titleize}:#{v.humanize}\r\n"
end
end
So in the console I get this:
Inspection.first.print_results
Soiled:Oil on back
Assigned To:Warehouse
Contaminated:Blood on left cuff
Inspection Date:01/01/2017
Physical Damage Seam Integrity:
Physical Damage Thermal Damage:
Physical Damage Reflective Trim:
Physical Damage Rips Tears Cuts:Small tear on right sleeve
Correct Assembly Size Compatibility Of Shell Liner And Drd:
Physical Damage Damaged Or Missing Hardware Or Closure Systems:
=> {"soiled"=>"oil on back",
"assigned_to"=>"Warehouse",
"contaminated"=>"blood on left cuff",
"inspection_date"=>"01/01/2017",
"physical_damage_seam_integrity"=>"",
"physical_damage_thermal_damage"=>"",
"physical_damage_reflective_trim"=>"",
"physical_damage_rips_tears_cuts"=>"small tear on right sleeve",
"correct_assembly_size_compatibility_of_shell_liner_and_drd"=>"",
"physical_damage_damaged_or_missing_hardware_or_closure_systems"=>""}
But when I put this in the index.xlsx.axlsx file
wb = xlsx_package.workbook
wb.add_worksheet(name: "Inspections") do |sheet|
sheet.add_row ['Serial Number', 'Category', 'Inspection Type', 'Date',
'Pass/Fail', 'Assigned To', 'Inspected By', 'Inspection Details']
#inspections.each do |inspection|
sheet.add_row [inspection.ppe.serial, inspection.ppe.category,
inspection.advanced? ? 'Advanced' : 'Routine',
inspection.results['inspection_date'],
inspection.passed? ? 'Pass' : 'Fail',
inspection.ppe.user.last_first_name,
inspection.user.last_first_name,
inspection.print_results]
end
end
The output in the spreadsheet is the original hash, not the results of the print statement.
{"soiled"=>"oil on back",
"assigned_to"=>"Warehouse",
"contaminated"=>"blood on left cuff", "inspection_date"=>"01/01/2017",
"physical_damage_seam_integrity"=>"",
"physical_damage_thermal_damage"=>"",
"physical_damage_reflective_trim"=>"",
"physical_damage_rips_tears_cuts"=>"small tear on right sleeve",
"correct_assembly_size_compatibility_of_shell_liner_and_drd"=>"",
"physical_damage_damaged_or_missing_hardware_or_closure_systems"=>""}
Is it possible to get the output of the method into the cell rather than the hash object?
The problem is that your print_results method prints out what you want to stdout (that is, the console), but still returns the original hash. The return value of the method is all that matters to Roo.
What you want to do is rewrite print_results to return the formatted string:
def print_results
self.results.map do |k,v|
"#{k.titleize}:#{v.humanize}\r\n"
end.join
end
This will return a string (note the use of .join to combine the array of strings returned by .map) that you can throw into Roo and get your desired output.

Use gem 'postgres-copy' to import csv file

currently, I want to import above 55,000 records into my database from a CSV file. This is the code that I am using:
CSV.foreach(Rails.root.join('db/seeds/locations.csv'), headers: true) do |row|
val = Location.find_or_initialize_by(code: row[0])
val.name = row[1]
val.ecc = row[2] || 'MISSING'
val.created_by = User.find_by(name: 'anh')
val.updated_by = User.find_by(name: 'anh')
val.save!
end
However, it is too slow and I have just installed the gem 'postgres-copy'. I read the official documentation, and I believe I can use the class method copy_from to do the job, but if you read my current code, you can see that I am referring the data to the another table(association), and the documentation doesn't mention anything about association or validation. Therefore, I am wondering if there are any ways to solve it. This is the first time I use this gem. Thanks for reading.
I don't know that gem, but I would be very surprised if it can support multi-table copy since PostgreSQL's COPY works on a single table. 50K rows isn't all that many. You might try wrapping your insertions in transactions to avoid one commit per transaction. Doubt you want to wrap all 50K in a transaction though, but something like this:
User.connection.begin_transaction
i = 0
CSV.foreach(...) do |row|
... # your original code here
i += 1
if i % 500 == 0
User.connection.commit_transaction
User.connection.begin_transaction
end
end
User.connection.commit_transaction
This will insert your rows 500 records at a time and you should see a noticeable speed up. Play around with the value of 500 to find the sweet spot.
So, now I understand that I cannot take advantage of the COPY command in POSTGRESQL since it can't copy multiple tables. Therefore, I switch to the gem activerecord-import. Comparing with the method that Philip Hallstrom mentioned above, using activerecord-import give a faster result, 1m20s vs 1m54s to import above 8000 records.
This is my code after installing the gem activerecord-import. Hopefully, it can help other people.
locations = []
columns = [:code, :name, :ecc]
CSV.foreach(Rails.root.join('db/seeds/locations.csv'), headers: true) do |row|
val = Location.find_or_initialize_by(code: row[0])
val.name = row[1]
val.ecc = row[2] || 'MISSING'
val.created_by = User.find_by(name: 'anh')
val.updated_by = User.find_by(name: 'anh')
locations << val
end
Location.import columns, locations, validate: false

Parse web scraped data

So i have this coming from my web scrape
pastebin.com/CMrFcBMX
What i'm wanting is all the prices and ticket description. Heres what i have
doc.xpath("//script[#type='text/javascript']/text()").each do |text|
if text.content =~ /more_options_on_polling/
price1 = text.to_s.scan(/\"(formatted_(?:total_price))\":\"(.+?)\"/).uniq
description = text.to_s.scan(/\"(ticket_desc)\":\"(.+?)\"/).uniq
price = price1 + description
render json: price
end
end
So this is what i have at the moment. However i'm needing to do some major edits.
Firstly i'm needing the description to ignore any plus symbols, e.g. Later Owl + Chance For VIP Upgrade\ would need to be ignored.
Secondly I need to remove the json rendering nice, So that the first price and fees match with the first description.
Once i have this rendered i should be sorted. I'd be using this in a js file afterwards so a format like this would be best:
Ticket{
[
Price:
Fees:
Description:
]
}
Once i have it like this i should be good to finish my application ^_^
Thanks
Sam

When reading CSV files the output is strange

I have a CSV file which looks like this:
1ttAAAttAnaattFrench PolynesiattPFttAustralia and Oceaniatt-17.352606tt-145.509956
2ttAAEttAnnabattAlgeriattDZttAfricatt36.822225tt7.809167
3ttAAFttApalachicolattUnited StatesttUSttNorth Americatt29.7276066tt-85.0274416
4ttAAGtt\NttBrazilttBRttSouth Americatt\Ntt\N
I use this gem to fetch data: https://github.com/tilo/smarter_csv
This is the code I use to show data in terminal console:
filename = 'db/csv/airports_codes.csv'
options = {
:col_sep => 'tt',
}
records = SmarterCSV.process(filename, options)
puts records
I put these files in seeds.rb file because I will modify this code later to seed my database with data. This last line of code is there so I can see how it looks like. So I run rake db:seed
And the output is obviously huge because there are around ~5k lines. Now the first problem is that I can't see all of the data in my terminal. When I scroll to the top this is the first item (note that ID is 4674 which means it displayed last ~250 items):
{:"1"=>4674, :aaa=>"YPJ", :anaa=>"Aupaluk", :french_polynesia=>"Canada", :pf=>"CA", :australia_and_oceania=>"North America", :"_17.352606"=>59.2967, :"_145.509956"=>-69.5997}
How do I see others items?
The second problem is that key names are really weird. How do I rename them, or even better, how do I use arrays instead of hashes?
If you set the option
:headers_in_file => false
in options, that should sort the problem out.
i.e.
filename = 'db/csv/airports_codes.csv'
options = {
:col_sep => 'tt',
:headers_in_file => false
}
records = SmarterCSV.process(filename, options)

Resources