How to test this CSV import rake task? - ruby-on-rails

I have no idea where to start with testing this rake task. Do I need stubs? If so, how to use them? Any help would be appreciated. Thanks!
desc "Import CSV file"
task :import => [:environment] do
data = "db/data.csv"
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.where(name: c).first_or_create }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.where(id: row[0], f_name: row[1]).first_or_create
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.where(name: l.strip, model_1_id: cs[i].id).first_or_create
Model4.where(p_id: p.id, model_3_id: cl.id).first_or_create
}
end
end
end
end

Here's how I'd do it:
1) Happy-path test(s)
Rake tasks as such are a pain to test. Extract the body of the rake task into a class:
whatever.rake
desc "Import CSV file"
task :import => [:environment] do
CSVImporter.new.import "db/data.csv"
end
end
lib/csv_importer.rb
class CsvImporter
def import(data)
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.where(name: c).first_or_create }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.where(id: row[0], f_name: row[1]).first_or_create
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.where(name: l.strip, model_1_id: cs[i].id).first_or_create
Model4.where(p_id: p.id, model_3_id: cl.id).first_or_create
}
end
end
end
end
Now it's easy to write a test that calls CSVImporter.new.import on a test file (that's why import takes the file as a parameter instead of hardcoding it) and expects the results. If it's reasonable to import db/data.csv in the test environment, you can do that in a test if you want. You probably only need one test like this. No stubs are required.
2) Edge and error cases
There is a lot of logic here which, for simplicity and speed, you'll want to test without creating actual model objects. That is, yes, you'll want to stub. Model2.find_ls and FastImage.size are already easy to stub. Let's extract a method to make the other model calls easy to stub:
class CsvImporter
def import(data)
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.first_or_create_with(name: c) }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.first_or_create_with(id: row[0], f_name: row[1])
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.first_or_create_with(name: l.strip, model_1_id: cs[i].id)
Model4.first_or_create_with(p_id: p.id, model_3_id: cl.id)
}
end
end
end
end
app/models/concerns/active_record_extensions.rb
module ActiveRecordExtensions
def first_or_create_with(attributes)
where(attributes).first_or_create
end
end
and include the module in all of the models that need it.
Now it's easy to stub all of the model methods so you can write tests that simulate any database situation you like.

Related

CSV in RUBY custom string

I have 1 field delivery_time It is in an array
include :
DELIVERY_TIME = [
I18n.t("activerecord.attributes.order.none_delivery_time"),
"09:00~12:00",
"12:00~14:00",
"14:00~16:00",
"16:00~18:00",
"18:00~20:00",
"19:00~21:00",
"20:00~21:00",
].freeze
when I downloaded the csv directory it was in the form
"09:00~12:00"
but i want now when I download it will take the form :
"0912"
how to customize it?
my code:
def perform
CSV.generate(headers: true) do |csv|
csv << attributes
orders.each do |order|
csv << create_row(order)
end
end
end
def create_row(order)
row << order.delivery_time
end
AFAIU, you need to modify DELIVERY_TIME to fit your format. CSV is absolutely out of scope here. So to transform values, one should split by ~ and take the hour from the result.
DELIVERY_TIME = [
"09:00~12:00",
"12:00~14:00",
"14:00~16:00",
"16:00~18:00",
"18:00~20:00",
"19:00~21:00",
"20:00~21:00",
].freeze
DELIVERY_TIME.map { |s| s.split('~').map { |s| s[0...2] }.join }
#⇒ ["0912", "1214", "1416", "1618", "1820", "1921", "2021"]
A safer method would be to use DateTime#parse for this
require 'time'
DELIVERY_TIME.map do |s|
s.split('~').map { |s| DateTime.parse(s).strftime("%H") }.join
end
#⇒ ["0912", "1214", "1416", "1618", "1820", "1921", "2021"]
It's not real clear what you're asking, but I'd probably start with something like this:
"09:00~12:00".scan(/\d{2}/).values_at(0, 2).join # => "0912"
Using that in some code:
"09:00~12:00".scan(/\d{2}/).values_at(0, 2).join # => "0912"
DELIVERY_TIME = [
'blah',
"09:00~12:00",
"12:00~14:00",
"14:00~16:00",
"16:00~18:00",
"18:00~20:00",
"19:00~21:00",
"20:00~21:00",
].freeze
ary = [] << DELIVERY_TIME.first
ary += DELIVERY_TIME[1..-1].map { |i|
i.scan(/\d{2}/).values_at(0, 2).join
}
# => ["blah", "0912", "1214", "1416", "1618", "1820", "1921", "2021"]

Rspec test rails generate CSV

I have simple CSV rails generator which tbh I've no idea how to test it with rspec. I was trying to follow this article https://making.dia.com/testing-csv-files-in-rails-on-the-fly-7a1285cc2aac and that one too Testing CSV.generate with RSpec but I just simply don't understand what is going on there, these examples seem to be different.
module AdminLogData
class CsvAdminLogGenerator
LOG_HEADERS = ['Date&Time', 'Action', 'New Data'].freeze
def initialize(start_date:, end_date:)
#start_date = start_date
#end_date = end_date
end
def call
AdminPanelLog.dates_between(start_date, end_date).find_each do |admin_log|
CSV.generate(headers: LOG_HEADERS, col_sep: ';', encoding: 'UTF-8') do |csv|
csv << admin_log.created_at
csv << admin_log.action_type
csv << admin_log.admin_email
csv << admin_log.new_data
end
end
end
end
end
Could someone explain me how to test this class? should I create some fake csv first?
spec
RSpec.describe AdminLogData::CsvAdminLogGenerator do
subject(:csv_file) { described_class.new(start_date, end_date).call }
let(:start_date) { 3.months.ago }
let(:end_date) { 2.months.ago }
let(:header) { 'column1, column2, column3' }
let(:row2) { 'value1, value2, value3' }
let(:row3) { 'value1, value2, value3' }
let(:rows) { [header, row2, row3] }
it 'creates CSV file with proper value' do
expect(csv).to receive(:<<).with(log.created_at)
end
end
Edit:
expected collection contained: ["2019-06-17 22:13:48 +0200,New,monte#kuphal.biz,\"{\"\"email\"\"=>\"\"jeanna#schinner.io\"\", \"\"ro...=>\"\"2019-09-16T22:13:48.752+02:00\"\", \"\"other_activities\"\"=>\"\"forbidden websites\"\"}\"\n"]
actual collection contained: ["2019-06-17 22:13:48 +0200", "New", "monte#kuphal.biz", "{\"email\"=>\"brandon#williamsonsporer.org\... \"last_update\"=>\"2019-09-16T22:13:48.752+02:00\", \"other_activities\"=>\"forbidden websites\"}"]
the missing elements were: ["2019-06-17 22:13:48 +0200,New,monte#kuphal.biz,\"{\"\"email\"\"=>\"\"jeanna#schinner.io\"\", \"\"ro...=>\"\"2019-09-16T22:13:48.752+02:00\"\", \"\"other_activities\"\"=>\"\"forbidden websites\"\"}\"\n"]
the extra elements were: ["2019-06-17 22:13:48 +0200", "New", "monte#kuphal.biz", "{\"email\"=>\"brandon#williamsonsporer.org\... \"last_update\"=>\"2019-09-16T22:13:48.752+02:00\", \"other_activities\"=>\"forbidden websites\"}"]
Step 1: Create a factory for AdminLogData
Using the gem factory_bot combined with faker, you will be able to generate fake AdminLogData objects for your tests.
Step 2: Write your test
In your test, you'll more likely want to test that the CSV lines have the right data instead that the class receives the "<<" message. For instance it could look like that:
RSpec.describe AdminLogData::CsvAdminLogGenerator do
subject(:csv_file) { described_class.new(start_date, end_date).call }
let(:start_date) { 3.months.ago }
let(:end_date) { 2.months.ago }
let(:admin_log_data) { FactoryBot.create(:admin_log_data, created_at: 3.months.ago) }
before { admin_log_data }
it 'creates CSV file with proper value' do
expect(csv_file.to_a[1]).to match_array(CSV.generate_line([
admin_log_data.created_at
admin_log_data.action_type
admin_log_data.admin_email
admin_log_data.new_data
]))
end
end

Rake task to update specific products in db

Hi all I've got a database full of products and I need to schedule a rake task to run every week.
I'm not exactly sure how to code the rake task. An example would be I have a product with id=1 name="testname" description="description" sku="198" price=12.99
I need to upload a csv with name="testname" and update the price to 13.99 while obviously keeping the id intact.
This is what I've tried so far but its not complete if anyone could help that would be great.
require 'csv'
desc "Updates Products inside an ActiveRecord table"
task :update_prods, [:filename] => :environment do
products = Spree::Product.all
CSV.foreach('update_prods.csv', :headers => true) do |row|
Spree::Product.update!(row.to_hash)
end
end
Here is a gist of how we imported products from Shopify to Spree which could give you some ideas on how to go about this https://gist.github.com/dgross881/b4f1ac96bafa2e29be7f.
def update_products
puts 'Updating Products...'
require 'csv'
products_csv = File.read(Rails.root.join('lib/assets/products_list.csv'))
products = CSV.parse(products_csv, headers: true)
products.each_with_index do |row, index|
Rails.logger.info { [#{index + 1}..#{products.length}] Updating product: #{row['title']} }
product = Spree::Product.find!(row['id'])
update_product = product.update_attributes(name: row['title'], description:row['description'],
meta_title: row['seo_title'], meta_description: row['seo_description'],
meta_keywords: "#{row['handle']}, #{row['title']}, the Squirrelz",
available_on: Time.zone.now, price: row['price'],
shipping_category: Spree::ShippingCategory.find_by!(name: 'Shipping'))
update_product.tag_list = row['tags']
update_product.slug = row['handle']
update_product.save!
end
Rails.logger.info { "Finished Updating Products" }
end
def update_variants
puts 'updating Variants...'
require 'csv'
products_variants_csv =File.read(Rails.root.join('lib/assets/variants_list.csv'))
products_variants = CSV.parse(products_variants_csv, headers: true)
products_variants.each_with_index do |row, index|
puts "[#{index + 1}..#{products_variants.length}] Adding Variant (#{row['sku']} to Product: #{Spree::Product.find_by!(slug: row['handle']).name})"
variant = Spree::Variant.find_by!(sku: row['sku']
update_variant = variant.update_attributes!(sku: row['sku'], stock_items_count: row['qty'], cost_price: row['price'], weight: row['weight']
unless row['option1'].blank?
variant.option_values << Spree::OptionValue.find_by!(name: row['option1'])
end
unless row['option2'].blank?
variant.option_values << Spree::OptionValue.find_by!(name: row['option2'])
end
variant.save!
end
puts 'Updated Variants'
end

PostgreSQL monkey patching query execution

I am trying to do something simple. I would like to print the pure sql before its execution and then to print the response when the query finishes. I think that I should monkey patch one of these two methods but the sql queries of the application do not use them.
Any idea how can I do it?
I know that it sounds stupid, but then I will extend this logic.
require 'active_record/connection_adapters/postgresql_adapter'
class ActiveRecord::ConnectionAdapters::PostgreSQLAdapter
def query(sql, name = nil) #:nodoc:
File.open("file1", "a") { |f| f.write sql + "\n\n" }
log(sql, name) do
result_as_array #connection.async_exec(sql)
File.open("file1", "a") { |f| f.write "RESPONSE \n\n" }
end
end
def execute(sql, name = nil)
File.open("file2", "a") { |f| f.write sql + "\n\n" }
log(sql, name) do
#connection.async_exec(sql)
File.open("file2", "a") { |f| f.write "RESPONSE \n\n" }
end
end
end
My main idea is to handle execution methods at least for MySQL and PostgreSQL. I found that I can do it for MySQL this way:
require 'active_record/connection_adapters/mysql2_adapter'
module ActiveRecord
module ConnectionAdapters
class Mysql2Adapter < AbstractMysqlAdapter
def exec_query(sql, name = 'SQL', binds = [])
File.open("path2", "a") { |f| f.write sql + "\n" }
result = execute(sql, name)
ActiveRecord::Result.new(result.fields, result.to_a)
end
end
end
end
Maybe there is a way to handle this for all DB-s in one place?

Rspec Ruby on Rails Test File System in model

I have a model that has a method that looks through the filesystem starting at a particular location for files that match a particular regex. This is executed in an after_save callback. I'm not sure how to test this using Rspec and FactoryGirl. I'm not sure how to use something like FakeFS with this because the method is in the model, not the test or the controller. I specify the location to start in my FactoryGirl factory, so I could change that to a fake directory created by the test in a set up clause? I could mock the directory? I think there are probably several different ways I could do this, but which makes the most sense?
Thanks!
def ensure_files_up_to_date
files = find_assembly_files
add_files = check_add_assembly_files(files)
errors = add_assembly_files(add_files)
if errors.size > 0 then
return errors
end
update_files = check_update_assembly_files(files)
errors = update_assembly_files(update_files)
if errors.size > 0 then
return errors
else
return []
end
end
def find_assembly_files
start_dir = self.location
files = Hash.new
if ! File.directory? start_dir then
errors.add(:location, "Directory #{start_dir} does not exist on the system.")
abort("Directory #{start_dir} does not exist on the system for #{self.inspect}")
end
Find.find(start_dir) do |path|
filename = File.basename(path).split("/").last
FILE_TYPES.each { |filepart, filehash|
type = filehash["type"]
vendor = filehash["vendor"]
if filename.match(filepart) then
files[type] = Hash.new
files[type]["path"] = path
files[type]["vendor"] = vendor
end
}
end
return files
end
def check_add_assembly_files(files=self.find_assembly_files)
add = Hash.new
files.each do |file_type, file_hash|
# returns an array
file_path = file_hash["path"]
file_vendor = file_hash["vendor"]
filename = File.basename(file_path)
af = AssemblyFile.where(:name => filename)
if af.size == 0 then
add[file_path] = Hash.new
add[file_path]["type"] = file_type
add[file_path]["vendor"] = file_vendor
end
end
if add.size == 0 then
logger.error("check_add_assembly_files did not find any files to add")
return []
end
return add
end
def check_update_assembly_files(files=self.find_assembly_files)
update = Hash.new
files.each do |file_type, file_hash|
file_path = file_hash["path"]
file_vendor = file_hash["vendor"]
# returns an array
filename = File.basename(file_path)
af = AssemblyFile.find_by_name(filename)
if !af.nil? then
if af.location != file_path or af.file_type != file_type then
update[af.id] = Hash.new
update[af.id]['path'] = file_path
update[af.id]['type'] = file_type
update[af.id]['vendor'] = file_vendor
end
end
end
return update
end
def add_assembly_files(files=self.check_add_assembly_files)
if files.size == 0 then
logger.error("add_assembly_files didn't get any results from check_add_assembly_files")
return []
end
asm_file_errors = Array.new
files.each do |file_path, file_hash|
file_type = file_hash["type"]
file_vendor = file_hash["vendor"]
logger.debug "file type is #{file_type} and path is #{file_path}"
logger.debug FileType.find_by_type_name(file_type)
file_type_id = FileType.find_by_type_name(file_type).id
header = file_header(file_path, file_vendor)
if file_vendor == "TBA" then
check = check_tba_header(header, file_type, file_path)
software = header[TBA_SOFTWARE_PROGRAM]
software_version = header[TBA_SOFTWARE_VERSION]
elsif file_vendor == "TBB" then
check = check_tbb_header(header, file_type, file_path)
if file_type == "TBB-ANNOTATION" then
software = header[TBB_SOURCE]
else
software = "Unified"
end
software_version = "UNKNOWN"
end
if check == 0 then
logger.error("skipping file #{file_path} because it contains incorrect values for this filetype")
asm_file_errors.push("#{file_path} cannot be added to assembly because it contains incorrect values for this filetype")
next
end
if file_vendor == "TBA" then
xml = header.to_xml(:root => "assembly-file")
elsif file_vendor == "TBB" then
xml = header.to_xml
else
xml = ''
end
filename = File.basename(file_path)
if filename.match(/~$/) then
logger.error("Skipping a file with a tilda when adding assembly files. filename #{filename}")
next
end
assembly_file = AssemblyFile.new(
:assembly_id => self.id,
:file_type_id => file_type_id,
:name => filename,
:location => file_path,
:file_date => creation_time(file_path),
:software => software,
:software_version => software_version,
:current => 1,
:metadata => xml
)
assembly_file.save! # exclamation point forces it to raise an error if the save fails
end # end files.each
return asm_file_errors
end
Quick answer: you can stub out model methods like any others. Either stub a specific instance of a model, and then stub find or whatever to return that, or stub out any_instance to if you don't want to worry about which model is involved. Something like:
it "does something" do
foo = Foo.create! some_attributes
foo.should_receive(:some_method).and_return(whatever)
Foo.stub(:find).and_return(foo)
end
The real answer is that your code is too complicated to test effectively. Your models should not even know that a filesystem exists. That behavior should be encapsulated in other classes, which you can test independently. Your model's after_save can then just call a single method on that class, and testing whether or not that single method gets called will be a lot easier.
Your methods are also very difficult to test, because they are trying to do too much. All that conditional logic and external dependencies means you'll have to do a whole lot of mocking to get to the various bits you might want to test.
This is a big topic and a good answer is well beyond the scope of this answer. Start with the Wikipedia article on SOLID and read from there for some of the reasoning behind separating concerns into individual classes and using tiny, composed methods. To give you a ballpark idea, a method with more than one branch or more than 10 lines of code is too big; a class that is more than about 100 lines of code is too big.

Resources