I have a CsvImport service object in my app/services and I'm trying to call one of the class methods from within a Worker.
class InventoryUploadWorker
include Sidekiq::Worker
def perform(file_path, company_id)
CsvImport.csv_import(file_path, Company.find(company_id))
end
end
But it seems that the worker doesn't know what the class is, I've attempted require 'csv_import' to no avail.
Heres where it breaks:
WARN: ArgumentError: undefined class/module CsvImport
The method being called in
csv_import.rb
class CsvImport
require "benchmark"
require 'csv'
def self.csv_import(filename, company)
time = Benchmark.measure do
File.open(filename) do |file|
headers = file.first
file.lazy.each_slice(150) do |lines|
Part.transaction do
inventory = []
insert_to_parts_db = []
rows = CSV.parse(lines.join, write_headers: true, headers: headers)
rows.map do |row|
part_match = Part.find_by(part_num: row['part_num'])
new_part = build_new_part(row['part_num'], row['description']) unless part_match
quantity = row['quantity'].to_i
row.delete('quantity')
row["condition"] = match_condition(row)
quantity.times do
part = InventoryPart.new(
part_num: row["part_num"],
description: row["description"],
condition: row["condition"],
serial_num: row["serial_num"],
company_id: company.id,
part_id: part_match ? part_match.id : new_part.id
)
inventory << part
end
end
InventoryPart.import inventory
end
end
end
end
puts time
end
your requires are inside the class. Put them outside the class so they're required right away when the file is loaded, not when the class is loaded.
Instead of
class CsvImport
require "benchmark"
require 'csv'
...
Do this
require "benchmark"
require 'csv'
class CsvImport
...
Try to add to application.rb
config.autoload_paths += Dir["#{config.root}/app/services"]
More details here: autoload-paths
Related
I get NoMethodError when I run test for the code below
csv_importer.rb
require 'csv_importer/engine'
class WebImport
def initialize(url)
#url = url
end
def call
url = 'http://example.com/people.csv'
csv_string = open(url).read.force_encoding('UTF-8')
string_to_users(csv_string)
end
def string_to_users(csv_string)
counter = 0
duplicate_counter = 0
user = []
CSV.parse(csv_string, headers: true, header_converters: :symbol) do |row|
next unless row[:name].present? && row[:email_address].present?
user = CsvImporter::User.create row.to_h
if user.persisted?
counter += 1
else
duplicate_counter += 1
end
end
p "Email duplicate record: #{user.email_address} - #{user.errors.full_messages.join(',')}" if user.errors.any?
p "Imported #{counter} users, #{duplicate_counter} duplicate rows ain't added in total"
end
end
csv_importer_test.rb
require 'csv_importer/engine'
require 'test_helper'
require 'rake'
class CsvImporterTest < ActiveSupport::TestCase
test 'truth' do
assert_kind_of Module, CsvImporter
end
test 'should override_application and import data' do
a = WebImport.new(url: 'http://example.com/people.csv')
a.string_to_users('Olaoluwa Afolabi')# <-- I still get error even I put a comma separated list of attributes that is imported into the db here.
assert_equal User.count, 7
end
end
csv format in the url in the code:
This saves into DB once I run the Rake Task
Name,Email Address,Telephone Number,Website
Coy Kunde,stone#stone.com,0800 382630,mills.net
What I have done to debug:
I use byebug and I figured out the in csv_importer_test.rb, the line where I have a.string_to_users('Olaoluwa Afolabi') is throwing error. See byebug error below:
So, I when I run rails test, I get the error below:
So, how do I solve this error, I have no clue what am doing wrong??
If you don't have any row in your csv_string, this line:
user = CsvImporter::User.create row.to_h
isn't executed, so user variable holds previous value, which is []:
user = []
As we know, there's no method errors defined for Array, yet you try to call it in this line:
p "Email duplicate record: #{user.email_address} - #{user.errors.full_messages.join(',')}" if user.errors.any?
and that's why you get an error.
Looking for advice on how to fix this error and refactor this code to improve it.
require 'mechanize'
require 'pry'
require 'pp'
module Mymodule
class WebBot
agent = Mechanize.new { |agent|
agent.user_agent_alias = 'Windows Chrome'
}
def form(response)
require "addressable/uri"
require "addressable/template"
template = Addressable::Template.new("http://www.domain.com/{?query*}")
url = template.expand({"query" => response}).to_s
page = agent.get(url)
end
def get_products
products = []
page.search("datatable").search('tr').each do |row|
begin
product = row.search('td')[1].text
rescue => e
p e.message
end
products << product
end
products
end
end
end
Calling the module:
response = {size: "SM", color: "BLUE"}
t = Mymodule::WebBot.new
t.form(response)
t.get_products
Error:
NameError: undefined local variable or method `agent'
Ruby has a naming convention. agent is a local variable in the class scope. To make it visible to other methods you should make it a class variable by naming it ##agent, and it'll be shared among all the objects of WebBot. The preferred way though is to make it an instance variable by naming it #agent. Every object of WebBot will have its own #agent. But you should put it in initialize, initialize will be invoked when you create a new object with new
class WebBot
def initialize
#agent = Mechanize.new do |a|
a.user_agent_alias = 'Windows Chrome'
end
end
.....
And the same error will occur to page. You defined it in form as a local variable. When form finishes execution, it'll be deleted. You should make it an instance variable. Fortunately, you don't have to put it in initialize. You can define it here in form. And the object will have its own #page after invoking form. Do this in form:
def form(response)
require "addressable/uri"
require "addressable/template"
template = Addressable::Template.new("http://www.domain.com/{?query*}")
url = template.expand({"query" => response}).to_s
#page = agent.get(url)
end
And remember to change every occurrence of page and agent to #page and #agent. In your get_products for example:
def get_products
products = []
#page.search("datatable").search('tr').each do |row|
.....
These changes will resolve the name errors. Refactoring is another story btw.
I have a Ruby gem which wraps an API. I have two classes: Client and Season with a Configuration module. But I can't access a change to the API Key, Endpoint made via Client in the Season class.
My ApiWrapper module looks like this:
require "api_wrapper/version"
require 'api_wrapper/configuration'
require_relative "api_wrapper/client"
require_relative "api_wrapper/season"
module ApiWrapper
extend Configuration
end
My Configuration module looks like this:
module ApiWrapper
module Configuration
VALID_CONNECTION_KEYS = [:endpoint, :user_agent, :method].freeze
VALID_OPTIONS_KEYS = [:api_key, :format].freeze
VALID_CONFIG_KEYS = VALID_CONNECTION_KEYS + VALID_OPTIONS_KEYS
DEFAULT_ENDPOINT = 'http://defaulturl.com'
DEFAULT_METHOD = :get
DEFAULT_API_KEY = nil
DEFAULT_FORMAT = :json
attr_accessor *VALID_CONFIG_KEYS
def self.extended(base)
base.reset
end
def reset
self.endpoint = DEFAULT_ENDPOINT
self.method = DEFAULT_METHOD
self.user_agent = DEFAULT_USER_AGENT
self.api_key = DEFAULT_API_KEY
self.format = DEFAULT_FORMAT
end
def configure
yield self
end
def options
Hash[ * VALID_CONFIG_KEYS.map { |key| [key, send(key)] }.flatten ]
end
end # Configuration
end
My Client class looks like this:
module ApiWrapper
class Client
attr_accessor *Configuration::VALID_CONFIG_KEYS
def initialize(options={})
merged_options = ApiWrapper.options.merge(options)
Configuration::VALID_CONFIG_KEYS.each do |key|
send("#{key}=", merged_options[key])
end
end
end # Client
end
My Season class looks like this:
require 'faraday'
require 'json'
API_URL = "/seasons"
module ApiWrapper
class Season
attr_accessor *Configuration::VALID_CONFIG_KEYS
attr_reader :id
def initialize(attributes)
#id = attributes["_links"]["self"]["href"]
...
end
def self.all
puts ApiWrapper.api_key
puts ApiWrapper.endpoint
conn = Faraday.new
response = Faraday.get("#{ApiWrapper.endpoint}#{API_URL}/") do |request|
request.headers['X-Auth-Token'] = "ApiWrapper.api_key"
end
seasons = JSON.parse(response.body)
seasons.map { |attributes| new(attributes) }
end
end
end
This is the test I am running:
def test_it_gives_back_a_seasons
VCR.use_cassette("season") do
#config = {
:api_key => 'ak',
:endpoint => 'http://ep.com',
}
client = ApiWrapper::Client.new(#config)
result = ApiWrapper::Season.all
# Make sure we got all season data
assert_equal 12, result.length
#Make sure that the JSON was parsed
assert result.kind_of?(Array)
assert result.first.kind_of?(ApiWrapper::Season)
end
end
Because I set the api_key via the client to "ak" and the endpoint to "http://ep.com" I would expect puts in the Season class's self.all method to print out "ak" and "http://ep.com", but instead I get the defaults set in the Configuration section.
What I am doing wrong?
The api_key accessors you have on Client and on ApiWrapper are independent. You initialize a Client with the key you want, but then Season references ApiWrapper directly. You've declared api_key, etc. accessors in three places: ApiWrapper::Configuration, ApiWrapper (by extending Configuration) and Client. You should probably figure out what your use cases are and reduce that down to being in just one place to avoid confusion.
If you're going to have many clients with different API keys as you make different requests, you should inject the client into Season and use it instead of ApiWrapper. That might look like this:
def self.all(client)
puts client.api_key
puts client.endpoint
conn = Faraday.new
response = Faraday.get("#{client.endpoint}#{API_URL}/") do |request|
request.headers['X-Auth-Token'] = client.api_key
end
seasons = JSON.parse(response.body)
seasons.map { |attributes| new(attributes) }
end
Note that I also replaced the "ApiWrapper.api_key" string with the client.api_key - you don't want that to be a string anyway.
Having to pass client into every request you make is going to get old, so then you might want to pull out something like a SeasonQuery class to hold onto it.
If you're only ever going to have one api_key and endpoint for the duration of your execution, you don't really need the Client as you've set it up so far. Just set ApiWrapper.api_key directly and continue using it in Season.
So I wrote some nokogiri code that works in a test .rb file but when I put it inside a rails app model it won't iterate and just returns the first value. Here is the code that iterates correctly:
require "rubygems"
require "open-uri"
require "nokogiri"
url = "http://www.ebay.com/sch/Cars-Trucks-/6001/i.html?_from=R40&_sac=1&_vxp=mtr&_nkw=car+projects&_ipg=200&rt=nc"
data = Nokogiri::HTML(open(url))
data.css(".li").each do |item|
item_link = item.at_css(".vip")[:href]
item_doc = Nokogiri::HTML(open(item_link))
puts item_doc.at_css("#itemTitle").text.sub! 'Details about', ''
end
Here is the same code in a rails app that only returns the first title it finds:
require "rubygems"
require "open-uri"
require "nokogiri"
class EbayScraper
attr_accessor :url, :data
def initialize(url)
#url = url
end
def data
#data ||= Nokogiri::HTML(open(#url))
end
def titles
data.css(".li").each do |item|
item_link = item.at_css(".vip")[:href]
item_data = Nokogiri::HTML(open(item_link))
return item_data.at_css("#itemTitle").text.sub! 'Details about', ''
end
end
ebay = EbayScraper.new("http://www.ebay.com/sch/Cars-Trucks-/6001/i.html?_from=R40&_sac=1&_vxp=mtr&_nkw=car+projects&_ipg=200&rt=nc")
titles = ebay.titles
puts titles
Why does the first code iterate through the whole thing and the second bunch of code just returns the first one?
Thanks for your time in advance!
Because you have a return statement in your loop that exits your titles function.
There is a table questions, and a data file questions.yml. Assume there is no 'Question' model.
'questions.yml' has some recodes dump from the table.
---
questions_001:
title: ttt1
content: ccc1
questions_002:
title: ttt2
content: ccc2
I want to load the data from the yml file, insert them to database. But I can't use rake db:fixtures:load, because it will treat the content as 'erb' template, which is not want I want
So I want to write another rake task, to load the data manually.
I can read the records by:
File.open("#{RAILS_ROOT}/db/fixtures/#{table_name}.yml", 'r') do |file|
YAML::load(file).each do |record|
# how to insert the record??
end
end
But I don't know how to insert them.
Edit:
I have tried:
Class.new(ActiveRecord::Base).create(record)
and
class Dummy < ActiveRecord::Base {}
Dummy.create(rcord)
But nothing inserted to database
Try this after loading the date from the yml file to records:
class Question < ActiveRecord::Base
# Question model just to import the yml file
end
records.each { |record| Question.create(record) }
You can simply create a model just for importing. You don't need to create the app/models/question.rb. Just write the code above in the script responsible for the importing.
UPDATE:
You can use the following function:
def create_class(class_name, superclass, &block)
klass = Class.new superclass, &block
Object.const_set class_name, klass
end
source
File.open("#{RAILS_ROOT}/db/fixtures/#{table_name}.yml", 'r') do |file|
YAML::load(file).each do |record|
model_name = table_name.singularize.camelize
create_class(model_name, ActiveRecod::Base) do
set_table_name table_name.to_sym
end
Kernel.const_get(model_name).create(record)
end
end
To use the connection directly you can use the following:
ActiveRecord::Base.connection.execute("YOUR SQL CODE")
Got it working thanks to #jigfox 's answer. Had to modify a bit for the full implementation now with Rails 4.
table_names = Dir.glob(Rails.root + 'app/models/**.rb').map { |s| Pathname.new(s).basename.to_s.gsub(/\.rb$/,'') }
table_names.each do |table_name|
table_name = table_name.pluralize
path = "#{Rails.root}/db/fixtures/#{table_name}.yml"
if File.exists?(path)
File.open(path, 'r') do |file|
y = YAML::load(file)
if !y.nil? and y
y.each do |record|
model_name = table_name.singularize.camelize
rec = record[1]
rec.tap { |hs| hs.delete("id") }
Kernel.const_get(model_name).create(rec)
end
end
end
end
end
This loads fixtures into the current RAILS_ENV, which, by default, is development.
$ rake db:fixtures:load