Hi all I've got a database full of products and I need to schedule a rake task to run every week.
I'm not exactly sure how to code the rake task. An example would be I have a product with id=1 name="testname" description="description" sku="198" price=12.99
I need to upload a csv with name="testname" and update the price to 13.99 while obviously keeping the id intact.
This is what I've tried so far but its not complete if anyone could help that would be great.
require 'csv'
desc "Updates Products inside an ActiveRecord table"
task :update_prods, [:filename] => :environment do
products = Spree::Product.all
CSV.foreach('update_prods.csv', :headers => true) do |row|
Spree::Product.update!(row.to_hash)
end
end
Here is a gist of how we imported products from Shopify to Spree which could give you some ideas on how to go about this https://gist.github.com/dgross881/b4f1ac96bafa2e29be7f.
def update_products
puts 'Updating Products...'
require 'csv'
products_csv = File.read(Rails.root.join('lib/assets/products_list.csv'))
products = CSV.parse(products_csv, headers: true)
products.each_with_index do |row, index|
Rails.logger.info { [#{index + 1}..#{products.length}] Updating product: #{row['title']} }
product = Spree::Product.find!(row['id'])
update_product = product.update_attributes(name: row['title'], description:row['description'],
meta_title: row['seo_title'], meta_description: row['seo_description'],
meta_keywords: "#{row['handle']}, #{row['title']}, the Squirrelz",
available_on: Time.zone.now, price: row['price'],
shipping_category: Spree::ShippingCategory.find_by!(name: 'Shipping'))
update_product.tag_list = row['tags']
update_product.slug = row['handle']
update_product.save!
end
Rails.logger.info { "Finished Updating Products" }
end
def update_variants
puts 'updating Variants...'
require 'csv'
products_variants_csv =File.read(Rails.root.join('lib/assets/variants_list.csv'))
products_variants = CSV.parse(products_variants_csv, headers: true)
products_variants.each_with_index do |row, index|
puts "[#{index + 1}..#{products_variants.length}] Adding Variant (#{row['sku']} to Product: #{Spree::Product.find_by!(slug: row['handle']).name})"
variant = Spree::Variant.find_by!(sku: row['sku']
update_variant = variant.update_attributes!(sku: row['sku'], stock_items_count: row['qty'], cost_price: row['price'], weight: row['weight']
unless row['option1'].blank?
variant.option_values << Spree::OptionValue.find_by!(name: row['option1'])
end
unless row['option2'].blank?
variant.option_values << Spree::OptionValue.find_by!(name: row['option2'])
end
variant.save!
end
puts 'Updated Variants'
end
Related
So, I have this controller, it gets the data from the uploaded file and save it to the right models. I made it work, but it is gigantic. I would like some directions of how to make it smaller. Should I create a class to extract the data and save to the db?
class UploadsController < ApplicationController
require 'csv'
def save_info_to_db
file = params[:file]
purchases_made = []
CSV.open(file.path, "r", { col_sep: "\t", headers: true }).each do |row|
data = Hash.new
data = {
purchaser: { name: row[0] },
item: { description: row[1], price: row[2], quantity: row[3] },
merchant: { name: row[4], address: row[5] }
}
merchant = Merchant.create_with(address: data[:merchant_address]).find_or_create_by(name: data[:merchant][:name])
item = merchant.items.create_with(price: data[:item][:price]).find_or_create_by(description: data[:item][:description])
purchaser = Purchaser.create(name: data[:purchaser][:name])
purchase = purchaser.purchases.create
purchase_items = PurchaseItem.create(item_id: item.id, quantity: data[:item][:quantity])
purchase.add_purchase_items(purchase_items)
purchases_made << purchase.total_price
end
session[:total_gross_income] = Purchase.total_gross_income(purchases_made)
redirect_to display_incomes_path
end
end
Thank you!
Lots of ways to do this, depending on what you prefer, but one idea is to use a Service class. In the service, you could handle the file processing.
Your controller would then end up looking something like:
class UploadsController < ApplicationController
require 'csv'
def save_info_to_db
file_service = FileProcessingService.new(file)
file_service.save_info!
session[:total_gross_income] = Purchase.total_gross_income(file_service.purchases_made)
redirect_to display_incomes_path
rescue
# perhaps some rescue logic in the event processing fails
end
All your file processing logic would go in the Service. You could even split that logic up -- logic for getting the data from the file, then separate logic for saving the data.
Lots of options, but the general idea is you can use a service to handle logic that may span multiple models.
Hello Ruby users i have Json Array format
[
"Can also work with any bluetooth-enabled smartphones and\ntablets",
"For calls and music, Hands-free",
"Very stylish design and lightweight",
"Function:Bluetooth,Noise Cancelling,Microphone",
"Compatible:For Any Device With Bluetooth Function",
"Chipset: CSR4.0", "Bluetooth Version:Bluetooth 4.0",
"Transmission Distance:10 Meters"
]
I want to save this array into html formal using the html format which is below.
<ul>
<li>Can also work with any bluetooth-enabled smartphones and\ntablets</li>
<li>For calls and music, Hands-free</li>
<li>Very stylish design and lightweight</li>
<li>Function:Bluetooth,Noise Cancelling,Microphone</li>
<li>Compatible:For Any Device With Bluetooth Function</li>
<li>Chipset: CSR4.0</li>
<li>Bluetooth Version:Bluetooth 4.0</li>
<li>Transmission Distance:10 Meters</li>
</ul>
This is my current code which is working correctly if i have to save it just the array however i need this to be html format so user can easily read it
result = JSON.parse(jsonparse)
result["mods"]["listItems"].each do |result|
#item = Item.new
#item.item_details = result["description"]
#item.save
end
Under my Previous Attempt to solve like this
result = JSON.parse(jsonparse)
result["mods"]["listItems"].each do |result|
#item = Item.new
item_list = result["description"]
item_list.each do |list|
#item.item_details = "<li>"+list+"</li>"
end
#item.save
end
This one only save one of the array and no <ul> head
heres the original code
namespace :scraper do
desc "Scrape Website"
task somesite: :environment do
require 'open-uri'
require 'nokogiri'
require 'json'
url = "url here!"
page = Nokogiri::HTML(open(url))
script = page.search('head script')[2]
jsonparse = script.content[/\{\"[a-zA-Z0-9\"\:\-\,\ \=\(\)\.\_\D\/\[\]\}]+/i]
result = JSON.parse(jsonparse)
result["mods"]["listItems"].each do |result|
#item = Item.new
item_details = result["description"].each {|list| "<li>#{list}</li>" }
puts item_details
#item.item_old_price = result["originalPrice"]
#item.item_final_price = result["price"]
#item.save
end
end
end
The Idea is to save the Array into database with the html format.
<ul>
<li>content 1</li>
<li>content 2</li>
<li>content and soon</li>
</ul>
Thanks
I'm not sure what you tried to do.
Please check below codes and give me a feedback.
Happy coding :)
<!-- language: ruby -->
require 'json'
class MyJsonParser
def initialize
#items = []
end
def parse(json)
result = JSON.parse(json)
generate_items(result)
items
end
private
attr_reader :items
def generate_items(result)
result['mods']['listItems'].each {|item_detail| items << Item.new(item_detail)}
end
end
class Item
attr_reader :details
def initialize(detail='')
#details = ''
before_initialize
details << detail
after_initialize
end
private
def before_initialize
details << '<li>'
end
def after_initialize
details << '</li>'
end
end
json_str = '{
"mods": {
"listItems":
[
"Can also work with any bluetooth-enabled smartphones and\ntablets",
"For calls and music, Hands-free",
"Very stylish design and lightweight",
"Function:Bluetooth,Noise Cancelling,Microphone",
"Compatible:For Any Device With Bluetooth Function",
"Chipset: CSR4.0",
"Bluetooth Version:Bluetooth 4.0",
"Transmission Distance:10 Meters"
]
}
}'
result = MyJsonParser.new.parse(json_str)
result.each do |i|
p i.details
end
I am doing scraping to fetch the data from the website to my database in rails.I am fetching the 32000 record with this script there isn't any issue but i want to fetch the data faster so i apply the thread in my rake task but then there is a issue while running the rake task some of the data is fetching then the rake task getting aborted.
I am not aware of what to do task if any help can be done i am really grateful . Here is my rake task code for the scraping.
task scratch_to_database: :environment do
time2 = Time.now
puts "Current Time : " + time2.inspect
client = Mechanize.new
giftcard_types=Giftcard.card_types
find_all_merchant=Merchant.all.pluck(:id, :name).to_h
#first index page of the merchant
index_page = client.get('https://www.twitter.com//')
document_page_index = Nokogiri::HTML::Document.parse(index_page.body)
#set all merchant is deteled true
# set_merchant_as_deleted = Merchant.update_all(is_deleted: true) if Merchant.exists?
# set_giftcard_as_deleted = Giftcard.update_all(is_deleted: true) if Giftcard.exists?
update_all_merchant_record = []
update_all_giftcard_record = []
threads = []
#Merchant inner page pagination loop
page_no_merchant = document_page_index.css('.pagination.pagination-centered ul li:nth-last-child(2) a').text.to_i
1.upto(page_no_merchant) do |page_number|
threads << Thread.new do
client.get("https://www.twitter.com/buy-gift-cards?page=#{page_number}") do |page|
document = Nokogiri::HTML::Document.parse(page.body)
#Generate the name of the merchant and image of the merchant loop
document.css('.product-source').each do |item|
merchant_name= item.children.css('.name').text.gsub("Gift Cards", "")
href = item.css('a').first.attr('href')
image_url=item.children.css('.img img').attr('data-src').text.strip
#image url to parse the url of the image
image_url=URI.parse(image_url)
#saving the record of the merchant
# #merchant=Merchant.create(name: merchant_name , image_url:image_url)
if find_all_merchant.has_value?(merchant_name)
puts "this if"
merchant_id=find_all_merchant.key(merchant_name)
puts merchant_id
else
#merchant= Merchant.create(name: merchant_name , image_url:image_url)
update_all_merchant_record << #merchant.id
merchant_id=#merchant.id
end
# #merchant.update_attribute(:is_deleted, false)
#set all giftcard is deteled true
# set_giftcard_as_deleted = Giftcard.where(merchant_id: #merchant.id).update_all(is_deleted: true) if Giftcard.where(merchant_id: #merchant.id).exists?
#first page of the giftcard details page
first_page = client.get("https://www.twitter.com#{href}")
document_page = Nokogiri::HTML::Document.parse(first_page.body)
page_no = document_page.css('.pagination.pagination-centered ul li:nth-last-child(2) a').text.to_i
hrefextra =document_page.css('.dropdown-menu li a').last.attr('href')
#generate the giftcard details loop with the pagination
# update_all_record = []
find_all_giftcard=Giftcard.where(merchant_id:merchant_id).pluck(:row_id)
puts merchant_name
# puts find_all_giftcard.inspect
card_page = client.get("https://www.twitter.com#{hrefextra}")
document_page = Nokogiri::HTML::Document.parse(card_page.body)
#table details to generate the details of the giftcard with price ,per_off and final value of the giftcard
document_page.xpath('//table/tbody/tr[#class="toggle-details"]').collect do |row|
type1=[]
row_id = row.attr("id").to_i
row.at("td[2] ul").children.each do |typeli|
type = typeli.text.strip if typeli.text.strip.length != 0
type1 << type if typeli.text.strip.length != 0
end
value = row.at('td[3]').text.strip
value = value.to_s.tr('$', '').to_f
per_discount = row.at('td[4]').text.strip
per_discount = per_discount.to_s.tr('%', '').to_f
final_price = row.at('td[5] strong').text.strip
final_price = final_price.to_s.tr('$', '').to_f
type1.each do |type|
if find_all_giftcard.include?(row_id)
update_all_giftcard_record<<row_id
puts "exists"
else
puts "new"
#giftcard= Giftcard.create(card_type: giftcard_types.values_at(type.to_sym)[0], card_value:value, per_off:per_discount, card_price: final_price, merchant_id: merchant_id , row_id: row_id )
update_all_giftcard_record << #giftcard.row_id
end
end
#saving the record of the giftcard
# #giftcard=Giftcard.create(card_type:1, card_value:value, per_off:per_discount, card_price: final_price, merchant_id: #merchant.id , gift_card_type: type1)
end
# Giftcard.where(:id =>update_all_record).update_all(:is_deleted => false)
#delete all giftcard which is not present
# giftcard_deleted = Giftcard.where(:is_deleted => true,:merchant_id => #merchant.id).destroy_all if Giftcard.where(merchant_id: #merchant.id).exists?
time2 = Time.now
puts "Current Time : " + time2.inspect
end
end
end
end
threads.each(&:join)
puts "-------"
puts threads
# merchant_deleted = Merchant.where(:is_deleted => true).destroy_all if Merchant.exists?
merchant_deleted = Merchant.where('id NOT IN (?)',update_all_merchant_record).destroy_all if Merchant.exists?
giftcard_deleted = Giftcard.where('row_id NOT IN (?)',update_all_giftcard_record).destroy_all if Giftcard.exists?
end
end
Error i am receiving:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use
Each thread requires a separate connection to your database. You need to increase the connection pool size that your application can use in your database.yml file.
But your database should also be capable of handling the incoming connections. If you are using mysql you can check this by running select ##MAX_CONNECTIONS on your console.
I have no idea where to start with testing this rake task. Do I need stubs? If so, how to use them? Any help would be appreciated. Thanks!
desc "Import CSV file"
task :import => [:environment] do
data = "db/data.csv"
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.where(name: c).first_or_create }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.where(id: row[0], f_name: row[1]).first_or_create
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.where(name: l.strip, model_1_id: cs[i].id).first_or_create
Model4.where(p_id: p.id, model_3_id: cl.id).first_or_create
}
end
end
end
end
Here's how I'd do it:
1) Happy-path test(s)
Rake tasks as such are a pain to test. Extract the body of the rake task into a class:
whatever.rake
desc "Import CSV file"
task :import => [:environment] do
CSVImporter.new.import "db/data.csv"
end
end
lib/csv_importer.rb
class CsvImporter
def import(data)
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.where(name: c).first_or_create }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.where(id: row[0], f_name: row[1]).first_or_create
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.where(name: l.strip, model_1_id: cs[i].id).first_or_create
Model4.where(p_id: p.id, model_3_id: cl.id).first_or_create
}
end
end
end
end
Now it's easy to write a test that calls CSVImporter.new.import on a test file (that's why import takes the file as a parameter instead of hardcoding it) and expects the results. If it's reasonable to import db/data.csv in the test environment, you can do that in a test if you want. You probably only need one test like this. No stubs are required.
2) Edge and error cases
There is a lot of logic here which, for simplicity and speed, you'll want to test without creating actual model objects. That is, yes, you'll want to stub. Model2.find_ls and FastImage.size are already easy to stub. Let's extract a method to make the other model calls easy to stub:
class CsvImporter
def import(data)
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.first_or_create_with(name: c) }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.first_or_create_with(id: row[0], f_name: row[1])
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.first_or_create_with(name: l.strip, model_1_id: cs[i].id)
Model4.first_or_create_with(p_id: p.id, model_3_id: cl.id)
}
end
end
end
end
app/models/concerns/active_record_extensions.rb
module ActiveRecordExtensions
def first_or_create_with(attributes)
where(attributes).first_or_create
end
end
and include the module in all of the models that need it.
Now it's easy to stub all of the model methods so you can write tests that simulate any database situation you like.
I have a rake task that should import a .csv file and save the data into the database. So far it runs but when I check the database - nothing was saved and I see no errors - leaving me with no direction.
I'm using Rails 4.2 with Postgresql for the db.
Here is my task..
namespace :import_users do
desc "imports user data from a csv file"
task :data => :environment do
require 'csv'
CSV.foreach('/Users/RAILS/Extra Files/2015 User Report.csv') do |row|
plan = row[0]
contact_ident = row[1]
prefer_fax = row[2]
pin = row[3]
name = row[4] #account
first_name = row[5]
last_name = row[6]
email = row[7]
phone = row[8]
prefer_fax == "Yes" ? p_fax = true : p_fax = false
p = Plan.where("name ~ ?","#{plan}( )")
user = User.create( contact_ident: contact_ident,
prefer_fax: p_fax,
first_name: first_name,
last_name: last_name,
email: email
)
account = Account.where("pin ~ ?", "#{pin}").first_or_create do |account|
account.name = name
account.phone = phone
end
user.plans << p
user.accounts << account
end
end
end
Can you try to replace
User.create
with
User.create!
so if there is any problem with creation it raise.