I am trying to save my csv data to table items which is associated with Item model.
This is what my csv have:
'name';'number';'sub_category_id';'category_id';'quantity';'sku'; 'description';'cost_price';'selling_price'
'Uploaded Item Number 1';'54';'KRT';'WN';'67';'WNKRT0054';'Some Description here!!';'780';'890'
'Uploaded Item Number 2';'74';'KRT';'WN';'98;'WNKRT0074';'Some Description here!!';'8660';'9790'
First row show the fields for items table.
Here I am using fastercsv to process my csv and paperclip to upload.
I am able to process file read content and able to fill up the field too here is the processing code:
def proc_csv
#import = Import.find(params[:id])
#lines = parse_csv_file(#import.csv.path)
#lines.shift
#lines.each do |line , j|
unless line.nil?
line_split = line.split(";")
unless ((line_split[0].nil?) or (line_split[1].nil?) or (line_split[2].nil?) or (line_split[3].nil?) or (line_split[4].nil?) or (line_split[5].nil?))
# I used puts to get to know about what's going on.
puts "*"*50+"line_split[0]: #{line_split[0]}"+"*"*50
puts "*"*50+"line_split[1]: #{line_split[1]}"+"*"*50
puts "*"*50+"line_split[2]: #{line_split[2]}"+"*"*50
puts "*"*50+"line_split[3]: #{line_split[3]}"+"*"*50
puts "*"*50+"line_split[4]: #{line_split[4]}"+"*"*50
puts "*"*50+"line_split[5]: #{line_split[5]}"+"*"*50
puts "*"*50+"line_split[6]: #{line_split[6]}"+"*"*50
puts "*"*50+"line_split[7]: #{line_split[7]}"+"*"*50
puts "*"*50+"line_split[8]: #{line_split[8]}"+"*"*50
#item = [:name => line_split[0], :number => line_split[1], :sub_category_id => line_split[2],:category_id => line_split[3],:quantity => line_split[4], :sku => line_split[5], :description => line_split[6], :cost_price => line_split[7], :selling_price => line_split[8]]
puts "#"*100+"#item is: #{#item.inspect}"+"#"*100
end
end
end
redirect_to import_path(#import)
end
but the problem is that when it process it and when I check the #item in console it looks like this:
#####################################################################################################item is: [{:quantity=>"\000'\0006\0007\000'\000", :name=>"\000'\000U\000p\000l\000o\000a\000d\000e\000d\000 \000I\000t\000e\000m\000 \000N\000u\000m\000b\000e\000r\000 \0001\000'\000", :sku=>"\000'\000W\000N\000K\000R\000T\0000\0000\0005\0004\000'\000", :cost_price=>"\000'\0007\0008\0000\000'\000", :number=>"\000'\0005\0004\000'\000", :selling_price=>"\000'\0008\0009\0000\000'\000", :sub_category_id=>"\000'\000K\000R\000T\000'\000", :description=>"\000'\000S\000o\000m\000e\000 \000D\000e\000s\000c\000r\000i\000p\000t\000i\000o\000n\000 \000h\000e\000r\000e\000!\000!\000'\000", :category_id=>"\000'\000W\000N\000'\000"}]####################################################################################################
#####################################################################################################item is: [{:quantity=>"\000'\0009\0008\000", :name=>"\000'\000U\000p\000l\000o\000a\000d\000e\000d\000 \000I\000t\000e\000m\000 \000N\000u\000m\000b\000e\000r\000 \0002\000'\000", :sku=>"\000'\000W\000N\000K\000R\000T\0000\0000\0007\0004\000'\000", :cost_price=>"\000'\0008\0006\0006\0000\000'\000", :number=>"\000'\0007\0004\000'\000", :selling_price=>"\000'\0009\0007\0009\0000\000'\000", :sub_category_id=>"\000'\000K\000R\000T\000'\000", :description=>"\000'\000S\000o\000m\000e\000 \000D\000e\000s\000c\000r\000i\000p\000t\000i\000o\000n\000 \000h\000e\000r\000e\000!\000!\000'\000", :category_id=>"\000'\000W\000N\000'\000"}]####################################################################################################
Can anyone kindly tell me why am I getting this kind of string instead of simple string I entered in my csv file? And because of this it's not being saved into the items table too, I have tried all possible formats but nothing seems to be working. I want :name => "Uploaded Item Number 1" instead of :name=>"\000'\000U\000p\000l\000o\000a\000d\000e\000d\000 \000I\000t\000e\000m\000 \000N\000u\000m\000b\000e\000r\000 \0001\000'\000" . Any help will be appreciated. Thanks in advance :)
After punching my head on to the wall and getting frustrated with this issue, I figured out that it was my CSV file that was not in proper format (was in UTF16le) and when I made an another csv file in UTF-8 encoding, I get the break through. Though I'm left with one more issue which is the string that comes is like this: :name => "'Uploaded Item number 1'" So when it saves into the database the column contains the data is: 'Uploaded Item number 1' . Do you have any idea how can I do it to like this: :name => "Uploaded Item number 1" ?
Related
I'm trying to download a CSV file from an FTP server and if the record exists I want to update that record and not create a duplicate. To give a little more context - I'm trying to upload a group of orders from an FTP folder into my Rails app. There is a new file every hour - sometimes the orders in a certain drop contain duplicates from the previous drop to prevent one from slipping through the tracks or on occasion the order has been updated by the customer (change in qty, change in address, etc.) w/ the next drop. So my question is if the order is purely a duplicate with no changes how can I skip over those orders and if a record has been changed how can I update that record?
Ruby on Rails 5.1.4 - Ruby 2.4.1
Thank you!
The code below is from my model:
class Geek < ApplicationRecord
require 'csv'
def self.download_walmart_orders(out)
out ||= "#{Rails.root}/test_orders.csv"
CSV.foreach(out, :headers => true,
:converters => :all,
:header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
) do |row|
geek = Geek.where(customer_order_id: row.to_h["customer_order_id"],
customer_name: row.to_h["customer_name"],
item_sku: row.to_h["item_sku"],
quantity_to_ship: row.to_h["quantity_to_ship"],
total_items_price: row.to_h["total_items_price"]).first_or_create
puts geek
end
end
end
I am assuming that customer_order_id is unique.
You could try something like this -
def self.update_or_create(attributes)
assign_or_new(attributes).save
end
Geek.where(customer_order_id: row.to_h["customer_order_id"]).update_or_create.(
customer_name: row.to_h["customer_name"],
item_sku: row.to_h["item_sku"],
quantity_to_ship: row.to_h["quantity_to_ship"],
total_items_price: row.to_h["total_items_price"])
^^^ Thank you, Michael, for the direction above. I ended up using this code and it worked perfectly. (For a slightly different project but exact same use case) my finalized model code is below:
class Wheel < ApplicationRecord
require 'csv'
def self.update_or_create(attributes)
obj = first || new
obj.assign_attributes(attributes)
obj.save!
end
def self.import(out)
out ||= "#{Rails.root}/public/300-RRW Daily Inv Report.csv"
CSV.foreach(out, :headers => true,
:converters => :all,
:header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
) do |row|
Wheel.where(item: row.to_h["item"]).update_or_create(
item_desc: row.to_h["item_desc"],
total_quantity: row.to_h["total_quantity"])
end
end
end
I'm trying to insert 200k records into three different tables. When I parse the CSV file and try to insert these records, Ruby locks up my entire system. Here is my code:
def upload_country
#upload = Upload.new(:upload => params[:upload])
if #upload.save
csv = CSV.parse(csv_text, :headers => true)
csv.each_with_index do |row, index|
unless row["name"].blank? or row["country_code"].blank? or row["destination"].blank? or row["code"].blank?
#country = Country.new(:name => row["name"].gsub(/\s+/, " ").strip, :country_code => row["country_code"].gsub(/\s+/, " ").strip, :user_id => current_user.id, :subscriber_id => get_subscriber_id)
#country.save
if row["country_code"] == "1"
p = #country.country_code.to_s+#destination.name+row["code"].gsub(/\s+/, " ").strip
else
p = #country.country_code.to_s+row["code"].gsub(/\s+/, " ").strip
end
#code = DestinationCode.create(:code => p, :country_destination_id => 1, :user_id => current_user.id)
end
end
#upload.destroy
#countries = Country.find_all_by_subscriber_id(get_subscriber_id)
render :partial => '/mycarriers/carrier_country', :layout => false
end
If you have long-running requests, it means that no other users can access your application at that time if you have only one rails instance running.
I would recommend you use delayed_job gem to do long processing task in the background. On the controller side, you should enqueue the job and response 202 (Accepted) to the browser. On the client side, you should periodically send the request to the server whether the job is finished or not. Then, update the ui accordingly.
Take slideshare.net as an example. When the user finishes upload, slideshare redirect to the new page, and periodically update ui while it converts the presentation file.
An alternative solution is that you can run rake script in the background. Check out this episode from railscasts.
why dont use use Mass Insert here a gem that can help you in the quest I personally used it in my application to insert 500K records with resque for the background processing
Hope it help you
I'm trying to create a pdf report using prawn in a rails application. There are lots of sections that contain user generated content that I want to try and group together. Sometimes this will go over more that one page which results in a cannot group error. I then tried to use a transaction so that in the event of an error I can rollback and then output the content without using the group method.
The problem is the rollback stuffs up the pages. It removes the extra page from the pdf but still has the wrong page count and outputs over lapping content when I try to redo it. I reset the y position after the rollback, as per the prawn documentation but I still get the problems.
eg. The following test code writes 2 pages of numbers, does a rollback to the start and then tries to write the same numbers again. It results in a single page pdf with the second page of numbers overlapping the first and a page count of 2. The page counts at the bottom of the page also overlap one another even though I'm using the prawn number_pages method
class TestReport < Prawn::Document
def to_pdf
font('Helvetica')
bounding_box([bounds.left, bounds.top - 50], :width => bounds.width, :height => bounds.height - 100) do
text 'begin'
y_pos = y
transaction do
begin
group do
64.times do|i|
text i.to_s
end
end
rescue
rollback
end
end
self.y = y_pos
64.times do|i|
text i.to_s
end
text 'end'
text page_number.to_s
end
page_numbers(1)
#render
end
def page_numbers(start)
string = "page <page> of <total>"
options = { :at => [bounds.right - 150, 40],
:width => 150,
:align => :right,
:start_count_at => start,
:color => "000000" }
number_pages string, options
end
end
def test_report
pdf = TestReport.new()
pdf.to_pdf
send_data pdf.render, filename: "test.pdf",
type: "application/pdf",
disposition: "inline"
end
The problems seem to be with transaction rollbacks. The main thing I want is to be able to use the group method. Is there another way?
Is my code wrong? Am I missing something or do transaction not currently work.
I'm currently using the master prawn branch in a ruby on rails application ( gem 'prawn', :git =>
'git://github.com/prawnpdf/prawn.git', :branch => 'master').
This question is quite old now, but I'll post an answer since it is one of the first hits on Google when searching for the exception.
Transactions still doesnt work with page breaks (v 1.0.0.rc2), so I created a helper method that tries to use grouping first and then if the exception occurs it just retries without grouping, making the content span more than one page.
def group_if_possible(pdf, &block)
begin
pdf.group { block.call }
rescue Prawn::Errors::CannotGroup
block.call
end
end
Example: Using it when creating a table:
group_if_possible(pdf) do
pdf.table(rows)
end
EDIT:
Grouping were removed from Prawn 1.x but there is an unofficial grouping gem that works well for Prawn 2:
https://github.com/ddengler/prawn-grouping
Looks like Brad Ediger answered your question on Google Groups, but for the benefit of anyone else looking for help with this, here's his response:
Sadly, transactions do not yet work correctly when they start new
pages or change the pages collection. It's a known issue:
https://github.com/prawnpdf/prawn/issues/268
-be
I get a CSV:MalFormedCSVError when I try to import a file using the following code:
def import_csv(filename, model)
CSV.foreach(filename, :headers => true) do |row|
item = {}
row.to_hash.each_pair do |k,v|
item.merge!({k.downcase => v})
end
model.create!(item)
end
end
The csv files are HUGE, so is there a way I can just log the bad formatted lines and CONTINUE EXECUTION with the remainder of the csv file?
You could try handling the file reading yourself and let CSV work on one line at a time. Something like this:
File.foreach(filename) do |line|
begin
CSV.parse(line) do |row|
# Do something with row...
end
rescue CSV::MalformedCSVError => e
# complain about line
end
end
You'd have to do something with the header line yourself of course. Also, this won't work if you have embedded newlines in your CSV.
One problem with using File to manually go through each line in the file is that CSV files can contain fields with \n (newline character) in them. File will take that to indicate a newline and you will end up trying to parse a partial row.
Here is an another approach that might work for you:
#csv = CSV.new('path/to/file.csv')
loop do
begin
row = #csv.shift
break unless row
# do stuff
rescue CSV::MalformedCSVError => error
# handle the error
next
end
end
The main downside that I see with this approach is that you don't have access to the CSV row string when handling the error, just the CSV::MalformedCSVError itself.
I currently have a controller that will handle a call to export a table into a CSV file using the FasterCSV gem. The problem is the information stored in the database isn't clear sometimes and so I want to change the output for a particular column.
My project.status column for instance has numbers instead of statuses ie 1 in the database corresponds to Active, 2 for Inactive and 0 for Not Yet decided. When I export the table it shows 0,1,2 instead of Active, Inactive or Not Yet decided. Any idea how to implement this?
I tried a simple loop that would check the final generated CSV file and change each 0,1,2 to its corresponding output, but the problem is every other column that had a 0,1,2 would change as well. I'm not sure how to isolate the column.
Thanks in advance
def csv
qt = params[:selection]
#lists = Project.find(:all, :order=> (params[:sort] + ' ' + params[:direction]), :conditions => ["name LIKE ? OR description LIKE ?", "%#{qt}%", "%#{qt}%"])
csv_string = FasterCSV.generate(:encoding => 'u') do |csv|
csv << ["Status","Name","Summary","Description","Creator","Comment","Contact Information","Created Date","Updated Date"]
#lists.each do |project|
csv << [project.status, project.name, project.summary, project.description, project.creator, project.statusreason, project.contactinfo, project.created_at, project.updated_at]
end
end
filename = Time.now.strftime("%Y%m%d") + ".csv"
send_data(csv_string,
:type => 'text/csv; charset=UTF-8; header=present',
:filename => filename)
end
This is actually fairly easy. In your controller code:
#app/controllers/projects_controller.rb#csv
#lists.each do |project|
csv << [project.descriptive_status, project.name, project.summary, project.description, project.creator, project.statusreason, project.contactinfo, project.created_at, project.updated_at]
end
Then in your model code. You probably already have a method that decodes the DB status to a more descriptive one though:
#app/models/project.rb
ACTIVE_STATUS = 0
INACTIVE_STATUS = 1
NOT_YET_DECIDED_STATUS = 2
def descriptive_status
case status
when ACTIVE_STATUS
"Active"
when INACTIVE_STATUS
"Inactive"
when NOT_YET_DECIDED_STATUS
"Not Yet Decided"
end
end
There are probably a number of ways you can then refactor this. In the controller at least, it would probably be best to make that finder a more descriptive named scope. The constants in the model could be brought into SettingsLogic configuration or another similar gem.