I'm trying to use Ruby's csv module to import the records contained in a csv file to my local table in a Ruby on Rails 3 application.
The table was created through the creation of model Movie.
Here is what I've been executing in console:
require 'csv'
CSV.foreach('public/uploads/VideoTitles2.csv') do |row|
record = Movie.new(
:media_format => row[0],
:title => row[1],
:copies_at_home => row[2],
:order => row[3]
)
record.save
end
The rows of the csv file match (in data type) the columns they're being passed into. Here is a shortened version of the csv file (VideoTitles2.csv) I'm attempting to import:
"DVD","LEAP OF FAITH",1,1
"DVD","COCOON",1,2
"DVD","TITANIC",1,3
where each record is separated by \n I believe. This csv file was exported from Access and its original file extension was .txt. I've manually changed it to .csv for sake of the import.
The problem is that, after executing the above lines in rails console, I get the following output:
=> nil
The import doesn't seem to happen. If anyone has an idea as to how I could remedy this I'd really appreciate it.
I don't see the problem. This code snippet returns nil because CSV.foreach returns nil, but this is no indication if the loop is run or not. Did you checked if any Movie was created? did you include any debug lines to follow the process?
You may want to check the output of record.save (or call record.save!), maybe validations errors are preventing the record from being created. Also, if you want the loop to return the created records, you can write this (Ruby >= 1.8.7):
require 'csv'
records = CSV.foreach('public/uploads/VideoTitles2.csv').map do |media_format, title, copies_at_home, order|
Movie.create!({
media_format: media_format,
title: title,
copies_at_home: copies_at_home,
order: order,
})
end
Okay there were two things I had wrong:
The exported csv file should not have quotations around the strings - I just removed them.
Thanks to tokland, the record.save! was necessary (as opposed to the record.save I was doing) - validation errors were preventing the records from being created.
So to conclude, one could just create the following function after creating the model/table Movie:
class Movie < ActiveRecord::Base
attr_accessible :media_format, :title, :copies_at_home, :order
require 'csv'
def self.import_movies()
CSV.foreach('public/uploads/movies.csv') do |row|
record = Movie.new(
:media_format => row[0],
:title => row[1],
:copies_at_home => row[2],
:order => row[3]
)
record.save!
end
end
end
Where movies.csv looks like the following:
Blu-ray, Movie 1, 1, 1
DVD, Movie 2, 1, 2
Blu-ray, Movie 3, 1, 3
then call this function in console as such:
Movie.import_movies()
and, as expected, all that would be returned in the console would be:
=> nil
Check your index view (if you've created one) and you should see that the records were successfully imported into the movies table.
Related
I'm currently using SmarterCSV to do bulk CSV import via MongoDB's upsert commands. I have the following code excerpt:
SmarterCSV.process(csv, csv_options) do |chunk|
chunk.each do |row|
#creates a temporary user to store the object
user = User.new
#converts row info to populate user object
#creates an array of commands that can be executed by MongoDB via user.as_document
updates << {:q => {:email => user.email},
:u => {:$set => user.as_document},
:multi => false,
:upsert => true}
user = nil
end
end
However, I'm noticing that the memory usage keeps growing as the Garbage Collection (using Rails 3.2.14 & Ruby 2.0.0p353) doesn't seem to clear the temporary user objects fast enough.
So I tried to create user = User.new outside of the SmarterCSV process (see below) and reuse the user object within the process. This saves memory. However, user.as_document would overwrite previous elements in the updates array on each iteration. I was able to solve the problem by using user.as_document.to_json, but that doesn't set any of User's relationship correctly. For example, instead of saving a BSON reference for an relation's id, it only saves the id in string format.
Any ideas? Is there a way that I can optimize the bulk import process?
user = User.new
SmarterCSV.process(csv, csv_options) do |chunk|
chunk.each do |row|
#creates a temporary user to store the object
#converts row info to populate & reuse user object
#creates an array of commands that can be executed by MongoDB via user.as_document.to_json
updates << {:q => {:email => user.email},
:u => {:$set => user.as_document.to_json},
:multi => false,
:upsert => true}
end
end
I ended fixing this by using 'user.as_document.deep_dup'
I am learning Ruby on Rails and in the process I am creating a sample app that takes data from a csv file (information involving Chicago homicide victims), saves the data into the database, and displays this information to the user.
I am grabbing dates from each row of the .csv file, and I am encountering an issue where seemingly random dates will appear as 'nil' in the database but others will get imported properly. I am also grabbing Name and Age attributes, and these are getting imported correctly and without issue. It is only the dates I am having trouble with
Here is the model which I am using:
class CreateVictims < ActiveRecord::Migration
def change
create_table :victims do |t|
t.string :name
t.integer :age
t.date :date
t.timestamps
end
end
end
And here is the rake task I am using to import the data:
require 'csv'
namespace :import_victims_csv do
task :create_victims => :environment do
csv_text = File.read('public/seed_data/2013all.csv')
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
Victim.create!(date: row[1], age: row[5], name: row[8])
end
end
end
I'd say about 50% of the dates will be imported correctly, but the other 50% will just result in 'nil'. The source .csv file has the dates listed for all of the names.
For reference, here is a link to the csv file I am importing from: CSV
Looking at your dataset, it seems that the dates are in the format mm/dd/yyyy, according to this question/answer parsing of that format is no longer supported. So perhaps you could try reformatting the date prior to creating the model:
csv.each do |row|
formatted_date = Date.strptime row[1], '%m/%d/%Y'
Victim.create!(date: formatted_date, age: row[5], name: row[8])
end
I have a structured comma delimited file that has two record types. The different records are differentiated by a header entry: H or P. The file format follows:
"H","USA","MD","20904"
"P","1","A","Female","W"
"P","2","A","Male","H"
I'd like to import the file and then create activerecord models with the imported data. The approach that I am using is to create a field map that includes the number of fields, object name and columns.
I then utilize the field map
$field_map =
{
'H' =>
{
:count => 4,
:object => :Header,
:cols => [:record_type, :country_id, :state, :zip]
},
'R' =>
{
:count => 4,
:object => :RaceData,
:cols => [:record_type, :household_size, :gender, :race]
}
}
I then use FastCSV to import the file and use a case statement to how the file will be transformed and then used in activerecord create statements.
FasterCSV.foreach(filename) do |row|
tbl_type = row[0]
tbl_info = $field_map[tbl_type]
unless (tbl_info.nil?)
field_no = tbl_info[:count]
object = tbl_info[:object]
columns = tbl_info[:cols]
record_type = new_record[:record_type]
case record_type
when "H"
factory_build_h_record(new_record)
when "P"
factory_build_p_record(new_record)
end
end
end
The code above is summarized due to space constraints. My approach works just fine, but my I'm new to ruby and I'm always interested in best practices and the "true" Ruby-way of doing things. I'd be interested in hearing how more experienced programmers would tackle this problem. Thanks for your input.
I suggest the gem 'roo'
You have an example source code here but I rather watch the 10 min video
I am trying to import data into rails (3.1) and I have created this rake task to parse a CSV file (generated by Excel on Mac)
desc "Import users."
task :import_users => :environment do
File.open("users.csv", "r").each do |line|
id, name, age, email = line.strip.split(',')
u = User.new(:id => id, :name => name, :age => age, :email => email)
u.save
end
end
However when I run the rake task, only the first line of the CSV file gets imported. It does not iterate over every line in the file besides the first one. Can anyone tell me why?
Not sure, but I think what is happening here is the each is representing each file rather than each line. And as there's only one file, this may not work as expected. I'd try a CSV parser instead:
CSV.foreach("users.csv") do |line|
id, name, age, email = line
u = User.new(:id => id, :name => name, :age => age, :email => email)
u.save
end
When parsing any kind of text file using ruby, be sure to check encoding and/or line endings to make sure it's a format that Ruby likes.
In this case, Ruby disliked the Mac OS X line ending format, but liked the Unix one.
I am trying to import data into rails (3.1) and I have created this rake task to parse a tab delimited text file (generated by Excel on Mac), the file has standard Mac OS X line endings.
desc "Import users."
task :import_users => :environment do
File.open("users.txt", "r", '\r').each do |line|
id, name, age, email = line.strip.split('/t')
u = User.new(:id => id, :name => name, :age => age, :email => email)
u.save
end
end
However, when I try to run this rake task I get the following error:
rake aborted!
can't convert String into Integer
My guess is that Ruby doesn't like converting the Age heading into a numerical age variable in my User class. Is there a way I can either
(a) skip the header line in the file OR
(b) do this cast on the fly in Ruby?
Note: This is one of many attempts to read in some data to Ruby. Whenever I tried to read in the data before, I never seemed to get this error. The string value always got casted to 0.0.
Simpliest solution, which come to mind was:
u = User.new(:id => id, :name => name, :age => Integer(age), :email => email)
Of course, you will still have an error on a first line of a file, if you got your headers there.