I am learning Ruby on Rails and in the process I am creating a sample app that takes data from a csv file (information involving Chicago homicide victims), saves the data into the database, and displays this information to the user.
I am grabbing dates from each row of the .csv file, and I am encountering an issue where seemingly random dates will appear as 'nil' in the database but others will get imported properly. I am also grabbing Name and Age attributes, and these are getting imported correctly and without issue. It is only the dates I am having trouble with
Here is the model which I am using:
class CreateVictims < ActiveRecord::Migration
def change
create_table :victims do |t|
t.string :name
t.integer :age
t.date :date
t.timestamps
end
end
end
And here is the rake task I am using to import the data:
require 'csv'
namespace :import_victims_csv do
task :create_victims => :environment do
csv_text = File.read('public/seed_data/2013all.csv')
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
Victim.create!(date: row[1], age: row[5], name: row[8])
end
end
end
I'd say about 50% of the dates will be imported correctly, but the other 50% will just result in 'nil'. The source .csv file has the dates listed for all of the names.
For reference, here is a link to the csv file I am importing from: CSV
Looking at your dataset, it seems that the dates are in the format mm/dd/yyyy, according to this question/answer parsing of that format is no longer supported. So perhaps you could try reformatting the date prior to creating the model:
csv.each do |row|
formatted_date = Date.strptime row[1], '%m/%d/%Y'
Victim.create!(date: formatted_date, age: row[5], name: row[8])
end
Related
This Rails app (using a postgresql db) imports data from a CSV to create new records of a model "Client." The import in Models/client.rb looks like this:
def self.import(file, industry)
CSV.foreach(file.path, headers: true, encoding:'iso-8859-1:utf-8') do |row|
industry.clients.create! row.to_hash
end
This works properly in creating the records and populating each record's attributes, per the CSV data, for all record types except for an Array.
Clients have an array type attribute of "emails" (among other array attributes).
The array attributes were created in a migration like this:
add_column :clients, :emails, :text, array: true, default: []
In the CSV, they are stored in cells like this:
["email1#domain.com", "email2#domain.com", "email3#domain.com"]
Upon uploading, these emails would show on my local server:
INSERT INTO "clients"... ["emails", "{\"email1#domain.com",\"email2#domain.com\"}"]
As you can see, it chops off the third element of the array "email3#domain.com", and this is true for the last element of all Arrays uploaded from the CSV.
My guess is that the Postgresql array type is having trouble with the format that the array is saved in the CSV (the - ["element1", "element2", ...] )- I have tried several different formats, but no success yet. Any thoughts on how to do this?
Instead of trying to upload these attributes as an array I changed the migration to a normal string.
add_column :clients, :emails, :string
After I upload the CSV data to the rails app with:
def self.import(file, industry)
CSV.foreach(file.path, headers: true, encoding:'iso-8859-1:utf-8') do |row|
industry.clients.create! row.to_hash
end
I am now just taking that string and using this to manipulate the data:
JSON.parse(#client.emails)
Because the data uploaded from the CSV cell is already in a format that works with the JSON.parse: ["element1", "element2", "element3",... ] this was an effective method.
*NOTE This does not achieve the exact result desired in my posted question, but is functionally serving the same purpose for what is needed in this rails app.
My problem is that i have to export an excel sheet save some rows to the database without duplication or redundancy
so i started it with importing CSV instead of XLS then when i finish i might be able to parse the xls
this is my model code:
require 'csv'
class Machine < ActiveRecord::Base
def self.assign_row(row)
a, b, c, d = row
#c = c.slice(1,4)
Machine.create(name: c, mid: #c)
end
def self.import(file)
CSV.foreach(file.path) do |row|
machine = Machine.assign_row(row)
end
end
end
Import method in machines_controller
def import
count = Machine.import params[:file]
redirect_to machines_path, notice: "file imported successfully!"
end
Migration code
def change
create_table :machines do |t|
t.string :name
t.string :mid
t.timestamps null: false
end
add_index :machines, :name, :unique => true
end
and the view code
<%= form_tag import_machines_path, multipart: true do %>
<%= file_field_tag :file %>
<%= submit_tag "upload" %>
<% end %>
routes
Rails.application.routes.draw do
resources :errors
resources :machines do
collection do
post :import
end
end
root "machines#index
end
any thoughts on how to skip duplicated records from saving into database would be appreciated
thanks
Unique Identifier:
To avoid duplicate records saving to database you should maintain a unique identifier other than primary key. This helps you to identify if the record already available in DB, if it is available you can skip that record from saving again.
I guess you can use name in this case, which should be unique for each record in database. write a uniqueness validation in model to implement this.
After changes:
validates_uniqueness_of :name
def self.assign_row(row)
a, b, c, d = row
#c = c.slice(1,4)
machine = Machine.find_by(name: c)
Machine.create(name: c, mid: #c) if machine.blank?
end
Hope it helps!!
Thank you.
I need to upload a text file without saving it in the database. My goal is to upload this file and automatically have to take your content and save it in my database.
my file: data.txt
name age occupation
julio 19 student
ramon 20 student
my database:
class CreateStudents < ActiveRecord::Migration
def change
create_table: students do |t|
t.string "name"
t.integer "age"
t.string "occupation"
t.timestamps
end
end
end
Does anyone have any idea how this can be done? I searched on the internet, but found no solution to my case. I need help.
= form_tag url, {multipart: true} do
= file_field_tag :file
....
in controller
if params[:file]
lines = params[:file].tempfile.readlines.map(&:chomp) #readlines from file & removes newline symbol
lines.shift #remove first line
lines.each do |l|
m = l.match(/(\S+)\s(\d+)\s(\S+)/) #parse line
Student.create {name: m[1],age: m[2], occupation: m[3]}
end
end
I am trying to import data into rails (3.1) and I have created this rake task to parse a CSV file (generated by Excel on Mac)
desc "Import users."
task :import_users => :environment do
File.open("users.csv", "r").each do |line|
id, name, age, email = line.strip.split(',')
u = User.new(:id => id, :name => name, :age => age, :email => email)
u.save
end
end
However when I run the rake task, only the first line of the CSV file gets imported. It does not iterate over every line in the file besides the first one. Can anyone tell me why?
Not sure, but I think what is happening here is the each is representing each file rather than each line. And as there's only one file, this may not work as expected. I'd try a CSV parser instead:
CSV.foreach("users.csv") do |line|
id, name, age, email = line
u = User.new(:id => id, :name => name, :age => age, :email => email)
u.save
end
When parsing any kind of text file using ruby, be sure to check encoding and/or line endings to make sure it's a format that Ruby likes.
In this case, Ruby disliked the Mac OS X line ending format, but liked the Unix one.
I'm trying to use Ruby's csv module to import the records contained in a csv file to my local table in a Ruby on Rails 3 application.
The table was created through the creation of model Movie.
Here is what I've been executing in console:
require 'csv'
CSV.foreach('public/uploads/VideoTitles2.csv') do |row|
record = Movie.new(
:media_format => row[0],
:title => row[1],
:copies_at_home => row[2],
:order => row[3]
)
record.save
end
The rows of the csv file match (in data type) the columns they're being passed into. Here is a shortened version of the csv file (VideoTitles2.csv) I'm attempting to import:
"DVD","LEAP OF FAITH",1,1
"DVD","COCOON",1,2
"DVD","TITANIC",1,3
where each record is separated by \n I believe. This csv file was exported from Access and its original file extension was .txt. I've manually changed it to .csv for sake of the import.
The problem is that, after executing the above lines in rails console, I get the following output:
=> nil
The import doesn't seem to happen. If anyone has an idea as to how I could remedy this I'd really appreciate it.
I don't see the problem. This code snippet returns nil because CSV.foreach returns nil, but this is no indication if the loop is run or not. Did you checked if any Movie was created? did you include any debug lines to follow the process?
You may want to check the output of record.save (or call record.save!), maybe validations errors are preventing the record from being created. Also, if you want the loop to return the created records, you can write this (Ruby >= 1.8.7):
require 'csv'
records = CSV.foreach('public/uploads/VideoTitles2.csv').map do |media_format, title, copies_at_home, order|
Movie.create!({
media_format: media_format,
title: title,
copies_at_home: copies_at_home,
order: order,
})
end
Okay there were two things I had wrong:
The exported csv file should not have quotations around the strings - I just removed them.
Thanks to tokland, the record.save! was necessary (as opposed to the record.save I was doing) - validation errors were preventing the records from being created.
So to conclude, one could just create the following function after creating the model/table Movie:
class Movie < ActiveRecord::Base
attr_accessible :media_format, :title, :copies_at_home, :order
require 'csv'
def self.import_movies()
CSV.foreach('public/uploads/movies.csv') do |row|
record = Movie.new(
:media_format => row[0],
:title => row[1],
:copies_at_home => row[2],
:order => row[3]
)
record.save!
end
end
end
Where movies.csv looks like the following:
Blu-ray, Movie 1, 1, 1
DVD, Movie 2, 1, 2
Blu-ray, Movie 3, 1, 3
then call this function in console as such:
Movie.import_movies()
and, as expected, all that would be returned in the console would be:
=> nil
Check your index view (if you've created one) and you should see that the records were successfully imported into the movies table.