This Rails app (using a postgresql db) imports data from a CSV to create new records of a model "Client." The import in Models/client.rb looks like this:
def self.import(file, industry)
CSV.foreach(file.path, headers: true, encoding:'iso-8859-1:utf-8') do |row|
industry.clients.create! row.to_hash
end
This works properly in creating the records and populating each record's attributes, per the CSV data, for all record types except for an Array.
Clients have an array type attribute of "emails" (among other array attributes).
The array attributes were created in a migration like this:
add_column :clients, :emails, :text, array: true, default: []
In the CSV, they are stored in cells like this:
["email1#domain.com", "email2#domain.com", "email3#domain.com"]
Upon uploading, these emails would show on my local server:
INSERT INTO "clients"... ["emails", "{\"email1#domain.com",\"email2#domain.com\"}"]
As you can see, it chops off the third element of the array "email3#domain.com", and this is true for the last element of all Arrays uploaded from the CSV.
My guess is that the Postgresql array type is having trouble with the format that the array is saved in the CSV (the - ["element1", "element2", ...] )- I have tried several different formats, but no success yet. Any thoughts on how to do this?
Instead of trying to upload these attributes as an array I changed the migration to a normal string.
add_column :clients, :emails, :string
After I upload the CSV data to the rails app with:
def self.import(file, industry)
CSV.foreach(file.path, headers: true, encoding:'iso-8859-1:utf-8') do |row|
industry.clients.create! row.to_hash
end
I am now just taking that string and using this to manipulate the data:
JSON.parse(#client.emails)
Because the data uploaded from the CSV cell is already in a format that works with the JSON.parse: ["element1", "element2", "element3",... ] this was an effective method.
*NOTE This does not achieve the exact result desired in my posted question, but is functionally serving the same purpose for what is needed in this rails app.
Related
This question is asked many times on SO. The main problem is nothing got fits into my situation.
Case is, I am not able to store typed content as array in database column.
text_field whose code is:
= text_field_tag 'product[keywords][]', #product.keywords, class: 'tab-input
product_keywords'
In controller strong parameters are:
params.require(:product).permit(:id, :name, :keywords => [])
Jquery code that is not not removing value upon deletion when typed wrong value but it add commas after each element as I want to take commas seperated value in one column.
$(document).on 'keyup', '.product_keywords', ->
keyword = #value.replace(/(\w)[\s,]+(\w?)/g, '$1, $2')
if keyword != #value
#value = keyword
return
model code:
serialize :keywords, Array
migration code:
class AddKeywordsToProducts < ActiveRecord::Migration[5.1]
def change
add_column :products, :keywords, :text
end
end
So, if someone writes, abc and hit space a comma is added in the end. after three typed words it will look like:
abc, dbx, she
now I want to store it as array in column but its not storing properly.
it stores as:
["abc, dbx, she"]
Also please can anybody tell me the best cases to handle these cases?
Plus best practices to deal with such cases using ruby so I will learn it for future?
You probably want a custom serializer as shown here. So instead of:
serialize :keywords, Array
You might do somewhat like:
serialize :keywords, KeywordSerializer
And somewhere in helpers:
class KeywordSerializer
def self.dump(what)
what.join(", ")
end
def self.load(what)
what.split(/\s*,\s*/)
end
end
Passing array elements using single form tag is not possible to pass as a array and passing array as a string, you need to process it near white-listing your params,
permitted_params = params.require(:product).permit(:id, :name, :keywords => [])
permitted_params[:keywords] = permitted_params[:keywords][0].split(/\s*,\s*/)
I have setup elastic search rails with the 'elasticsearch-model',
'elasticsearch-rails' gems.
I am trying to search for these attachments by their content. It works well when I have an indexed PDF, Word, plain text, or a plefora of other formats. But it does not work when I index a mac format, e.g. .pages, .keynote and .numbers files.
I made sure that mac files get indexed, but it feels like they are not indexed properly. When I look at raw index data for a .word file vs .pages file, they both have their respective attachment fields populated as base64 representation of the document content. It seems like for the mac extensions, however, this base64 representation isn't accurate.
My index model definition:
settings index: { number_of_shards: 3 } do
mappings do
indexes :filename
indexes :uploaded_by
indexes :project_id, index: :not_analyzed
indexes :attachment, type: 'attachment'
end
end
def as_indexed_json(options={})
self.as_json({
only: [:project_id, :filename, :uploaded_by, :attachment],
methods: [:attachment]
})
end
My attachment method:
def attachment
if url
key = url.sub("https://s3.amazonaws.com/#{ENV['BUCKETNAME']}/", "")
content = AWS.s3.buckets[ENV['BUCKETNAME']].objects[key].read
Base64.encode64(content)
end
end
The file first gets uploaded to s3 (since client side sends it there directly), then read by the server from s3 to get indexed. this is a proof of concept code only, future dev will upload from client to server, index, then upload to s3, then delete from server.
E.S version: "1.7.1",
Lucene version: "4.10.4"
Rails 4, Mongoid instead of ActiveRecord (but this should change anything for the sake of the question).
Let's say I have a MyModel domain class with some validation rules:
class MyModel
include Mongoid::Document
field :text, type: String
field :type, type: String
belongs_to :parent
validates :text, presence: true
validates :type, inclusion: %w(A B C)
validates_uniqueness_of :text, scope: :parent # important validation rule for the purpose of the question
end
where Parent is another domain class:
class Parent
include Mongoid::Document
field :name, type: String
has_many my_models
end
Also I have the related tables in the database populated with some valid data.
Now, I want to import some data from an CSV file, which can conflict with the existing data in the database. The easy thing to do is to create an instance of MyModel for every row in the CSV and verify if it's valid, then save it to the database (or discard it).
Something like this:
csv_rows.each |data| # simplified
my_model = MyModel.new(data) # data is the hash with the values taken from the CSV row
if my_model.valid?
my_model.save validate: false
else
# do something useful, but not interesting for the question's purpose
# just know that I need to separate validation from saving
end
end
Now, this works pretty smoothly for a limited amount of data. But when the CSV contains hundreds of thousands of rows, this gets quite slow, because (worst case) there's a write operation for every row.
What I'd like to do, is to store the list of valid items and save them all at the end of the file parsing process. So, nothing complicated:
valids = []
csv_rows.each |data|
my_model = MyModel.new(data)
if my_model.valid? # THE INTERESTING LINE this "if" checks only against the database, what happens if it conflicts with some other my_models not saved yet?
valids << my_model
else
# ...
end
end
if valids.size > 0
# bulk insert of all data
end
That would be perfect, if I could be sure that the data in the CSV does not contain duplicated rows or data that goes against the validation rules of MyModel.
My question is: how can I check each row against the database AND the valids array, without having to repeat the validation rules defined into MyModel (avoiding to have them duplicated)?
Is there a different (more efficient) approach I'm not considering?
What you can do is validate as model, save the attributes in a hash, pushed to the valids array, then do a bulk insert of the values usint mongodb's insert:
valids = []
csv_rows.each |data|
my_model = MyModel.new(data)
if my_model.valid?
valids << my_model.attributes
end
end
MyModel.collection.insert(valids, continue_on_error: true)
This won't however prevent NEW duplicates... for that you could do something like the following, using a hash and compound key:
valids = {}
csv_rows.each |data|
my_model = MyModel.new(data)
if my_model.valid?
valids["#{my_model.text}_#{my_model.parent}"] = my_model.as_document
end
end
Then either of the following will work, DB Agnostic:
MyModel.create(valids.values)
Or MongoDB'ish:
MyModel.collection.insert(valids.values, continue_on_error: true)
OR EVEN BETTER
Ensure you have a uniq index on the collection:
class MyModel
...
index({ text: 1, parent: 1 }, { unique: true, dropDups: true })
...
end
Then Just do the following:
MyModel.collection.insert(csv_rows, continue_on_error: true)
http://api.mongodb.org/ruby/current/Mongo/Collection.html#insert-instance_method
http://mongoid.org/en/mongoid/docs/indexing.html
TIP: I recommend if you anticipate thousands of rows to do this in batches of 500 or so.
I have a system for updating InventoryItem records using a CSV upload.
I have this controller method:
def import
InventoryItem.import(params[:file], params[:store_id])
redirect_to vendors_dashboard_path, notice: "Inventory Imported."
end
Which of course calls this model method:
def self.import(file, store_id)
CSV.foreach(file.path, headers: true) do |row|
inventory_item = InventoryItem.find_or_initialize_by_code_and_store_id(row[0], store_id)
inventory_item.update_attributes(:price => row.to_hash.slice(:price))
end
end
I want to update only the :price attribute in the update because and :code and :store_id won't change. Currently the records being imported have price all as 0.0 (big decimal). Not nil, or the correct value, but 0.0, so clearly I'm doing something wrong to make this work. I know when I do this in the console it looks something like this:
inventory_item = InventoryItem.find_by_id(1)
inventory_item.update_attributes(:price => 29.99)
Any ideas on why I'm not updating the price attribute correctly?
Trying this, it doesn't seem like csv returns symbolized hash keys
and slice doesn't seem to work there. how about this inside your
CSV.foreach loop:
inventory_item.update_attributes(:price => row.to_hash["price"])
I'm trying to use Ruby's csv module to import the records contained in a csv file to my local table in a Ruby on Rails 3 application.
The table was created through the creation of model Movie.
Here is what I've been executing in console:
require 'csv'
CSV.foreach('public/uploads/VideoTitles2.csv') do |row|
record = Movie.new(
:media_format => row[0],
:title => row[1],
:copies_at_home => row[2],
:order => row[3]
)
record.save
end
The rows of the csv file match (in data type) the columns they're being passed into. Here is a shortened version of the csv file (VideoTitles2.csv) I'm attempting to import:
"DVD","LEAP OF FAITH",1,1
"DVD","COCOON",1,2
"DVD","TITANIC",1,3
where each record is separated by \n I believe. This csv file was exported from Access and its original file extension was .txt. I've manually changed it to .csv for sake of the import.
The problem is that, after executing the above lines in rails console, I get the following output:
=> nil
The import doesn't seem to happen. If anyone has an idea as to how I could remedy this I'd really appreciate it.
I don't see the problem. This code snippet returns nil because CSV.foreach returns nil, but this is no indication if the loop is run or not. Did you checked if any Movie was created? did you include any debug lines to follow the process?
You may want to check the output of record.save (or call record.save!), maybe validations errors are preventing the record from being created. Also, if you want the loop to return the created records, you can write this (Ruby >= 1.8.7):
require 'csv'
records = CSV.foreach('public/uploads/VideoTitles2.csv').map do |media_format, title, copies_at_home, order|
Movie.create!({
media_format: media_format,
title: title,
copies_at_home: copies_at_home,
order: order,
})
end
Okay there were two things I had wrong:
The exported csv file should not have quotations around the strings - I just removed them.
Thanks to tokland, the record.save! was necessary (as opposed to the record.save I was doing) - validation errors were preventing the records from being created.
So to conclude, one could just create the following function after creating the model/table Movie:
class Movie < ActiveRecord::Base
attr_accessible :media_format, :title, :copies_at_home, :order
require 'csv'
def self.import_movies()
CSV.foreach('public/uploads/movies.csv') do |row|
record = Movie.new(
:media_format => row[0],
:title => row[1],
:copies_at_home => row[2],
:order => row[3]
)
record.save!
end
end
end
Where movies.csv looks like the following:
Blu-ray, Movie 1, 1, 1
DVD, Movie 2, 1, 2
Blu-ray, Movie 3, 1, 3
then call this function in console as such:
Movie.import_movies()
and, as expected, all that would be returned in the console would be:
=> nil
Check your index view (if you've created one) and you should see that the records were successfully imported into the movies table.