Ruby 2.2, Ruby on Rails 4.2
I'm genarating some CSV data in Ruby on Rails, and want empty fields to be empty, like ,, not like ,"", .
I wrote codes like below:
somethings_cotroller.rb
def get_data
respond_to do |format|
format.html
format.csv do
#data = SheetRepository.accounts_data
send_data render_to_string, type: :csv
end
end
end
somethings/get_data.csv.ruby
require 'csv'
csv_str = CSV.generate do |csv|
csv << [1,260,37335,'','','','','','']
...
end
And this generates CSV file like this.
get_data.csv
1,260,37335,"","","","","",""
I want CSV data like below.
1,260,37335,,,,,,
It seems like Ruby adds "" automatically.
How can I do this??
In order to get CSV to output an empty column, you need to tell it that nothing is in the column. An empty string, in ruby, is still something, you'll need to replace those empty strings with nil in order to get the output you want:
csv_str = CSV.generate do |csv|
csv << [1,260,37335,'','','','','',''].map do |col|
col.respond_to?(:empty?) && col.empty? ? nil : col
end
end
# => 1,260,37335,,,,,,
In rails you can clean that up by making use of presence, though this will blank out false as well:
csv_str = CSV.generate do |csv|
csv << [1,260,37335,'',false, nil,'','',''].map(&:presence)
end
# => 1,260,37335,,,,,,
The CSV documentation shows an option that you can use for this case. There are not examples but you can guess what it does.
The only consideration is, you need to send an array of Strings, otherwise, you will get a NoMethodError
csv_str = CSV.generate(write_empty_value: nil) do |csv|
csv << [1,260,37335,'','','','','','', false, ' ', nil].map(&:to_s)
end
=> "1,260,37335,,,,,,,false, ,\n"
The benefit of this solution is, you preserve the false.
I resolved by myself!
in somethings_controller.rb
send_data render_to_string.gsub("\"\"",""), type: :csv
I have a million of records and I want to export that data into CSV. I used find_each method to fetch the records. But it also taking too much time to fetch data and download CSV. I am not able to do other activity in the application because its taking more memory. Its just showing me loading the page in the browser.
I have written the following code in the controller
def export_csv
require 'csv'
lines = []
csv_vals = []
User.where(status:ACTIVE).order('created_atdesc').find_each(batch_size: 10000) do |user|
csv_vals << user.email if user.email.present?
csv_vals << user.name if user.name.present?
.......
........
.......etc
lines << CSV.generate_line(csv_vals)
end
send_data(line, type: 'text/csv; charset=iso-8859-1; header=present', \
disposition: "attachment; filename=file123.csv"
end
Is there another way to load the millions of records and download quickly?
this may help:
genereating and streaming potentially large csv files using ruby on rail
There is CSV-export of some objects (such as tasks, contacts, etc) in my application. It just renders CSV-file like this:
respond_to do |format|
format.html
format.csv { render text: Task.to_csv } # I have self.to_csv def in model
end
It generates a CSV file when I go to '/tasks.csv' without a problem.
Now I want to export all the objects and zip them. I'm using rubyzip gem to create zip-files. Now my code for creating zip-file with all the CSVs looks like that:
Zip::ZipFile.open("#{path_to_file}.zip", Zip::ZipFile::CREATE) do |zipfile|
zipfile.file.open("tasks.csv", "w") { |f| f << open("http://#{request.host}:#{request.port.to_s}/tasks.csv").read }
# the same lines for contacts and other objects
end
But it seems that there is something wrong with it because it's executing for a long time (I'm getting Timeout::Error even if there is just one line in CSV) and the resulting zip-archive contains something broken.
How can I save my "/tasks.csv", "/contacts.csv", etc as a file on server (inside of zip-archive in this case)?
I did it! The code is:
Zip::ZipFile.open("#{path_to_file}.zip", Zip::ZipFile::CREATE) do |zipfile|
zipfile.file.open("tasks.csv", "w") do |f|
CSV.open(f, "w") do |csv|
CSV.parse(Task.to_csv) { |row| csv << row }
end
end
end
I'm attempting to DRY up a method I've been using for a few months:
def export(imagery_requests)
csv_string = FasterCSV.generate do |csv|
imagery_requests.each do |ir|
csv << [ir.id, ir.service_name, ir.description, ir.first_name, ir.last_name, ir.email,
ir.phone_contact, ir.region, ir.imagery_type, ir.file_type, ir.pixel_type,
ir.total_images, ir.tile_size, ir.progress, ir.expected_date, ir.high_priority,
ir.priority_justification, ir.raw_data_location, ir.service_overviews,
ir.is_def, ir.isc_def, ir.special_instructions, ir.navigational_path,
ir.fyqueue, ir.created_at, ir.updated_at]
end
end
# send it to the browser with proper headers
send_data csv_string,
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=requests_as_of-#{Time.now.strftime("%Y%m%d")}.csv"
end
I figured it would be a LOT better if instead of specifying EVERY column manually, I did something like this:
def export(imagery_requests)
csv_string = FasterCSV.generate do |csv|
line = []
imagery_requests.each do |ir|
csv << ir.attributes.values.each do |i|
line << i
end
end
end
# send it to the browser with proper headers
send_data csv_string,
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=requests_as_of-#{Time.now.strftime("%Y%m%d")}.csv"
end
That should be creating an array of arrays. It works just fine in the Rails console. But in the production environment, it just produces garbage output. I'd much rather make this method extensible so I can add more fields to the ImageryRequest model at a later time. Am I going about this all wrong?
I'm guessing that it probably works in the console when you do it for just one imagery_request, yes?
But when you do multiple it fails?
Again I'm guessing that's because you never reset line to be an empty array again. So you're continually filling a single array.
Try the simple way first, to check it works, then start going all << on it then:
csv_string = FasterCSV.generate do |csv|
imagery_requests.each do |ir|
csv << ir.attributes.values.clone
end
end
PS - in the past I've even used clone on my line-by-line array, just to be sure I wasn't doing anything untoward with persisted stuff...
I'm developing a "script generator" to automatize some processes at work.
It has a Rails application running on a server that stores all data needed to make the script and generates the script itself at the end of the process.
The problem I am having is how to export the data from the ActiveRecord format to Plain Old Ruby Objects (POROs) so I can deal with them in my script with no database support and a pure-ruby implementation.
I thought about YAML, CSV or something like this to export the data but it would be a painful process to update these structures if the process changes. Is there a simpler way?
Ty!
By "update these structures if the process changes", do you mean changing the code that reads and writes the CSV or YAML data when the fields in the database change?
The following code writes and reads any AR object to/from CSV (requires the FasterCSV gem):
def load_from_csv(csv_filename, poro_class)
headers_read = []
first_record = true
num_headers = 0
transaction do
FCSV.foreach(csv_filename) do |row|
if first_record
headers_read = row
num_headers = headers_read.length
first_record = false
else
hash_values = {}
for col_index in 0...num_headers
hash_values[headers_read[col_index]] = row[col_index]
end
new_poro_obj = poro_class.new(hash_values) # assumes that your PORO has a constructor that accepts a hash. If not, you can do something like new_poro_obj.send(headers_read[col_index], row[col_index]) in the loop above
#work with your new_poro_obj
end
end
end
end
#objects is a list of ActiveRecord objects of the same class
def dump_to_csv(csv_filename, objects)
FCSV.open(csv_filename,'w') do |csv|
#get column names and write them as headers
col_names = objects[0].class.column_names()
csv << col_names
objects.each do |obj|
col_values = []
col_names.each do |col_name|
col_values.push obj[col_name]
end
csv << col_values
end
end
end