Rails 3.1 active record query to an array of arrays for CSV export via FastCSV - ruby-on-rails

I'm attempting to DRY up a method I've been using for a few months:
def export(imagery_requests)
csv_string = FasterCSV.generate do |csv|
imagery_requests.each do |ir|
csv << [ir.id, ir.service_name, ir.description, ir.first_name, ir.last_name, ir.email,
ir.phone_contact, ir.region, ir.imagery_type, ir.file_type, ir.pixel_type,
ir.total_images, ir.tile_size, ir.progress, ir.expected_date, ir.high_priority,
ir.priority_justification, ir.raw_data_location, ir.service_overviews,
ir.is_def, ir.isc_def, ir.special_instructions, ir.navigational_path,
ir.fyqueue, ir.created_at, ir.updated_at]
end
end
# send it to the browser with proper headers
send_data csv_string,
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=requests_as_of-#{Time.now.strftime("%Y%m%d")}.csv"
end
I figured it would be a LOT better if instead of specifying EVERY column manually, I did something like this:
def export(imagery_requests)
csv_string = FasterCSV.generate do |csv|
line = []
imagery_requests.each do |ir|
csv << ir.attributes.values.each do |i|
line << i
end
end
end
# send it to the browser with proper headers
send_data csv_string,
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=requests_as_of-#{Time.now.strftime("%Y%m%d")}.csv"
end
That should be creating an array of arrays. It works just fine in the Rails console. But in the production environment, it just produces garbage output. I'd much rather make this method extensible so I can add more fields to the ImageryRequest model at a later time. Am I going about this all wrong?

I'm guessing that it probably works in the console when you do it for just one imagery_request, yes?
But when you do multiple it fails?
Again I'm guessing that's because you never reset line to be an empty array again. So you're continually filling a single array.
Try the simple way first, to check it works, then start going all << on it then:
csv_string = FasterCSV.generate do |csv|
imagery_requests.each do |ir|
csv << ir.attributes.values.clone
end
end
PS - in the past I've even used clone on my line-by-line array, just to be sure I wasn't doing anything untoward with persisted stuff...

Related

When CSV.generate, generate empty field without ""

Ruby 2.2, Ruby on Rails 4.2
I'm genarating some CSV data in Ruby on Rails, and want empty fields to be empty, like ,, not like ,"", .
I wrote codes like below:
somethings_cotroller.rb
def get_data
respond_to do |format|
format.html
format.csv do
#data = SheetRepository.accounts_data
send_data render_to_string, type: :csv
end
end
end
somethings/get_data.csv.ruby
require 'csv'
csv_str = CSV.generate do |csv|
csv << [1,260,37335,'','','','','','']
...
end
And this generates CSV file like this.
get_data.csv
1,260,37335,"","","","","",""
I want CSV data like below.
1,260,37335,,,,,,
It seems like Ruby adds "" automatically.
How can I do this??
In order to get CSV to output an empty column, you need to tell it that nothing is in the column. An empty string, in ruby, is still something, you'll need to replace those empty strings with nil in order to get the output you want:
csv_str = CSV.generate do |csv|
csv << [1,260,37335,'','','','','',''].map do |col|
col.respond_to?(:empty?) && col.empty? ? nil : col
end
end
# => 1,260,37335,,,,,,
In rails you can clean that up by making use of presence, though this will blank out false as well:
csv_str = CSV.generate do |csv|
csv << [1,260,37335,'',false, nil,'','',''].map(&:presence)
end
# => 1,260,37335,,,,,,
The CSV documentation shows an option that you can use for this case. There are not examples but you can guess what it does.
The only consideration is, you need to send an array of Strings, otherwise, you will get a NoMethodError
csv_str = CSV.generate(write_empty_value: nil) do |csv|
csv << [1,260,37335,'','','','','','', false, ' ', nil].map(&:to_s)
end
=> "1,260,37335,,,,,,,false, ,\n"
The benefit of this solution is, you preserve the false.
I resolved by myself!
in somethings_controller.rb
send_data render_to_string.gsub("\"\"",""), type: :csv

How to download CSV data using ActionController::Live from MongoDB?

I have created a CSV downloader in a controller like this
format.csv do
#records = Model.all
headers['Content-Disposition'] = "attachment; filename=\"products.csv\""
headers['Content-Type'] ||= 'text/csv'
end
Now I want to create server sent events to download CSV from this for optimising purpose. I know I can do this in Rails using ActionController::Live but I have have no experience with it.
Can some one explain to me how I can
Query records as batches
Add records to stream
Handle sse from browser side
Write records to CSV files
Correct me if any of my assumptions are wrong. Help me do this in a better way. Thanks.
Mongoid automatically query your records in batches (More info over here)
To add your records to a CSV file, you should do something like:
records = MyModel.all
# By default batch_size is 100, but you can modify it using .batch_size(x)
result = CSV.generate do |csv|
csv << ["attribute1", "attribute2", ...]
records.each do |r|
csv << [r.attribute1, r.attribute2, ...]
end
end
send_data result, filename: 'MyCsv.csv'
Remember that send_data is an ActionController method!
I think you donĀ“t need SSE for generating a CSV. Just include ActionController::Live into the controller to use the response.stream.write iterating your collection:
include ActionController::Live
...
def some_action
format.csv do
# Needed for streaming to workaround Rack 2.2 bug
response.headers['Last-Modified'] = Time.now.httpdate
headers['Content-Disposition'] = "attachment; filename=\"products.csv\""
headers['Content-Type'] ||= 'text/csv'
[1,2,3,4].each do |i| # --> change it to iterate your DB records
response.stream.write ['SOME', 'thing', "interesting #{i}", "#{Time.zone.now}"].to_csv
sleep 1 # some fake delay to see chunking
end
ensure
response.stream.close
end
end
Try it with curl or similar to see the output line by line:
$ curl -i http://localhost:3000/test.csv

extremely slow csv exporting

We are trying to generate a CSV file for users to download. However, it's extremely slow, about 5 minutes for 10k lines of CSV.
Any good idea on improving? Code below:
def download_data
start_date, end_date = get_start_end_date
report_lines = #report.report_lines.where("report_time between (?) and (?)", start_date, end_date)
csv_string = CSV.generate do |csv|
report_lines.each do |report_data|
csv << [report_data.time, report_data.name, report_data.value]
end
end
respond_to do |format|
format.csv { send_data(csv_string, :filename => "#{Time.now}.csv", :type => "text/csv") }
end
end
I would start by checking if report_time is indexed, unindexed report_time is certainly going to contribute to the slowness. Refer to Active Record Migration for details on adding index.
Second, you could trim down the result to only what you need, i.e. instead of selecting all columns, select only time, name, and value:
report_lines = #report.report_lines
.where("report_time between (?) and (?)", start_date, end_date)
.select('time, name, value')
Try with:
def download_data
start_date, end_date = get_start_end_date
report_lines = #report.report_lines.where("report_time between (?) and (?)", start_date, end_date).select('time, name, value')
csv_string = CSV.generate do |csv|
report_lines.map { |row| csv << row }
end
respond_to do |format|
format.csv { send_data(csv_string, :filename => "#{Time.now}.csv", :type => "text/csv") }
end
end
Check which part takes so much time.
If it's SQL query, then try to optimize it: check if you have indexes, maybe you need to force another index.
If query is fast, but Ruby is slow, then you can try 2 things:
either try some faster CSV lib (like fastest-csv)
generate as much of CSV as possible in SQL. What I mean is: instead of creating ActiveRecord objects there, fetch just concatenated string for each row. If it's better, but still too slow, you can end up generating just one big string in database and just rendering it for ust (this solution might not seem good, and it looks quite ugly, but in one project I had to use it, because it was the fastest way)

ArgumentError on CSV output

i'm getting the following error when trying to generate a CSV:
ArgumentError in ProductsController#schedulecsv
wrong number of arguments (0 for 1)
My Products controller is set up as follows:
def schedulecsv
products = Product.find(:all)
filename ="schedule_#{Date.today.strftime('%d%b%y')}"
csv_data = CSV.generate do |csv|
csv << Product.csv_header
products.each do |p|
csv << p.to_csv
end
end
send_data csv_data,
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=#{filename}.csv"
end
Does anyone have any pointers here? Driving me bonkers!
Thanks!
From source of csv.rb place in /usr/lib/ruby/(version of your ruby gem)/csv.rb (on my machine)
Here is source code of CSV class's generate method
def CSV.generate(path, fs = nil, rs = nil, &block)
open_writer(path, 'w', fs, rs, &block)
end
generate method require filename as parameter.it will make file with given name,but You are calling CSV.generate filename was missed
so you have to passed name of file in generate call!
filename ="schedule_#{Date.today.strftime('%d%b%y')}"
CSV.generate filename do |csv|
csv << Product.csv_header
products.each do |p|
csv << p.to_csv
end
end

Changing Output for FasterCSV

I currently have a controller that will handle a call to export a table into a CSV file using the FasterCSV gem. The problem is the information stored in the database isn't clear sometimes and so I want to change the output for a particular column.
My project.status column for instance has numbers instead of statuses ie 1 in the database corresponds to Active, 2 for Inactive and 0 for Not Yet decided. When I export the table it shows 0,1,2 instead of Active, Inactive or Not Yet decided. Any idea how to implement this?
I tried a simple loop that would check the final generated CSV file and change each 0,1,2 to its corresponding output, but the problem is every other column that had a 0,1,2 would change as well. I'm not sure how to isolate the column.
Thanks in advance
def csv
qt = params[:selection]
#lists = Project.find(:all, :order=> (params[:sort] + ' ' + params[:direction]), :conditions => ["name LIKE ? OR description LIKE ?", "%#{qt}%", "%#{qt}%"])
csv_string = FasterCSV.generate(:encoding => 'u') do |csv|
csv << ["Status","Name","Summary","Description","Creator","Comment","Contact Information","Created Date","Updated Date"]
#lists.each do |project|
csv << [project.status, project.name, project.summary, project.description, project.creator, project.statusreason, project.contactinfo, project.created_at, project.updated_at]
end
end
filename = Time.now.strftime("%Y%m%d") + ".csv"
send_data(csv_string,
:type => 'text/csv; charset=UTF-8; header=present',
:filename => filename)
end
This is actually fairly easy. In your controller code:
#app/controllers/projects_controller.rb#csv
#lists.each do |project|
csv << [project.descriptive_status, project.name, project.summary, project.description, project.creator, project.statusreason, project.contactinfo, project.created_at, project.updated_at]
end
Then in your model code. You probably already have a method that decodes the DB status to a more descriptive one though:
#app/models/project.rb
ACTIVE_STATUS = 0
INACTIVE_STATUS = 1
NOT_YET_DECIDED_STATUS = 2
def descriptive_status
case status
when ACTIVE_STATUS
"Active"
when INACTIVE_STATUS
"Inactive"
when NOT_YET_DECIDED_STATUS
"Not Yet Decided"
end
end
There are probably a number of ways you can then refactor this. In the controller at least, it would probably be best to make that finder a more descriptive named scope. The constants in the model could be brought into SettingsLogic configuration or another similar gem.

Resources