parsing CSV file with delayed job issue - ruby-on-rails

I have a CSV file to parse with two ways:
Upload the file and parse it in real time
Upload the file and parse it with a delayed_job
When I parse the file in real time, there is no problem, but when I parse the file with a delayed job, I get a CSV::Table but with nil fields [nil,nil.....]
This is how I parse the file in the helper:
CSV.parse(file.read, :headers => true, :row_sep =>:auto)
This is how I save the csv file and I create the delayed job:
yeah its CSV, I made a typo, but I still have the problem, I read the file in an action like this:
def create
if params[:dump]
file = File.new(params[:dump][:file].original_filename, "w")
File.open(file.path, "w") { |f| f.write(params[:dump][:file].read.force_encoding('UTF-8')) }
#import = Import.create(:created_by => current_user.id, :import_file_path => file.path, :size => 0)
file.close
file_to_parse = File.open(#import.import_file_path)
if params[:dump][:file].size < 10000
parsed_file = parsed file_to_parse
if parsed_file && parsed_file.count > 0
#imported_contacts, #imported_organizations = extract_contacts_and_organizations_from parsed_file
validate_and_save #imported_organizations if #imported_organizations.any?
validate_and_save #imported_contacts if #imported_contacts.any?
#import.update_attributes(:done => true, :size => (#imported_contacts.size + #imported_organizations.size))
redirect_to org_import_path(current_org, #import)
else
redirect_to new_org_import_path(current_org) , :notice => "fichier vide ou introuvable ou ne pas de format cvs/vcf"
end
else
Delayed::Job.enqueue(ImportJob.new(current_org, current_user, #import.id))
redirect_to org_import_path(current_org, #import)
end
else
redirect_to new_org_import_path(current_org) , :notice => "vous devrez selctionner un fichier"
end
end

In your code, #import object has the :import_file_path variable that points to a tempfile (file.path) uploaded by a user.
Since this tempfile will be automatically removed as soon as the action finishes, the delayed job can't find the file when it attempts to run the job in the background (the action is already finished).
What about to copy a tempfile to some place and use it instead?
In this case, you need to add a line of code that deletes the copied file at the end of the delayed job process.
I hope this works for your problem.

Related

Temp files disappears when CarrierWave processing

I created an ActiveJob to process my carrier waves uploads. However, when I upload more than one image, I get the following error for the second file:
Errno::ENOENT (No such file or directory # rb_sysopen - C:/Users/tdavi/AppData/Local/Temp/RackMultipart20180830-392-z2s2i.jpg)
Here's the code in my controller:
if #post.save
files = params[:post_attachments].map { |p|
{image: p['photo'][:image].tempfile.path, description: p['photo'][:decription]}
}
ProcessPhotosJob.perform_later(#post.id, files.to_json)
format.html { render :waiting }
end
And my ActiveJob
require 'json'
class ProcessPhotosJob < ApplicationJob
queue_as :default
def perform(post_id, photos_json)
post = Post.friendly.find(post_id)
photos = JSON.parse photos_json
photos.each do |p|
src_file = File.new(p['image'])
post.post_attachments.create!(:photo => src_file, :description => p[:description])
end
post.processed = true
post.save
end
end
When I upload only one file to upload, it works okay.
You should not pass Tempfile to the queued jobs.
First of all - TempFiles can be deleted automatically by Ruby (docs, explanation)
If you would like to upload file(s) and process them later (in a background), then I would suggest you check this question.

Rails console vs a rake task: returning File.size is not consistent

I'm having a strange issue where when I check the File.size of a particular file in Rails console, it returns the correct size. However when I run the same code in a rake task, it returns 0. Here is the code in question (I've tidied it up a bit to help with readability):
def sum_close
daily_closed_tickets = Fst.sum_retrieve_closed_tickets
daily_closed_tickets.each do |ticket|
CSV.open("FILE_NAME_HERE", "w+", {force_quotes: false}) do |csv|
if (FileCopyReceipt.exists?(path: "#{ticket.attributes['TroubleTicketNumber']}_sum.txt"))
csv << ["GENERATE CSV WITH ATTRIBUTES HERE"]
files = Dir.glob("/var/www/html/harmonize/public/close/CLOSED_#{ticket.attributes['TroubleTicketNumber']}_sum.txt")
files.each do |f|
Rails.logger.info "File size (should return non-0): #{File.size(f)}" #returns 0, but not in Rails Console
Rails.logger.info "File size true or false, should be true: #{File.size(f) != 0}" #returns false, should return true
Rails.logger.info "Rails Environment: #{Rails.env}" #returns production
if(!FileCopyReceipt.exists?(path: f) && (File.size(f) != 0))
Rails.logger.info("SUM CLOSE, GOOD => FileUtils.cp_r occurred and FileCopyReceipt object created")
else
Rails.logger.info("SUM CLOSE, WARNING: => no data transfer occurred")
end
end
else
Rails.logger.info("SUM CLOSE => DID NOT make it into initial if ClosedDate.present? if block")
end
end
end
close_tickets.rake
task :close_tickets => :environment do
tickets = FstController.new
tickets.sum_close
tickets.dais_close
end
It is beyond me why this File.size comes back as 0 when this is run as a rake task. I thought it may be a environment issue, but that does not seem to be the case.
Any insight on the matter is appreciated.
The CSV.open block and everything being wrapped in there was causing issues. So I just made CSV generation it's own snippet instead of wrapping everything in there.
daily_closed_tickets.each do |ticket|
CSV.open("generate csv here.txt") do |csv|
#enter ticket.attributes here for the csv
end
#continue on with the rest of the code and File.size() works properly
end

Export large mount of data in a zip

I'm exporting some data from my server to the client.
It's an zip archive but when the amount of data is to big : TimeOut !
#On my controller
def export
filename = 'my_archive.zip'
temp_file = Tempfile.new(filename)
begin
Zip::OutputStream.open(temp_file) { |zos| }
Zip::File.open(temp_file.path, Zip::File::CREATE) do |zip|
#videos.each do |v|
video_file_name = v.title + '.mp4'
zip.add(video_file_name, v.source.file.path(:original))
end
end
zip_data = File.read(temp_file.path)
send_data(zip_data, :type => 'application/zip', :filename => filename)
ensure
temp_file.close
temp_file.unlink
end
end
I'm using PaperClip to attach my video on my app.
Is there any way to create and upload the zip (with a stream?) without a too long wait?
You could try the zipline gem. It claims to be "Hacks on Hacks on Hacks" so heads up! Looks very easy to use though, worth a shot.

Invalid encoding with rqrcode

I'm having an invalid encoding error that doesn't let me save the image to a carrierwave uploader.
require 'rqrcode_png'
img = RQRCode::QRCode.new( 'test', :size => 4, :level => :h ).to_img.to_s
img.valid_encoding?
=> false
I'm not sure if this is what you're looking for, in my case I needed to associate the generated QR code with a Rails model using carrierwave, what I ended up doing was saving the image to a temp file, associating that file with the model and afterwards deleting the temp file, here's my code:
def generate_qr_code!
tmp_path = Rails.root.join('tmp', "some-filename.png")
tmp_file = RQRCode::QRCode.new(self.hash_value).to_img.resize(200,200).save(tmp_path)
# Stream is handed closed, we need to reopen it
File.open(tmp_file.path) do |file|
self.qr_code = file
end
File.delete(tmp_file.path)
self.save!
end

Difficulty with send_data in Ruby on Rails in conjunction with Spreadsheet plug-in

I have a function in a controller that takes in some specifications and generates a report on them. This function user_report is called in a view:
< %= submit_to_remote 'submit-button', "Export Report to Excel", :url => { :controller => :reports, :action => :user_report, :print_state => 'print'} % >
In reports_controller I use the Spreadsheet plugin to generate an Excel file within the user_report function. I want to use send_data to stream the file to the user without creating it on the server first. The research I've done shows that using StringIO is the way to go, as shown below. Frustratingly, nothing happens when I call send_data. The plugin seems to work well creating a file and saving it on the server, but does nothing when I use send_file, suggesting that the problem doesn't lie in the plugin. But then what am I doing wrong with send_file/send_data?
The function itself looks like this:
def user_report
if request.post?
unless params[:reports][:userid].blank?
#userid=params[:reports][:userid]
end
if params[:print_state]=='print'
report = Spreadsheet::Workbook.new
info = report.create_worksheet :name => 'User Information'
info.row(1).push 'User ID', #userid
#outfile = "Report_for_#{#userid}.xls"
require 'stringio'
data = StringIO.new ''
report.write data
send_data data.string, :type=>"application/excel", :disposition=>'attachment', :filename => #outfile
end
respond_to do |format|
format.js { }
end
end
end
The log file reads
2010-10-18 14:13:59 INFO -- Sending data Report_for_jjohnson.xls
but no download begins in-browser. I've succeed in using send_data on this app before, which is confusing.
I'm using Rails v2.3, Ruby v1.8.7, and Spreadsheet v6.4.1 at spreadsheet.rubyforge.org.
Just change the line:
send_data data.string, :type=>"application/excel", :disposition=>'attachment', :filename => #outfile
to:
send_data data.string.bytes.to_a.pack("C*"), :type=>"application/excel", :disposition=>'attachment', :filename => #outfile
Even though I dont like to write and delete , but with spreadsheet seems like the only solution.
# write the file
book.write "Employee_History_#{ params[:id]}.xls"
# send the file
send_file "Employee_History_#{ params[:id]}.xls", :type => "application/vnd.ms-excel", :filename => "data.xls", :stream => false
# and then delete the file
File.delete("Employee_History_#{ params[:id]}.xls")
For someone looking at this in (or after) 2022, a possible solution to this would be to use Axlsx Gem. The interface provides a method for converting it to a StringIO object. (From Axlsx Documentation)
#serialize to a file
p = Axlsx::Package.new
# ......add cool stuff to your workbook......
# Serialize to a stream
s = p.to_stream()
send_data(
s.read,
:type => 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
:disposition => 'attachment',
:filename => #filename
)

Resources