I am trying to write a helper method that can download a CSV file from S3 storage, read the first few rows of the file and then save those first few rows to a new local file.
All is working well when I include the helper in the rails console and call the methods on the object, but when calling it in exactly the same way through the controller, the local file contains all of the rows from the S3 file, rather than just the first few.
My code, in the helper file (I've replaced AWS credentials with comments for the purpose of posting the question):
def download_file(data_source)
s3 = Aws::S3::Client.new(#API keys etc.)
File.open(data_source.file.data['id'], 'wb') do |file|
reap = s3.get_object({ bucket:#Bucket Name, key: 'store/' + data_source.file.data['id'] }, target: file)
end
end
def reduce_csv(filename)
data = CSV.open(filename, 'r') { |csv| csv.first(3) }
csv_string = CSV.generate do |csv|
data.each do |d|
csv << d
end
end
File.open('test.csv', 'wb') do |file|
file << csv_string
end
end
def make_small_data_source(data_source)
download_file(data_source)
reduce_csv(data_source.file.data['id'])
end
And in the controller:
if #data_source.save
make_small_data_source(#data_source)
Any ideas would be much appreciated!
Related
I want to write header only 1 time in first row when import data to csv in ruby, but the header is written many time on output file.
job_datas.each do |job_data|
#company_job = job data coverted etc....
save_job_to_csv(#company_job)
end
def save_job_to_csv(job_data)
filepath = "tmp/jobs/jobs.csv"
CSV.open(filepath, "a", :headers => true) do |csv|
if csv.blank?
csv << CompanyJob.attribute_names
end
csv << job_data.attributes.values
end
end
Any one can give me solution? Thank you so much!
You are calling save_job_to_csv the method for each job_data and pushing header every time csv << CompanyJob.attribute_names
filepath = "tmp/jobs/jobs.csv"
CSV.open(filepath, "a", :headers => true) do |csv|
# push header once
csv << CompanyJob.attribute_names
# push every job record
job_datas.each do |job_data|
#company_job = job data coverted etc....
csv << #company_job.attributes.values
end
end
The above script can be created wrapped a method but if you like to write a separate method that just saves the CSV, then you need to refactor the script when you first prepare an array of values holding header and pass it to a method that just saves to CSV.
You could do something similar to this:
def save_job_to_csv(job_data)
filepath = "tmp/jobs/jobs.csv"
unless File.file?(filepath)
File.open(filepath, 'w') do |file|
file.puts(job_data.attribute_names.join(','))
end
end
CSV.open(filepath, "a", :headers => true) do |csv|
csv << job_data.attributes.values
end
end
It just checks beforehand if the file exists and if not it adds the header. If you want tabs as column separators, you just have to change the value for the join function and add the col_sep parameter to CSV.open():
file.puts(job_data.attribute_names.join("\t"))
CSV.open(filepath, "a", :headers => true, col_sep: "\t") do |csv|
I have an app whose sole purpose is to seed data files and add the data to different CSVs which are zipped and exported by the user. My application controller is filled with lines that all look like this:
def export_tips
#appointments = Appointment.order('service_id')
send_data #appointments.to_csv_tips, filename: 'tips.csv'
end
def export_ticketpayments
#appointments = Appointment.order('service_id')
send_data #appointments.to_csv_ticketpayments, filename: 'ticketspaymentitems.csv'
end
def export_batchmanifest
#batchmanifests = Batchmanifest.all
send_data #batchmanifests.to_csv_batchmanifest, filename: "batch_manifest-#{Date.today}.csv"
end
def export_pets
#clients = Client.all
send_data #clients.to_csv_pets, filename: 'pets.csv'
end
def export_clients
#clients = Client.all
send_data #clients.to_csv_clients, filename: 'clients.csv'
end
I have it in the application controller because I used it in multiple different areas including creating single CSV exports and creating complex zip files with multiple zips and CSVs inside.
Some things that I have tried to cleanup the code include:
Different variables of this:
def csv_export (model, filename)
#model.pluralize = (model.titleize).all
send_data #model.pluralize.filename, filename: filename
end
Having each one in its own controller (could not access them from different views and other controllers easily)
I also tried to figure out how to create my own module, but was unable to do so.
My application record is just as bad with repeated lines simply meant to export the CSVs:
def self.to_csv_appointments
attributes = %w[appointment_id location_id employee_id client_id child_id notes
has_specific_employee start_time end_time]
CSV.generate(headers: true) do |csv|
csv << attributes
all.each do |appointment|
csv << attributes.map { |attr| appointment.send(attr) }
end
end
end
def self.to_csv_appointmentservices
attributes = %w[appointment_id service_id price duration]
CSV.generate(headers: true) do |csv|
csv << attributes
all.each do |appointment|
csv << attributes.map { |attr| appointment.send(attr) }
end
end
end
def self.to_csv_tickets
attributes = %w[ticket_id location_id client_id ticket_status employee_id
employee_id start_time]
headers = %w[ticket_id location_id client_id status employee_id
closed_by_employee_id closed_at]
CSV.generate(headers: true) do |csv|
csv << headers
all.each do |appointment|
csv << attributes.map { |attr| appointment.send(attr) }
end
end
end
For the application record, I have tried similar methods as those listed for the application controller, but to no avail. Again, I use the code in application record instead of in the individual model files because I need to access these in multiple parts of the site.
The code from the application controller is used mostly in the static controller and buttons on the view files. I need the ability to create the file sets, as listed below, but also allow the user to export just one CSV.
Examples from static controller to built the zip files:
def create_appointments_zip
file_stream = Zip::OutputStream.write_buffer do |zip|
#appointments = Appointment.order('service_id')
zip.put_next_entry "appointment_manifest.csv"; zip << File.binread("#{Rails.root}/app/assets/csvs/appointment_manifest.csv")
zip.put_next_entry "appointments.csv"; zip << #appointments.to_csv_appointments
zip.put_next_entry "appointment_services.csv"; zip << #appointments.to_csv_appointmentservices
zip.put_next_entry "appointment_statuses.csv"; zip << #appointments.to_csv_appointmentstatuses
end
file_stream.rewind
File.open("#{Rails.root}/app/assets/csvs/appointments.zip", 'wb') do |file|
file.write(file_stream.read)
end
end
def export_salonset
create_appointments_zip
create_tickets_zip
create_inventory_zip
create_memberships_zip
file_stream = Zip::OutputStream.write_buffer do |zip|
#saloncategories = Saloncategory.all
#salonservices = Salonservice.all
#clients = Client.all
#locations = Location.all
#salonpricings = Salonpricing.all
#staffs = Staff.order("location_id")
zip.put_next_entry "batch_manifest.csv"; zip << File.binread("#{Rails.root}/app/assets/csvs/batch_manifest_simple_salon.csv")
zip.put_next_entry "categories.csv"; zip << #saloncategories.to_csv_saloncategories
zip.put_next_entry "clients.csv"; zip << #clients.to_csv_clients
zip.put_next_entry "employees.csv"; zip << #staffs.to_csv_staff
zip.put_next_entry "locations.csv"; zip << #locations.to_csv_locations
zip.put_next_entry "pricings.csv"; zip << #salonpricings.to_csv_pricings
zip.put_next_entry "services.csv"; zip << #salonservices.to_csv_salonservices
zip.put_next_entry "appointments.zip"; zip << File.binread("#{Rails.root}/app/assets/csvs/appointments.zip")
zip.put_next_entry "inventories.zip"; zip << File.binread("#{Rails.root}/app/assets/csvs/inventories.zip")
zip.put_next_entry "tickets.zip"; zip << File.binread("#{Rails.root}/app/assets/csvs/tickets.zip")
zip.put_next_entry "addonmappings.csv"; zip << File.binread("#{Rails.root}/app/assets/csvs/addonmappings.csv")
end
file_stream.rewind
respond_to do |format|
format.zip do
send_data file_stream.read, filename: "salon_set.zip"
end
end
file_stream.rewind
File.open("#{Rails.root}/app/assets/csvs/salon_set.zip", 'wb') do |file|
file.write(file_stream.read)
end
end
Link to my repository, if that is helpful
https://github.com/atayl16/data-wizard/blob/master/app/controllers/application_controller.rb
https://github.com/atayl16/data-wizard/blob/master/app/models/application_record.rb
I know there must be a better way than writing these same lines over and over. The code works, my site works (amazingly), but I would be embarrassed for any seasoned developer to see the repository without laughing. Any help is appreciated!
In this end, I ended up using metaprogramming to clean this up. Here is an example in which I excluded some items from the array for brevity:
["bundle", "attendee", "location", "membership", "client", "staff"].each do |new_method|
define_method("#{new_method.pluralize}") do
instance_variable_set("##{new_method.pluralize}", new_method.camelcase.constantize.all)
instance_var = instance_variable_get("##{new_method.pluralize}")
send_data instance_var.public_send("to_csv_#{new_method.pluralize}"), filename: "#{new_method.pluralize}.csv"
end
end
I was able to remove 30 methods from my newly created export controller. Here is the code after pushing up the changes https://github.com/atayl16/data-wizard/blob/0011b6cf8c1fe967d73a569fa573cedc52cb8c72/app/controllers/export_controller.rb
I am running a transaction download script through Ruby. I was wondering if it is possible to label each .csv it creates with the current date/time the script was run. Below is the end of the script.
CSV.open("transaction_report.csv", "w") do |csv|
csv << header_row
search_results.each do |transaction|
transaction_details_row = header_row.map{ |attribute| transaction.send(attribute) }
csv << transaction_details_row
end
end
Like this?
CSV.open("transaction_report-#{Time.now}.csv", "w") do |csv|
csv << header_row
search_results.each do |transaction|
transaction_details_row = header_row.map{ |attribute| transaction.send(attribute) }
csv << transaction_details_row
end
end
This just appends the time of generation to the file name. For example:
"transaction_report-#{Time.now}.csv"
# => "transaction_report-2019-10-10 16:09:07 +0100.csv"
If you want to avoid spaces in the file name, you can sub these out like so:
"transaction_report-#{Time.now.to_s.gsub(/\s/, '-')}.csv"
# => "transaction_report-2019-10-10-16:09:40-+0100.csv"
Is that what you're after? It sounds right based on the question, though happy to update if you're able to correct me :)
I have the below resque job that produces a csv file and sends it to a mailer. I want to validate that the csv file has data so I do not email blank files. For some reason, when I write a method outside of the perform method, it will not work. For example, the below code will print invalid when I know the csv file has data on the first line. If I uncomment the line below ensure it works properly, however I want to extract this checking of the file into a separate method. Is this correct?
class ReportJob
#queue = :report_job
def self.perform(application_id, current_user_id)
user = User.find(current_user_id)
client_application = Application.find(client_application_id)
transactions = application.transactions
file = Tempfile.open(["#{Rails.root}/tmp/", ".csv"]) do |csv|
begin
csv_file = CSV.new(csv)
csv_file << ["Application", "Price", "Tax"]
transactions.each do |transaction|
csv_file << [application.name, transaction.price, transaction.tax]
end
ensure
ReportJob.email_report(user.email, csv_file)
#ReportMailer.send_report(user.email, csv_file).deliver
csv_file.close(unlink=true)
end
end
end
def self.email_report(email, csv)
array = csv.to_a
if array[1].blank?
puts "invalid"
else
ReportMailer.send_report(email, csv).deliver
end
end
end
You should invoke your method as such:
ReportJob.email_report(email, csv)
Otherwise, get rid of the self in:
def self.email_report(email, csv)
# your implementation here.
end
and define your method as follows:
def email_report(email, csv)
# your implementation.
end
This is something that we call Class Methods and Instance Methods.
There is CSV-export of some objects (such as tasks, contacts, etc) in my application. It just renders CSV-file like this:
respond_to do |format|
format.html
format.csv { render text: Task.to_csv } # I have self.to_csv def in model
end
It generates a CSV file when I go to '/tasks.csv' without a problem.
Now I want to export all the objects and zip them. I'm using rubyzip gem to create zip-files. Now my code for creating zip-file with all the CSVs looks like that:
Zip::ZipFile.open("#{path_to_file}.zip", Zip::ZipFile::CREATE) do |zipfile|
zipfile.file.open("tasks.csv", "w") { |f| f << open("http://#{request.host}:#{request.port.to_s}/tasks.csv").read }
# the same lines for contacts and other objects
end
But it seems that there is something wrong with it because it's executing for a long time (I'm getting Timeout::Error even if there is just one line in CSV) and the resulting zip-archive contains something broken.
How can I save my "/tasks.csv", "/contacts.csv", etc as a file on server (inside of zip-archive in this case)?
I did it! The code is:
Zip::ZipFile.open("#{path_to_file}.zip", Zip::ZipFile::CREATE) do |zipfile|
zipfile.file.open("tasks.csv", "w") do |f|
CSV.open(f, "w") do |csv|
CSV.parse(Task.to_csv) { |row| csv << row }
end
end
end