I'm trying to save the results of a survey to a csv file, so every time the survey is completed it adds a new line to the file. I have code that exports database rows to a csv and lets you download it, but i don't know how to incorporate saving the survey to begin with, or if this is even possible? I have a csv file set up with the correct headers.
When your create function is called (the action in controller where form’s submit is directed to; create on REST controllers), you can just add some custom logic to there to convert the data from form into csv structure you want.
Ruby has CSV module builtin, which can be used to both read and write CSV files.
So you want something like following
require "csv"
CSV.open "output.csv", "a+" do |csv|
# example logic from another script how to populate the file
times.each do |key, value|
csv << [ key, value ]
end
end
You just need to define structure of rows how you want, this example throws two columns per row.
EDIT: a+ makes file to be written from the end (new rows) rather than original w+ that truncates the files.
A possible solution could be to use a logger. In your application controller:
def surveys
##surveys_log ||= Logger.new("#{Rails.root}/log/surveys.log")
end
Anywhere where you would like to log the survey:
surveys.info #survey.to_csv # you'll need to implement "to_csv" yourself
Which will result in a surveys.log in your log/ folder.
Related
I am rather new to using rails and website programming, apologizes for possible basic questions.
My website needs to upload a CSV file with the format:
Course_name #header
course1
Module_name #header
module1
Task_name,Task_description,Task_expected_result #header
mod1 task1 test,mod1 task2 test description,mod1 task1 test result
mod1 task2 test,mod1 task2 test description,mod1 task2 test result
mod1 task3 test,mod1 task3 test description,mod1 task3 test result
Module_name #header
module2
Task_name,Task_description,Task_expected_result
mod2 task1 test,mod2 task1 test description,mod2 task1 test result
mod2 task2 test,mod2 task2 test description,mod2 task2 test result
My database is set up that a course will have many modules, which in turn have many tasks.
Course > many modules > many tasks.
On my website I would like to upload the .csv file and then hit a button to upload the course.
I need the reading of the file, and thus the creation of the table entries to go as following:
read course_name until blank line is hit, create course using that name, grab the course_id of the newly created course.
read Module_name until blank line is hit, create the module using that name and the course_id(is how they are connected) and grab that module_id.
then read task_name, task_description and task_expected_results and create a task using all those values and the module_id, do this until a blank line is read.
then if not EOF and another module_name is read, repeat from module creation to task creation until EOF.
I know this is a lot to ask, I've tried searching online for help but i have not had any luck there. Any help with model/controller/view code would be appreciated greatly.
You will need to modify this in order to get it to find/populate your objects, but this uses strings to get the data from the CSV.
First off I split your "CSV" based on there being a blank line gap between sections (as I said before, this is a lot like trying to send several dictionary/database files inside a CSV, rather than using a better computer interchange format)
using the sample of view that you added to the comment (as HAML):
= form_with(multipart: true) do |f|
= f.file_field :file, accept: '.csv'
= submit_tag 'Read CSV'
your file will now be passed into the Rails as an ActionDispatch::Http::UploadedFile.
We can access the contents of this file, using the `#read' method:
mycsv = params[:file].read
my code for splitting, formatting, etc (assuming that the format of the file never changes):
require 'csv'
# first split the "string" into sections
csvarr = mycsv.split(/^$/).map(&:strip)
# the first section is going to be the course. so "shift" it out of the array,
# leaving us with an even number of remaining elements.
# since it's assumed that the first section will be the course, I don't bother
# checking that it is, and just store it's value
course_name = csvarr.shift.lines.last
# You probably want to do a "course = Course.find_or_create_by(name: course_name)"
# now loop through the remaining elements, 2 at a time
csvarr.each_slice(2) do |mod|
# the first element in a block of 2 will be the header
modname = mod[0].lines.last
# You probably want to do a "module = Module.find_or_create_by(course: course, name: modname)"
# the second element will be the associated data
modcsv = CSV.parse(mod[1], headers: true)
# loop through the data to show that we have it all correctly
modcsv.each do |task|
puts "Course: #{course_name} Module: #{modname} Task: #{task["Task_name"]}\n"
end
end
I have searched a lot. I have no choice unless asking this here. Do you guys know an online convertor which has API or Gem/s that can convert PDF to Excel or CSV file?
I am not sure if here is the best place to ask this either.
My application is in Rails 4.2.
PDF file has contains a header and a big table with about 10 columns.
More info:
User upload the PDF via a form then I need to grab the PDF parse it to CSV and read the content. I tried to read the content with PDF Reader Gem however the result wasn't really promising.
I have used: freepdfconvert.com/pdf-excel Unfortunately then don't supply API. (I have contacted them)
Sample PDF
This piece of code convert the PDF into the text which is handy.
Gem: pdf-reader
def self.parse
reader = PDF::Reader.new("pdf_uploaded_by_user.pdf")
reader.pages.each do |page|
puts page.text
end
end
Now if you check the sample attached PDF you will see some fields might be empty which it means I simply can't split the text line with space and put it in an array as I won't be able to map the array to the correct fields.
Thank you.
Ok, After lots of research I couldn't find an API or even a proper software that does it. Here how I did it.
I first extract the Table out of the PDF into the Table with this API pdftables. It is cheap.
Then I convert the HTML table to CSV.
(This is not ideal but it works)
Here is the code:
require 'httmultiparty'
class PageTextReceiver
include HTTMultiParty
base_uri 'http://localhost:3000'
def run
response = PageTextReceiver.post('https://pdftables.com/api?key=myapikey', :query => { f: File.new("/path/to/pdf/uploaded_pdf.pdf", "r") })
File.open('/path/to/save/as/html/response.html', 'w') do |f|
f.puts response
end
end
def convert
f = File.open("/path/to/saved/html/response.html")
doc = Nokogiri::HTML(f)
csv = CSV.open("path/to/csv/t.csv", 'w',{:col_sep => ",", :quote_char => '\'', :force_quotes => true})
doc.xpath('//table/tr').each do |row|
tarray = []
row.xpath('td').each do |cell|
tarray << cell.text
end
csv << tarray
end
csv.close
end
end
Now Run it like this:
#> page = PageTextReceiver.new
#> page.run
#> page.convert
It is not refactored. Just proof of concept. You need to consider performance.
I might use the gem Sidekiq to run it in background and move the result to the main thread.
Check Tabula-Extractor project and also check how it is used in projects like NYPD Moving Summonses Parser and CompStat criminal complaints parser.
Ryan Bates covers csv exports in his rails casts > http://railscasts.com/episodes/362-exporting-csv-and-excel this might give you some pointers.
Edit: as you now mention you need the raw data from an uploaded PDF, you could use JavaScript to read the PDF file and the populate the data into Ryan Bates' export method. Reading PDF's was covered excellently in the following question:
extract text from pdf in Javascript
I would imagine the flow would be something like:
PDF new action
user uploads PDF
PDF show action
PDF is displayed
JavaScript reads PDF
JavaScript populates Ryan's raw data
Raw data is exported with PDF data included
I have this text file in which I have to refer to values inside it constantly. I must compare certain values from the database against certain entries in the text file. Both datasets are very large. I do not load this text file (I'll call it the changefile) into the database, because doing the comparisons in the database takes way too long. And I don't put the changefile into a database table, because it's easier to read from a file, and I don't want to have to search for an upload a new changefile every time a new one is released.
What I do instead is read a bunch of records from the db into text strings, read the file into text strings, and compare those.
I have a model in which the text strings from the changefile are represented in a variable. It looks like this:
class Standard < ActiveRecord::Base
def changes
#changes ||= read_the_file_into_an_array
end
end
This is good because I am only doing the file read once. However, that's once per instantiation of the Standard class. What I want to do is ensure that I only read the file once per deployment.
Outside of reading the file into some kind of ugly global variable in an initializer, what can I do to make sure I only ever read the file once after Rails boots up?
* UPDATE *
class MyObject < ParentObject
#changes ||= get_changes
class << self
attr_accessor :changes, :get_changes
def get_changes
<read file and return array>
end
end
end
Can't get this working. Error:
NameError: undefined local variable or method `get_changes' for MyObject:Class
I don't get it at all. Why is get_changes being accessed as a local variable?
make it be read as part of an initializer and load it into a variable or config depending on the file context
In my application I wish to allow the user to upload a CSV file and then be presented with a view of their data mapped to my columns so that the user can confirm their data is correct. Ideally allowing them to edit incorrect data.
Are there any existing solutions to this via a gem, any other standard solution or any resources that might help with what I want to achieve.
Help very much appreciated.
you can do something like:
require 'csv'
file_content = File.read(params[:file].tempfile.path)
csv = CSV.parse(file_content, :headers => true)
File.unlink(params[:file].tempfile.path)
depends on your params passed to the controller, but CVS can parse a file which is usually written to a tmp dir if uploaded, presentation of the result is up to your view layer
I'm trying to export data from my models to an excel spreadsheet. I have seen 3 ways
Using the spreadsheet gem which I didn't understand how to use it,
the examples I saw was writing to a local file but I'm looking to
generate a file every time user clicks on a link.
Creating a method called export, and running the query there, then
making a export.xls file in my view, and that file creating the
table I want to be exported to the excel file, but this approach
don't allow me to create multiple sheets.
Followed this tutorial, http://oldwiki.rubyonrails.org/rails/pages/HowToExportToExcel,
but here doesn't show how to put the link in the view, looks to me that I'm missing something in the routes, I can give github so you can take a look at my code if needed.
My choice is to just manualy generate CSV file. Like:
File.new("data.csv", "w+") do |f|
#my_data.each do |data|
f << [data.title, data.body, ...].join(", ") + "\n"
end
end
CSV file can be opened with excel or any other spreadsheet soft.
I'm using writeexcel in my most recent Rails project. A fast and simple to use way to export excel files directly - no CSV!
To use it directly in your views you have to register writeexcel as a template handler - this is excalty what my gist does. Then create a new template like export.xls.writeexcel, insert your code and you're good to go.
Plugging my own gem here, but you might have a look at https://github.com/randym/acts_as_xlsx
It gives you a bit more than writeexcel or spreadsheet in terms of localization, graphs, tables and formatting from the axlsx gem.
It also integrated with active record scoping and method chains.
Blogpost with detailed usage examples:
http://axlsx.blogspot.com/
http://axlsx.blogspot.jp/2011/12/using-actsasxlsx-to-generate-excel-data.html
http://axlsx.blogspot.jp/2011/12/axlsx-making-excel-reports-with-ruby-on.html
On Github: https://github.com/randym/axlsx
On Rubygems: https://rubygems.org/gems/axlsx
On Rubytookbox: https://www.ruby-toolbox.com/projects/axlsx
Basically it involves setting up a responder in your controller
format.xlsx {
xlsx_package = Post.to_xlsx
begin
temp = Tempfile.new("posts.xlsx")
xlsx_package.serialize temp.path
send_file temp.path, :filename => "posts.xlsx", :type => "application/xlsx"
ensure
temp.close
temp.unlink
end
}
and the following on your model
class Post < ActiveRecord::Base
acts_as_xlsx
The two blog posts above give a fairly clear walk-through.