Ruby 2.0.0, Rails 4.0.3, Windows 8.1 Update, Roo 1.13.2
I am trying to open an Excel spreadsheet with embedded spaces using Roo. So far, I am unable to do that. I don't really know if this problem is restricted to Roo. If I rename it to eliminate the spaces, I have no problem with it. I tried encoding it, but then it simply said the file doesn't exist. Can I open the file while it contains spaces?
Code sample:
exceptions = [URI::InvalidURIError, IOError]
puts "f is #{f}"
puts "f exist? #{File.exist?(f)}"
begin
xls = Roo::Spreadsheet.open(f)
rescue *exceptions => e
puts e.message
end
encoded_f = URI.encode(f).to_s
puts "encoded_f is #{encoded_f}"
puts "encoded_f exist? #{File.exist?(encoded_f)}"
begin
xls = Roo::Spreadsheet.open(encoded_f)
rescue *exceptions => e
puts e.message
end
gsub_f = f.gsub(" ", "") # Rename file without spaces
File.rename(f, gsub_f)
puts "gsub_f is #{gsub_f}"
puts "gsub_f exist? #{File.exist?(gsub_f)}"
begin
xls = Roo::Spreadsheet.open(gsub_f)
rescue *exceptions => e
puts e.message
end
Output sample:
f is Whitt Report 2014-07-28-0803.xls
f exist? true
bad URI(is not URI?): Whitt Report 2014-07-28-0803.xls
encoded_f is Whitt%20Report%202014-07-28-0803.xls
encoded_f exist? false
file Whitt%20Report%202014-07-28-0803.xls does not exist
gsub_f is WhittReport2014-07-28-0803.xls
gsub_f exist? true
No message is given in the end because the file opens successfully.
This is caused by the way in which the URI module is called in the Roo::Spreadsheet#open method.
I posted a fix to this problem which has now been merged. If you update your Roo gem you should no longer have this issue.
Related
I have just wrote a code where I get a csv file passed in argument and treat it line by line ; so far, everything is okay. Now, I would like to secure my code by making sure that what we receive in argument is a .csv file.
I saw in the Ruby doc that it exist a == "--file" option but using it generate an error : the way I understood it, it seems this option only work for the txt files.
Is there a method specific that allowed to check if my file is a csv ? Here some of my code :
if ARGV.empty?
puts "j'ai rien reçu"
# option to check, don't work
elsif ARGV[0].shift == "--file"
# my code so far, whithout checking
else CSV.foreach(ARGV.shift) do |row|
etc, etc...
I think it is unpossible to make a real safe test without additional information.
Just some notes what you can do:
You get a filename in a variable filename.
First, check if it is a file:
File.exist?
Then you could check, if the encoding is correct:
raise "Wrong encoding" unless content.valid_encoding?
Has your csv always the same number of columns? And do you have only one liner?
This can be a possibility to make the next check:
content.each_line{|line|
return false if line.count(sep) < columns - 1
}
This check can be modified for your case, e.g. if you have always an exact number of rows.
In total you can define something like:
require 'csv'
#columns defines the expected numer of columns per line
def csv?(filename, sep: ';', columns: 3)
return false unless File.exist?(filename) #"No file"
content = File.read(filename, :encoding => 'utf-8')
return false unless content.valid_encoding? #"Wrong encoding"
content.each_line{|line|
return false if line.count(sep) < columns - 1
}
CSV.parse(content, :col_sep => sep)
end
if csv = csv?('test.csv')
csv.each do |row|
p row
end
end
You can use ruby-filemagic gem
gem install ruby-filemagic
Usage:
$ irb
irb(main):001:0> require 'filemagic'
=> true
irb(main):002:0> fm = FileMagic.new
=> #<FileMagic:0x7fd4afb0>
irb(main):003:0> fm.file('foo.zip')
=> "Zip archive data, at least v2.0 to extract"
irb(main):004:0>
https://github.com/ricardochimal/ruby-filemagic
Use File.extname() to check the origin file
File.extname("test.rb") #=> ".rb"
I am using a supplier's api and the response they send to our server includes a url to a file, upon trying to save this file locally I fail miserably.
def self.create_file_new(filename, ext, url)
require 'open-uri'
file = Tempfile.new(filename + ext)
file.binmode
# data = open(url).read
# data = open(url, :http_basic_authentication => [username, password])
file << open(url, :http_basic_authentication => [username, password]).read
# file.write CGI::unescape(data)
file.close
file = File.open(file.path)
return file
end
I was originally getting a OpenURI::HTTPError (401 Unauthorised): but I have since created a file named bypass_ssl_verification_for_open_uri in app/initializers containing the following:
# Make open-uri work with https
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
which I found whilst Googling on how to fix it.
I then started to get this error message: NoMethodError (undefined method 'tr' for #<StringIO:0xb5b728c4>):, I tried creating another file (cgi_escape_fix.rb in app/initializers) containing this:
require 'cgi'
class << CGI
alias_method :orig_escape, :escape
def escape(str)
orig_escape(str.to_str)
end
end
Which I also found on my Google travels but that doesn't seem to have solved anything, so I commented out the file.write CGI::unescape(data) to try a different way but still no joy.
Now in the log I am just getting a plain 500 Internal Server Error with no useful information.
The file I'm attempting to save will always be a pdf.
Ruby 1.8.7
Rails 2.3.14
Got it to work with the following (two new initializer scripts removed):
file = Tempfile.new(filename + ext)
file.binmode
file << open(url, :http_basic_authentication => [username, password]).read
file.close
file = File.open(file.path)
return file
Should also mention that this is being passed to the attachment_fu plugin incase anyone else has problems with it.
My rb file reads:
require "csv"
puts "Program1 initialized."
contents = CSV.open "data.csv", headers: true
contents.each do |row|
name = row[4]
puts name
end
...but when i run it in ruby it wont load the program. it gives me the error message about the headers:
syntax error, unexpected ':', expecting $end
contents = CSV.open "data.csv", headers: true
so I'm trying to figure out, why won't ruby let me parse this file? I've tried using other csv files I have and it won't load, and gives me an error message. I'm trying just to get the beginning of the program going! I feel like it has to do with the headers. I've updated as much as I can, mind you I'm using ruby 1.8.7. I read somewhere else that I could try to run the program in irb but it didn't seem like it needed it. so yeah... thank you in advance!!!!
Since you are using this with Ruby 1.8.7, :headers => true won't work in this way.
The simplest way to ignore the headers and get your data is to shift the first row in the data, which would be the headers:
require 'csv'
contents = CSV.open("data.csv", 'r')
contents.shift
contents.each do |row|
name = row[4]
puts name
end
If you do want to use the syntax with headers in ruby 1.8, you would need to use FasterCSV, something similar to this:
require 'fastercsv'
FasterCSV.foreach("data.csv", :headers => true) do |fcsv_obj|
puts fcsv_obj['name']
end
(Refer this question for further read: Parse CSV file with header fields as attributes for each row)
I am trying to process some very large tab-separated files. The process is:
begin
Dir["#{#data_path}*.tsv"].each do |file|
begin
CSV.foreach(file, :col_sep => "\t") do |row|
# assign columns to model and save
end
#log.info("Loaded all files into MySQL database illu.datafeeds")
rescue Exception => e
#log.warn("Unable to process the data feed: #{file} because #{e.message}")
next
end
end
However, when I execute this I get the following error:
Unable to process the file: /Users/XXXXX_2013-06-12.tsv because Illegal quoting in line 153.
The files are too big for me to go in and fix the error rows. I would like the process to continue the loop and process the file even if there are error rows.
Any suggestions?
Thanks.
just ... rescue nil the row causing the error
you can even log it with logger
before the loop:
error_log ||= Logger.new("#{Rails.root}/log/my.log")
inside the loop instead of just rescue nil use
rescue error_log.info(row.to_s)
in case you get the error before file begins to parse (before .foreach procedure) you can open it as raw file and read it as CSV later - inside the loop (like mentioned here)
..or just rescue full file parsing procedure
CSV.foreach(file, :col_sep => "\t") do |row|
...
end rescue error_log.info(row.to_s)
I'm trying to put a file on a site with WEB_DAV. (a ruby gem)
When I follow the example, I get a nil exception
#### GEMS
require 'rubygems'
begin
gem "net_dav"
rescue LoadError
system("gem install net_dav")
Gem.clear_paths
end
require 'net/dav'
uri = URI('https://staging.web.mysite');
user = "dave"
pasw = "correcthorsebatterystaple"
dav = Net::DAV.new(uri, :curl => false)
dav.verify_server = false
dav.credentials(user, pasw)
cargo = ("testing.txt")
File.open(cargo, "rb") { |stream|
dav.put(urI.path +'/'+ cargo, stream, File.size(cargo))
}
when I run this I get
`digest_auth': can't convert nil into String (TypeError)
this relates to line 197 in my nav.rb file.
request_digest << ':' << params['nonce']
So what I'm wondering is what step did I not add?
Is there a reasonable example of the correct use of this gem? Something that does something that works would be sweet :)
SIDE QUESTION: Is this the correct gem to use to do web_DAV? It seems an old unmaintained gem, perhaps there's something used by more to accomplish the task?
Try referencing the hash with a symbol rather than a string, i.e.
request_digest << ':' << params[:nonce]
In a simple test
baz = "baz"
params = {:foo => "bar"}
baz << ':' << params['foo']
results in the same error as you're getting.