FasterCSV: Read Remote CSV Files - ruby-on-rails

I can't seem to get this to work. I want to pull a CSV file from a different webserver to read in my application. This is how I'd like to call it:
url = 'http://www.testing.com/test.csv'
records = FasterCSV.read(url, :headers => true, :header_converters => :symbol)
But that doesn't work. I tried Googling, and all I came up with was this excerpt: Practical Ruby Gems
So, I tried modifying it as follows:
require 'open-uri'
url = 'http://www.testing.com/test.csv'
csv_url = open(url)
records = FasterCSV.read(csv_url, :headers => true, :header_converters => :symbol)
... and I get a can't convert Tempfile into String error (coming from the FasterCSV gem).
Can anyone tell me how to make this work?

require 'open-uri'
url = 'http://www.testing.com/test.csv'
open(url) do |f|
f.each_line do |line|
FasterCSV.parse(line) do |row|
# Your code here
end
end
end
http://www.ruby-doc.org/core/classes/OpenURI.html
http://fastercsv.rubyforge.org/

I would retrieve the file with Net::HTTP for example and feed that to FasterCSV
Extracted from ri Net::HTTP
require 'net/http'
require 'uri'
url = URI.parse('http://www.example.com/index.html')
res = Net::HTTP.start(url.host, url.port) {|http|
http.get('/index.html')
}
puts res.body

You just had a small typo. You should have used FasterCSV.parse instead of FasterCSV.read:
data = open('http://www.testing.com/test.csv')
records = FasterCSV.parse(data)

I would download it with rio - as easy as:
require 'rio'
require 'fastercsv'
array_of_arrays = FasterCSV.parse(rio('http://www.example.com/index.html').read)

I upload CSV file with Paperclip and save it to Cloudfiles and then start file processing with Delayed_job.
This worked for me:
require 'open-uri'
url = 'http://www.testing.com/test.csv'
open(url) do |file|
FasterCSV.parse(file.read) do |row|
# Your code here
end
end

Related

download and extract remote zip file using rubyzip

I am trying to download a zip file, extract the zip and read the files. Below is my code snippet:
url = "http://localhost/my.zip"
response = RestClient::Request.execute({:url => url, :method => :get, :content_type => 'application/zip'})
zipfile = Tempfile.new("downloaded")
zipfile.binmode #someone suggested to use binary for tempfile
zipfile.write(response)
Zip::ZipFile.open(zipfile.path) do |file|
file.each do |content|
data = file.read(content)
end
end
When I run this script, I see below error:
zip_central_directory.rb:97:in `get_e_o_c_d': Zip end of central directory signature not found (Zip::ZipError)
I am not able to understand what this error is for ? I can download and view the zip from the zip file url.
Couldn't get the download to work with Restclient so I used net/http instead, tested and works. Working with tempfiles and Zip gave me trouble in the past so I rather use a normal file. You can delete it afterwards.
require 'net/http'
require 'uri'
require 'zip/zip'
url = "http://localhost/my.zip"
uri = URI.parse(url)
req = Net::HTTP::Get.new(uri.path)
filename = './test.zip'
# download the zip
File.open(filename,"wb") do |file|
Net::HTTP::Proxy(proxy_host, proxy_port, proxy_user, proxy_pass).start(uri.host, uri.port) do |http|
http.get(uri.path) do |str|
file.write str
end
end
end
# and show it's contents
Zip::ZipFile.open(filename) do |zip|
# zip.each { |entry| p entry.get_input_stream.read } # show contents
zip.each { |entry| p entry.name } # show the name of the files inside
end
I suspect you have a corrupted zip.
Unzip cannot find the line of code that mark the end of the archive, so either:
The archive is corrupt.
It is not a .zip archive.
There are more than one parts to the archive.

How to save pictures from URL to disk

I want to download pictures from a URL, like: http://trinity.e-stile.ru/ and save images to a directory like "C:\pickaxe\pictures". It is important to use Nokogiri.
I read similar questions on this site, but I didn't find how it works and I didn't understand the algorithm.
I wrote the code where I parse the URL and put parts of the webpage source code with "img" tag into a links object:
require 'nokogiri'
require 'open-uri'
PAGE_URL="http://trinity.e-stile.ru/"
page=Nokogiri::HTML(open(PAGE_URL)) #parsing into object
links=page.css("img") #object with html code with img tag
puts links.length # it is 24 images on this url
puts
links.each{|i| puts i } #it looks like: <img border="0" alt="" src="/images/kroliku.jpg">
puts
puts
links.each{|link| puts link['src'] } #/images/kroliku.jpg
What method is used to save pictures after grabbing the HTML code?
How can I put the images into a directory on my disk?
I changed the code, but it has an error:
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:879:in `initialize': getaddrinfo: Name or service not known (SocketError)
This is the code now:
require 'nokogiri'
require 'open-uri'
require 'net/http'
LOCATION = 'pics'
if !File.exist? LOCATION # create folder if it is not exist
require 'fileutils'
FileUtils.mkpath LOCATION
end
#PAGE_URL = "http://ruby.bastardsbook.com/files/hello-webpage.html"
#PAGE_URL="http://trinity.e-stile.ru/"
PAGE_URL="http://www.youtube.com/"
page=Nokogiri::HTML(open(PAGE_URL))
links=page.css("img")
links.each{|link|
Net::HTTP.start(PAGE_URL) do |http|
localname = link.gsub /.*\//, '' # left the filename only
resp = http.get link['src']
open("#{LOCATION}/#{localname}", "wb") do |file|
file.write resp.body
end
end
}
You are almost done. The only thing left is to store files. Let’s do it.
LOCATION = 'C:\pickaxe\pictures'
if !File.exist? LOCATION # create folder if it is not exist
require 'fileutils'
FileUtils.mkpath LOCATION
end
require 'net/http'
.... # your code with nokogiri etc.
links.each{|link|
Net::HTTP.start(PAGE_URL) do |http|
localname = link.gsub /.*\//, '' # left the filename only
resp = http.get link['src']
open("#{LOCATION}/#{localname}", "wb") do |file|
file.write resp.body
end
end
end
That’s it.
The correct version:
require 'nokogiri'
require 'open-uri'
LOCATION = 'pics'
if !File.exist? LOCATION # create folder if it is not exist
require 'fileutils'
FileUtils.mkpath LOCATION
end
#PAGE_URL="http://trinity.e-stile.ru/"
PAGE_URL="http://www.youtube.com/"
page=Nokogiri::HTML(open(PAGE_URL))
links=page.css("img")
links.each{|link|
uri = URI.join(PAGE_URL, link['src'] ).to_s # make absolute uri
localname=File.basename(link['src'])
File.open("#{LOCATION}/#{localname}",'wb') { |f| f.write(open(uri).read) }
}

Can't open CSV file: no implicit conversion of StringIO into String

I want to read a CSV file produced by an export from one of my controllers. Here is the simple code:
def export_houses
houses_file = open("http://127.0.0.1:3000/houses/export.csv")
houses = CSV.open(houses_file, 'r:bom|utf-8', { headers: true })
...
end
The problem occurs on the CSV.open line, where I get the following error message:
TypeError: no implicit conversion of StringIO into String
from /Users/htaidirt/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/csv.rb:1256:in `initialize'
houses_file is correctly found. It's a StringIO class, but I wanted a File class to use with CSV.open.
Do you have an idea how to proceed? Thanks.
You can read a csv file from http server like this :
require 'open-uri'
require 'csv'
url = 'http://127.0.0.1:3000/houses/export.csv'
houses = CSV.new(open(url), :headers => :first_row)
or you can parse it with parse method
require 'open-uri'
require 'csv'
url = 'http://127.0.0.1:3000/houses/export.csv'
houses = CSV.parse(open(url).read, :headers => true)
Hope this helps
Try adding .string at the end of open(uri)
StringIO to String Conversion
def export_houses
houses_file = open("http://127.0.0.1:3000/houses/export.csv").string
houses = CSV.open(houses_file, 'r:bom|utf-8', { headers: true })
...
end

Ruby CSV File Parsing, Headers won't format?

My rb file reads:
require "csv"
puts "Program1 initialized."
contents = CSV.open "data.csv", headers: true
contents.each do |row|
name = row[4]
puts name
end
...but when i run it in ruby it wont load the program. it gives me the error message about the headers:
syntax error, unexpected ':', expecting $end
contents = CSV.open "data.csv", headers: true
so I'm trying to figure out, why won't ruby let me parse this file? I've tried using other csv files I have and it won't load, and gives me an error message. I'm trying just to get the beginning of the program going! I feel like it has to do with the headers. I've updated as much as I can, mind you I'm using ruby 1.8.7. I read somewhere else that I could try to run the program in irb but it didn't seem like it needed it. so yeah... thank you in advance!!!!
Since you are using this with Ruby 1.8.7, :headers => true won't work in this way.
The simplest way to ignore the headers and get your data is to shift the first row in the data, which would be the headers:
require 'csv'
contents = CSV.open("data.csv", 'r')
contents.shift
contents.each do |row|
name = row[4]
puts name
end
If you do want to use the syntax with headers in ruby 1.8, you would need to use FasterCSV, something similar to this:
require 'fastercsv'
FasterCSV.foreach("data.csv", :headers => true) do |fcsv_obj|
puts fcsv_obj['name']
end
(Refer this question for further read: Parse CSV file with header fields as attributes for each row)

Rails: Cannot Parse XML Response using Nokogiri

I'm basically trying to get the Lyric tag from the response I make to the ChartLyrics API. Here is the code I've written:
require 'nokogiri'
require 'open-uri'
request = Net::HTTP.get(URI.parse('http://api.chartlyrics.com/apiv1.asmx/GetLyric?lyricId=1710&lyricCheckSum=a4a56a99ee00cd8e67872a7764d6f9c6'))
puts request
response = Nokogiri::XML(request)
puts response.xpath("//Lyric")[0].to_s
I've read to the documentation but I did not find an answer. What I am doing wrong here?
Try the below code
require 'open-uri'
require 'nokogiri'
xml_doc = Nokogiri::XML(open('http://api.chartlyrics.com/apiv1.asmx/GetLyric?lyricId=1710&lyricCheckSum=a4a56a99ee00cd8e67872a7764d6f9c6'))
#I always prefer css than xpath
lyrics = xml_doc.css('Lyric')
if lyrics.empty?
puts "COuld not find any lyric in the XML document'
else
puts lyrics[0].to_s
end
Do response.remove_namespaces! before response.xpath

Resources