I'm trying to import a CSV file from a url but i'm getting Errno::ENAMETOOLONG: file name too long. I process the file as follow:
require 'open-uri'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
SmarterCSV.process(url_data, {
...
})
What am i missing ?
You have to pass a filename which should be on server. rightnow you are passing all data . Do something like this
require 'open-uri'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
File.open('/tmp/file_name', 'w') { |file| file.write(url_data) }
SmarterCSV.process('/tmp/file_name',{ })
I had the same problem using the standard CSV library to pull in a CSV file via an http url. I was able to solve the issue without needing to write to a temporary server file with code like this:
require 'open-uri'
require 'csv'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
CSV.parse(url_data, headers: true).each do |row|
# per row processing code ...
end
Hope this helps you.
# models/concerns/import.rb
require 'open-uri'
require 'import_error'
module Import
extend ActiveSupport::Concern
class_methods do
def import_remote(url)
csv = CSV.parse(open(url), headers: true)
begin
ActiveRecord::Base.transaction do
counter = 0
csv.each do |row|
row_hash = row.to_hash
begin
instance = self.name.constantize.create!(row_hash)
rescue => e
raise ImportError.new("#{e.message}. at row: #{row_hash}")
end
counter += 1 if instance.persisted?
end
end
rescue => e
return puts e.message
end
puts "Imported #{counter} records"
end
end
end
# lib/tasks/import.rake
namespace :remote_import do
desc "Import companies from CSV"
task :your_model, [:url] do |t, args|
YourModel.import_remote(args.url)
end
end
# lib/import_error.rb
class ImportError < StandardError
end
# models/your_model.rb
class YourModel < ActiveRecord::Base
include Import
end
Gist: https://gist.github.com/victorhazbun87/9ac786961bbf7c235f76
Related
I'm trying to create a Rails locale file from a CSV. The file is created and the CSV is correctly parsed, but the file is not filled. I don't have errors so I don't know what is wrong...
This is my code:
# frozen_string_literal: true
class FillLanguages
require 'csv'
def self.get
result = []
file = File.new('config/locales/languages.yml', 'w')
CSV.foreach('lib/csv/BCP-47_french.csv', headers: false, col_sep: ';') do |row|
result.push(row[0])
hash = {}
key = row[0]
hash[key] = row[1]
file.puts(hash.to_yaml)
end
result
end
end
Rails.logger.debug(hash) returns
{"af-ZA"=>"Africain (Afrique du Sud)"}
{"ar-AE"=>"Arabe (U.A.E.)"}
{"ar-BH"=>"Arabe (Bahreïn)"}
{"ar-DZ"=>"Arabe (Algérie)"}
{"ar-EG"=>"Arabe (Egypte)"}
{"ar-IQ"=>"Arabe (Irak)"}
...
as expected.
Rails.logger.debug(hash.to_yaml) returns
---
af-ZA: Africain (Afrique du Sud)
---
ar-AE: Arabe (U.A.E.)
---
ar-BH: Arabe (Bahreïn)
---
ar-DZ: Arabe (Algérie)
---
ar-EG: Arabe (Egypte)
---
ar-IQ: Arabe (Irak)
...
But the file still empty.
My CSV looks like:
https://i.gyazo.com/f3fa5ba8b1bfdd014018da5b46fa7ec0.png
Even if I try to puts a string like 'hello world' just after the line where I'm creating the file, it doesn't work...
You forgot to close the file.
You can either do it explicitly (best practice to do it in ensure block) or using File.open with block.
UPDATE:
IO#close → nil
Closes ios and flushes any pending writes to the operating system. The stream is unavailable for any further data operations; an IOError is raised if such an attempt is made. I/O streams are automatically closed when they are claimed by the garbage collector.
https://ruby-doc.org/core-2.5.0/IO.html#method-i-close
So your changes are not flushed to disc from IO buffers. You can also use explicit IO#flush to do that, but it's better to close files you opened.
# explicit close
class FillLanguages
require 'csv'
def self.get
result = []
file = File.new('config/locales/languages.yml', 'w')
CSV.foreach('lib/csv/BCP-47_french.csv', headers: false, col_sep: ';') do |row|
result.push(row[0])
hash = {}
key = row[0]
hash[key] = row[1]
file.puts(hash.to_yaml)
end
result
ensure
file.close
end
end
--
# block version
class FillLanguages
require 'csv'
def self.get
result = []
File.open('config/locales/languages.yml', 'w') do |file|
CSV.foreach('lib/csv/BCP-47_french.csv', headers: false, col_sep: ';') do |row|
result.push(row[0])
hash = {}
key = row[0]
hash[key] = row[1]
file.puts(hash.to_yaml)
end
end
result
end
end
I want to read a CSV file produced by an export from one of my controllers. Here is the simple code:
def export_houses
houses_file = open("http://127.0.0.1:3000/houses/export.csv")
houses = CSV.open(houses_file, 'r:bom|utf-8', { headers: true })
...
end
The problem occurs on the CSV.open line, where I get the following error message:
TypeError: no implicit conversion of StringIO into String
from /Users/htaidirt/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/csv.rb:1256:in `initialize'
houses_file is correctly found. It's a StringIO class, but I wanted a File class to use with CSV.open.
Do you have an idea how to proceed? Thanks.
You can read a csv file from http server like this :
require 'open-uri'
require 'csv'
url = 'http://127.0.0.1:3000/houses/export.csv'
houses = CSV.new(open(url), :headers => :first_row)
or you can parse it with parse method
require 'open-uri'
require 'csv'
url = 'http://127.0.0.1:3000/houses/export.csv'
houses = CSV.parse(open(url).read, :headers => true)
Hope this helps
Try adding .string at the end of open(uri)
StringIO to String Conversion
def export_houses
houses_file = open("http://127.0.0.1:3000/houses/export.csv").string
houses = CSV.open(houses_file, 'r:bom|utf-8', { headers: true })
...
end
Wow, what a vague quesetion, I know. I have a file called enc_file in my Rails repo.
In my environments/production.rb, I have:
authentication_file = "#{Rails.root}/enc_file"
unless File.exist?(authentication_file)
puts "ERROR: File not found! (#{authentication_file})"
raise SystemExit, 1
end
my_config = YAML.load(PaymentGatewayCipher.decrypt(authentication_file)).symbolize_keys!
config.app_config.pay_pal.merge!(pay_pal_config.slice(:login, :password, :business, :business_id, :cert_id, :private_key, :signature).merge(
:return_to_merchant => false,
:server => 'whatever.paypal.com'
))
Then in my payment_gateway_cipher.rb file, I have:
require 'openssl'
# Encapsulates payment gateway encryption / decryption utility functions
class PaymentGatewayCipher
class << self
def encrypt(file, options = {})
cipher = create_cipher
cipher.encrypt(cipher_key)
data = cipher.update(File.read(file))
data << cipher.final
if to_file = options[:to]
# Write it out to a different file
File.open(to_file, 'wb') do |f|
f << data
end
end
data
end
# Decrypts the given file
def decrypt(file)
cipher = create_cipher
cipher.decrypt(cipher_key)
encrypted_data = File.open(file, 'rb') {|io| io.read}
data = cipher.update(encrypted_data)
data << cipher.final
end
# Generates the cipher to be used for encryption/decryption
def create_cipher
OpenSSL::Cipher::Cipher.new('aes-256-cbc')
end
# Loads the cipher key used for the symmetric algorithm
def cipher_key
File.open(File.join(Rails.root, 'config/mystuff/live/cipher.key'), 'rb') {|io| io.read}
end
end
end
How would I decrypt the enc_file to see it's content outside of Rails? I want to view the contents, modify them, and resave the file if possible.
Thoughts?
You have the decrypt function right there, so presumably by outputting the result of that function?
puts decrypt("path/to/enc_file")
Or writing the same to a file which you can then view outside of Ruby:
File.open("decrypted_file", "w") do |f|
f.write decrypt("path/to/enc_file")
end
I am working on an application where I have to
1) get all the links of website
2) and then get the list of all the files and file extensions in each
of the web page/link.
I am done with the first part of it :)
I get all the links of website by below code..
require 'rubygems'
require 'spidr'
require 'uri'
Spidr.site('http://testasp.vulnweb.com/') do |spider|
spider.every_url { |url|
puts url
}
end
now I have to get the all the files/file-extensions in each of the
page so I tried the below code
require 'rubygems'
require 'nokogiri'
require 'open-uri'
require 'spidr'
site = 'http://testasp.vulnweb.com'
in1=[]
Spidr.site(site) do |spider|
spider.every_url { |url| in1.push url }
end
in1.each do |input1|
input1 = input1.to_s
#puts input1
begin
doc = Nokogiri::HTML(open(input1))
doc.traverse do |el|
[el[:src], el[:href]].grep(/\.(txt|css|gif|jpg|png|pdf)$/i).map{|l| URI.join(input1, l).to_s}.each do |link|
puts link
end
end
rescue => e
puts "errrooooooooor"
end
end
but Can anybody guide me how to parse the links/webpage and get the file-
extensions in the page?
You might want to take a look at URI#parse. The URI module is a part of the Ruby standard library and is a dependency of the spidr gem. Example implementation with a spec for good measure.
require 'rspec'
require 'uri'
class ExtensionExtractor
def extract(uri)
/\A.*\/(?<file>.*\.(?<extension>txt|css|gif|jpg|png|pdf))\z/i =~ URI.parse(uri).path
{:path => uri, :file => file, :extension => extension}
end
end
describe ExtensionExtractor do
before(:all) do
#css_uri = "http://testasp.vulnweb.com/styles.css"
#gif_uri = "http://testasp.vulnweb.com/Images/logo.gif"
#gif_uri_with_param = "http://testasp.vulnweb.com/Images/logo.gif?size=350x350"
end
describe "Common Extensions" do
it "should extract CSS files from URIs" do
file = subject.extract(#css_uri)
file[:path].should eq #css_uri
file[:file].should eq "styles.css"
file[:extension].should eq "css"
end
it "should extract GIF files from URIs" do
file = subject.extract(#gif_uri)
file[:path].should eq #gif_uri
file[:file].should eq "logo.gif"
file[:extension].should eq "gif"
end
it "should properly extract extensions even when URIs have parameters" do
file = subject.extract(#gif_uri_with_param)
file[:path].should eq #gif_uri_with_param
file[:file].should eq "logo.gif"
file[:extension].should eq "gif"
end
end
end
I want to import some icons from my old site. The size of those icons is less than 10kb. So when I am trying to import the icons its returning stringio.txt file.
require "open-uri"
class Category < ActiveRecord::Base
has_attached_file :icon, :path => ":rails_root/public/:attachment/:id/:style/:basename.:extension"
def icon_from_url(url)
self.icon = open(url)
end
end
In rake task.
category = Category.new
category.icon_from_url "https://xyz.com/images/dog.png"
category.save
Try:
def icon_from_url(url)
extname = File.extname(url)
basename = File.basename(url, extname)
file = Tempfile.new([basename, extname])
file.binmode
open(URI.parse(url)) do |data|
file.write data.read
end
file.rewind
self.icon = file
end
To override the default filename of a "fake file upload" in Paperclip (stringio.txt on small files or an almost random temporary name on larger files) you have 2 main possibilities:
Define an original_filename on the IO:
def icon_from_url(url)
io = open(url)
io.original_filename = "foo.png"
self.icon = io
end
You can also get the filename from the URI:
io.original_filename = File.basename(URI.parse(url).path)
Or replace :basename in your :path:
has_attached_file :icon, :path => ":rails_root/public/:attachment/:id/:style/foo.png", :url => "/:attachment/:id/:style/foo.png"
Remember to alway change the :url when you change the :path, otherwise the icon.url method will be wrong.
You can also define you own custom interpolations (e.g. :rails_root/public/:whatever).
You are almost there I think, try opening parsed uri, not the string.
require "open-uri"
class Category < ActiveRecord::Base
has_attached_file :icon, :path =>:rails_root/public/:attachment/:id/:style/:basename.:extension"
def icon_from_url(url)
self.icon = open(URI.parse(url))
end
end
Of course this doesn't handle errors
You can also disable OpenURI from ever creating a StringIO object, and force it to create a temp file instead. See this SO answer:
Why does Ruby open-uri's open return a StringIO in my unit test, but a FileIO in my controller?
In the past, I found the most reliable way to retrieve remote files was by using the command line tool "wget". The following code is mostly copied straight from an existing production (Rails 2.x) app with a few tweaks to fit with your code examples:
class CategoryIconImporter
def self.download_to_tempfile (url)
system(wget_download_command_for(url))
##tempfile.path
end
def self.clear_tempfile
##tempfile.delete if ##tempfile && ##tempfile.path && File.exist?(##tempfile.path)
##tempfile = nil
end
def self.set_wget
# used for retrieval in NrlImage (and in future from other sies?)
if !##wget
stdin, stdout, stderr = Open3.popen3('which wget')
##wget = stdout.gets
##wget ||= '/usr/local/bin/wget'
##wget.strip!
end
end
def self.wget_download_command_for (url)
set_wget
##tempfile = Tempfile.new url.sub(/\?.+$/, '').split(/[\/\\]/).last
command = [ ##wget ]
command << '-q'
if url =~ /^https/
command << '--secure-protocol=auto'
command << '--no-check-certificate'
end
command << '-O'
command << ##tempfile.path
command << url
command.join(' ')
end
def self.import_from_url (category_params, url)
clear_tempfile
filename = url.sub(/\?.+$/, '').split(/[\/\\]/).last
found = MIME::Types.type_for(filename)
content_type = !found.empty? ? found.first.content_type : nil
download_to_tempfile url
nicer_path = RAILS_ROOT + '/tmp/' + filename
File.copy ##tempfile.path, nicer_path
Category.create(category_params.merge({:icon => ActionController::TestUploadedFile.new(nicer_path, content_type, true)}))
end
end
The rake task logic might look like:
[
['Cat', 'cat'],
['Dog', 'dog'],
].each do |name, icon|
CategoryIconImporter.import_from_url {:name => name}, "https://xyz.com/images/#{icon}.png"
end
This uses the mime-types gem for content type discovery:
gem 'mime-types', :require => 'mime/types'