I'm trying to create a Rails locale file from a CSV. The file is created and the CSV is correctly parsed, but the file is not filled. I don't have errors so I don't know what is wrong...
This is my code:
# frozen_string_literal: true
class FillLanguages
require 'csv'
def self.get
result = []
file = File.new('config/locales/languages.yml', 'w')
CSV.foreach('lib/csv/BCP-47_french.csv', headers: false, col_sep: ';') do |row|
result.push(row[0])
hash = {}
key = row[0]
hash[key] = row[1]
file.puts(hash.to_yaml)
end
result
end
end
Rails.logger.debug(hash) returns
{"af-ZA"=>"Africain (Afrique du Sud)"}
{"ar-AE"=>"Arabe (U.A.E.)"}
{"ar-BH"=>"Arabe (Bahreïn)"}
{"ar-DZ"=>"Arabe (Algérie)"}
{"ar-EG"=>"Arabe (Egypte)"}
{"ar-IQ"=>"Arabe (Irak)"}
...
as expected.
Rails.logger.debug(hash.to_yaml) returns
---
af-ZA: Africain (Afrique du Sud)
---
ar-AE: Arabe (U.A.E.)
---
ar-BH: Arabe (Bahreïn)
---
ar-DZ: Arabe (Algérie)
---
ar-EG: Arabe (Egypte)
---
ar-IQ: Arabe (Irak)
...
But the file still empty.
My CSV looks like:
https://i.gyazo.com/f3fa5ba8b1bfdd014018da5b46fa7ec0.png
Even if I try to puts a string like 'hello world' just after the line where I'm creating the file, it doesn't work...
You forgot to close the file.
You can either do it explicitly (best practice to do it in ensure block) or using File.open with block.
UPDATE:
IO#close → nil
Closes ios and flushes any pending writes to the operating system. The stream is unavailable for any further data operations; an IOError is raised if such an attempt is made. I/O streams are automatically closed when they are claimed by the garbage collector.
https://ruby-doc.org/core-2.5.0/IO.html#method-i-close
So your changes are not flushed to disc from IO buffers. You can also use explicit IO#flush to do that, but it's better to close files you opened.
# explicit close
class FillLanguages
require 'csv'
def self.get
result = []
file = File.new('config/locales/languages.yml', 'w')
CSV.foreach('lib/csv/BCP-47_french.csv', headers: false, col_sep: ';') do |row|
result.push(row[0])
hash = {}
key = row[0]
hash[key] = row[1]
file.puts(hash.to_yaml)
end
result
ensure
file.close
end
end
--
# block version
class FillLanguages
require 'csv'
def self.get
result = []
File.open('config/locales/languages.yml', 'w') do |file|
CSV.foreach('lib/csv/BCP-47_french.csv', headers: false, col_sep: ';') do |row|
result.push(row[0])
hash = {}
key = row[0]
hash[key] = row[1]
file.puts(hash.to_yaml)
end
end
result
end
end
Related
This should have been such an easy thing... buy I can't for the life of me figure out how to parse a CSV file that doesn't seem to have a specific encoding.
File.open(Rails.root.join('data', 'mike/test-csv.csv'), 'rb') { |f| f.read }
=> "ID,\x00Q\x00u\x00a\x00n\x00t\x00i\x00t\x00y\n\x006\x00e\x005\x004\x009\x001\x00e\x007\x00-\x007\x00f\x001\x005\x00-\x004\x001\x007\x00d\x00-\x00a\x004\x000\x003\x00-345\x00,\x00\x005\x000\x00.\x000\x000\x000\x000\x000\x000\x000\x000\x00\n"
Here's a gist of it, can't figure out a way to post the specific CSV.
All I get from checking the encoding of the file is that it's in binary format, any thoughts on how I could get it into a normal csv?
Note: This is a downloaded CSV so converting it to another encoding via opening it in excel and exporting (or something like that) is not an option :)
Thanks!
Updating with attempted solution 1:
path = Rails.root.join('data', 'mike/test-csv.csv')
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).each do |d|
puts d
end
Result: 6e5491e7-7f15-417d-a403-345,50.00000000
While this is correct, it ONLY works with puts, for example:
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).map { |row| row }
=> [#<CSV::Row "ID":"\u00006\u0000e\u00005\u00004\u00009\u00001\u0000e\u00007\u0000-\u00007\u0000f\u00001\u00005\u0000-\u00004\u00001\u00007\u0000d\u0000-\u0000a\u00004\u00000\u00003\u0000-345\u0000" "\u0000Q\u0000u\u0000a\u0000n\u0000t\u0000i\u0000t\u0000y":"\u0000\u00005\u00000\u0000.\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u0000">]
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).map(&:to_s)
=> ["\u00006\u0000e\u00005\u00004\u00009\u00001\u0000e\u00007\u0000-\u00007\u0000f\u00001\u00005\u0000-\u00004\u00001\u00007\u0000d\u0000-\u0000a\u00004\u00000\u00003\u0000-345\u0000,\u0000\u00005\u00000\u0000.\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u0000\n"]
It's unfortunately still not the correct string :(
Final Solution (via #ashmaroli below):
path = Rails.root.join('data', 'mike/test-csv.csv')
csv_text = ''
File.open(path, 'r') do |csv|
csv.each_line do |line|
csv_text << line.gsub(/\u0000/, '')
end
end
CSV.parse(csv_text, headers:true).map do |row| row end
Result:
[#<CSV::Row "ID":"6e5491e7-7f15-417d-a403-345" "Quantity":"50.00000000">]
Github Gist
Download Example CSV File
path = Rails.root.join('data', 'mike/test-csv.csv')
file = ""
File.open(path, 'r') do |csv|
csv.each_line do |line|
file << line.gsub(/\u0000/, '')
end
end
print file
print file.inspect # same as above just wraps the string in a
# single line with "\n" chars
I am trying to write to a CSV but i ran into a problem. I have seen this so I have just applied the solution but I get an error.
This is my code:
require 'csv'
data = Owner.find(2).cats
CSV.open("file.csv", "w") do |csv|
data.each do |cat|
csv << cat.name
end
end
I have checked in console and I am getting data for Owner.find(2).cats.
When trying to write this to my CSV i get the error:
undefined method `map' for 0:Fixnum
and when I try the simple solution from the same question :
require 'csv'
CSV.open("file.csv", "w") do |csv|
csv << "\n"
end
I get this error:
undefined method `map' for "\n":String
Do you know what I am doing wrong?
I am new to ruby so maybe I am doing one of the roockie mistakes
A CSV is a collection of rows, each of which is a collection of columns; it's a two-dimensional array, that gets converted to text form. So the top-level CSV object expects you to append arrays to it, not individual cell values.
Note that in this code:
CSV.open('filename','w') do |csv|
do stuff
end
The do stuff is only run exactly once. It's up to you to create the structure of the CSV, usually with something like this:
CSV.open('filename','w') do |csv|
data.each |item|
row = [item.field1, item.field2, item.field3]
csv << row
end
end
or even a double loop:
CSV.open('filename','w') do |csv|
data.each |item|
row = []
fields.each do |field|
row << item[field]
end
csv << row
end
end
As an example:
$ irb
irb(main):001:0> require 'csv' #=> true
irb(main):002:0> CSV.open("cats.csv", "w") do |csv|
irb(main):003:1* csv << [ "cat1" ] << [ "cat2" ] << [ "cat3 " ]
irb(main):004:1> end
#=> <#CSV io_type:File io_path:"cats.csv" encoding:UTF-8 lineno:3 col_sep:"," row_sep:"\n" quote_char:"\"">
irb(main):005:0>
$ cat cats.csv
cat1
cat2
cat3
$
Notice that the file has no quotation marks or square brackets in it.
require 'csv'
data = Owner.find(2).cats
CSV.open("file.csv", "w") do |csv|
data.each { |cat| csv << [cat.name] }
end
My ruby application uses about 97 %CPU which eventually gets killed. The program is reading files from the folder and if a file name exists in the database, it skips it and checks another file. While executing this procedure, application usually gets killed.
COMMAND %CPU
ruby 96.5
Even if I insert almost all files and try to lunch an application again (because it was killed), system tends to kill it even sooner. How can I decrease the %CPU?
task :process_data, [:data_directory] => :environment do |_task, args|
# add data to a database
saver = CsvToSqlSaver.new
saver.fill_files_names
Dir.foreach(args.data_directory) do |filename|
# if not present in records already we read it
Base.logger.info "> Found #{filename}."
next if saver.files_names.to_s.include?(filename) ||
!filename.include?('csv')
Base.logger.info "> Reading #{filename}."
begin
saver.generate_db_rows_from_csv_file(args.data_directory, filename)
# handle Malformed .csv exception
rescue CSV::ArgumentError, CSV::MalformedCSVError => e
Base.logger.info e.message
next
end # we continue csv file loop?
unless integrator.insert_data_to_database
Base.logger.info '> No new data saved.'
end
end
end
This is fill_files_names:
def fill_files_names
#files_names = []
files_names = MyFilesTable.select(:filename).distinct
files_names.each do |row|
#files_names.push(row[:filename])
end
end
This is Base:
class Base
class << self
attr_accessor :logger
end
#logger ||= Logger.new(STDERR)
end
This is generate_db_rows_from_csv_file
def generate_db_rows_from_csv_file(directory, filename)
#incoming_data = []
CSV.foreach("#{directory}/#{filename}",
headers: true, quote_char: "\x00") do |csv_record|
# if invalid record, go further
next if record_invalid?(csv_record)
generate_row_in_the_database(csv_record, filename)
end
end
I'm trying to import a CSV file from a url but i'm getting Errno::ENAMETOOLONG: file name too long. I process the file as follow:
require 'open-uri'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
SmarterCSV.process(url_data, {
...
})
What am i missing ?
You have to pass a filename which should be on server. rightnow you are passing all data . Do something like this
require 'open-uri'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
File.open('/tmp/file_name', 'w') { |file| file.write(url_data) }
SmarterCSV.process('/tmp/file_name',{ })
I had the same problem using the standard CSV library to pull in a CSV file via an http url. I was able to solve the issue without needing to write to a temporary server file with code like this:
require 'open-uri'
require 'csv'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
CSV.parse(url_data, headers: true).each do |row|
# per row processing code ...
end
Hope this helps you.
# models/concerns/import.rb
require 'open-uri'
require 'import_error'
module Import
extend ActiveSupport::Concern
class_methods do
def import_remote(url)
csv = CSV.parse(open(url), headers: true)
begin
ActiveRecord::Base.transaction do
counter = 0
csv.each do |row|
row_hash = row.to_hash
begin
instance = self.name.constantize.create!(row_hash)
rescue => e
raise ImportError.new("#{e.message}. at row: #{row_hash}")
end
counter += 1 if instance.persisted?
end
end
rescue => e
return puts e.message
end
puts "Imported #{counter} records"
end
end
end
# lib/tasks/import.rake
namespace :remote_import do
desc "Import companies from CSV"
task :your_model, [:url] do |t, args|
YourModel.import_remote(args.url)
end
end
# lib/import_error.rb
class ImportError < StandardError
end
# models/your_model.rb
class YourModel < ActiveRecord::Base
include Import
end
Gist: https://gist.github.com/victorhazbun87/9ac786961bbf7c235f76
Wow, what a vague quesetion, I know. I have a file called enc_file in my Rails repo.
In my environments/production.rb, I have:
authentication_file = "#{Rails.root}/enc_file"
unless File.exist?(authentication_file)
puts "ERROR: File not found! (#{authentication_file})"
raise SystemExit, 1
end
my_config = YAML.load(PaymentGatewayCipher.decrypt(authentication_file)).symbolize_keys!
config.app_config.pay_pal.merge!(pay_pal_config.slice(:login, :password, :business, :business_id, :cert_id, :private_key, :signature).merge(
:return_to_merchant => false,
:server => 'whatever.paypal.com'
))
Then in my payment_gateway_cipher.rb file, I have:
require 'openssl'
# Encapsulates payment gateway encryption / decryption utility functions
class PaymentGatewayCipher
class << self
def encrypt(file, options = {})
cipher = create_cipher
cipher.encrypt(cipher_key)
data = cipher.update(File.read(file))
data << cipher.final
if to_file = options[:to]
# Write it out to a different file
File.open(to_file, 'wb') do |f|
f << data
end
end
data
end
# Decrypts the given file
def decrypt(file)
cipher = create_cipher
cipher.decrypt(cipher_key)
encrypted_data = File.open(file, 'rb') {|io| io.read}
data = cipher.update(encrypted_data)
data << cipher.final
end
# Generates the cipher to be used for encryption/decryption
def create_cipher
OpenSSL::Cipher::Cipher.new('aes-256-cbc')
end
# Loads the cipher key used for the symmetric algorithm
def cipher_key
File.open(File.join(Rails.root, 'config/mystuff/live/cipher.key'), 'rb') {|io| io.read}
end
end
end
How would I decrypt the enc_file to see it's content outside of Rails? I want to view the contents, modify them, and resave the file if possible.
Thoughts?
You have the decrypt function right there, so presumably by outputting the result of that function?
puts decrypt("path/to/enc_file")
Or writing the same to a file which you can then view outside of Ruby:
File.open("decrypted_file", "w") do |f|
f.write decrypt("path/to/enc_file")
end