Seeding database from YAML - ruby-on-rails

I'm ruby-newbie and i need to seed my database from YAML. After loading YAML in seeds.rb i got this array of hash :
{"projects"=>[{"title"=>"Family", "todos"=>[{"text"=>"buy a milk", "isCompleted"=>false},
{"text"=>"Change oil in engine", "isCompleted"=>false},
{"text"=>"To send the letter", "isCompleted"=>true},
{"text"=>"To drink smt", "isCompleted"=>false}, {"text"=>"Buy t-shirt", "isCompleted"=>false}]},
{"title"=>"Job", "todos"=>[{"text"=>"Call chief", "isCompleted"=>true},
{"text"=>"To send documents", "isCompleted"=>true},
{"text"=>"Make todolist", "isCompleted"=>false}]},
{"title"=>"Other", "todos"=>[{"text"=>"To call friend", "isCompleted"=>false},
{"text"=>"Prepare for trip", "isCompleted"=>false}]}]}
My code:
seed_file = Rails.root.join('db', 'seeds', 'seeds.yml')
config = HashWithIndifferentAccess.new(YAML::load_file(seed_file))
How i can iterate it and create new Projects and Todos? Please help!

You can do something like this to iterate each of the projects and todos:
Let my_hash be set to that hash you have, then
my_hash[“projects”].each do |project|
# do whatever you need to do with each item in the hash e.g.
puts project[“title”]
# then to get the todos…
project[“todos”].each do |todo|
puts todo[“text”]
end
end

The easiest way to do it is to put your YAML seed files in db/seeds/, then put this in your db/seeds.rb file:
require 'active_record/fixtures'
seeds_dir = File.join(Rails.root, 'db', 'seeds')
seed_files = Dir["#{seeds_dir}/**/*.yml"].map {|f| f[(seeds_dir.size + 1)..-5] }
ActiveRecord::FixtureSet.create_fixtures(seeds_dir, seed_files)
That will load all of your seeds the same way that fixtures are loaded during tests.

Related

Migrating uploaded files from Active Storage to Carrierwave

For a variety of reasons I am migrating my uploads from ActiveStorage (AS) to CarrierWave (CW).
I am making rake task and have the logic sorted out - I am stumped at how to feed the AS blob into the CW file.
I am trying something like ths:
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
a.attachment.download do |file|
a.file = file
end
a.save!
end
end
This is based on these two links:
https://edgeguides.rubyonrails.org/active_storage_overview.html#downloading-files
message.video.open do |file|
system '/path/to/virus/scanner', file.path
# ...
end
and
https://github.com/carrierwaveuploader/carrierwave#activerecord
# like this
File.open('somewhere') do |f|
u.avatar = f
end
I tested this locally and the files are not mounted via the uploader. My question(s) would be:
am I missing something obvious here?
is my approach wrong and needs a new one?
Bonus Karma Question:
I can't seem to see a clear path to set the CW filename when I do this?
Here is my final rack task (based on the accepted answer) - open to tweaks. Does the job for me:
namespace :carrierwave do
desc "Import the old AS files into CW"
task import: :environment do
#files = Attachment.all
puts "#{#files.count} files to be processed"
puts "+" * 50
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
puts "Attachment #{index}: Key: #{a.attachment.blob.key} ID: #{a.id} Filename: #{a.attachment.blob.filename}"
class FileIO < StringIO
def initialize(stream, filename)
super(stream)
#original_filename = filename
end
attr_reader :original_filename
end
a.attachment.download do |file|
a.file = FileIO.new(file, a.attachment.blob.filename.to_s)
end
a.save!
puts "-" * 50
end
end
end
desc "Purge the old AS files"
task purge: :environment do
#files = Attachment.all
puts "#{#files.count} files to be processed"
puts "+" * 50
#files.each.with_index(1) do | a, index |
if a.attachment.attached?
puts "Attachment #{index}: Key: #{a.attachment.blob.key} ID: #{a.id} Filename: #{a.attachment.blob.filename}"
a.attachment.purge
puts "-" * 50
#count = index
end
end
puts "#{#count} files purged"
end
end
Now in my case I am doing this in steps - I have branched my master with this rake task and the associated MCV updates. If my site was in true production would probably run the import rake task first then confirm all went well THEN purge the old AS files.
The file object you get from the attachment.download block is a string. More precisely, the response from .download is the file, "streamed and yielded in chunks" (see documentation). I validated this by calling file.class to make sure the class is what I expected.
So, to solve your issue, you need to provide an object on which .read can be called. Commonly that is done using the Ruby StringIO class.
However, considering Carrierwave also expects a filename, you can solve it using a helper model that inherits StringIO (from blogpost linked above):
class FileIO < StringIO
def initialize(stream, filename)
super(stream)
#original_filename = filename
end
attr_reader :original_filename
end
And then you can replace a.file = file with a.file = FileIO.new(file, 'new_filename')

Extract new keys from .yml file using Ruby and git

I have a Rails project with a .yml file containing I18n translation keys. I want to create a rake task (or similar) which extracts the paths for added keys (lines git recognizes as added). It doesn't matter if the result is being written to the terminal or a file.
Example .yml file:
en:
index: # <-- new key
greeting: "Hello world!" # <-- new key
show:
title: "Old text"
body: "This is a text" # <-- new key
Example output/result of rake task:
en.index.greeting
en.show.body
Is this somehow possible? Thanks!
Yes, you can. This function will print all of I18n keys
def print_translations(prefix, x)
if x.is_a? Hash
prefix += "." if prefix.present?
x.each do |key, value|
print_translations(prefix + key.to_s, value)
end
else
puts prefix
end
end
I18n.translate(:foo)
translations_hash = I18n.backend.send :translations
print_translations "", translations_hash

How do you use fixtures with attr_encrypted

I want to test a model that uses attr_encrypted to encrypt a secret in the database
class Thing
attr_encrypted :secret, encode: true
end
But when I define the secret in a fixture the encoded newline character gets escaped out.
one:
encrypted_secret: '<%= Thing.encrypt_secret(SecureRandom.uuid) %>'
That is:
'axZFZEknxUSYdUlPhwLBbj8CwSeCW5at2INA98EcCcY7MVFdmXvk7Sb4DZhC\nm6qD\n'
Is stored in the database as:
'axZFZEknxUSYdUlPhwLBbj8CwSeCW5at2INA98EcCcY7MVFdmXvk7Sb4DZhC
m6qD'
The problem with this is that this then fails:
thing = things(:one)
assert_equal thing, Thing.find_by_secret(thing.secret)
Thing.find_by_secret(thing.secret) returns nil because the resulting SQL query tries to match the two versions of the encryped secret and fails to get a match.
I have tried:
one:
encrypted_secret: 'axZFZEknxUSYdUlPhwLBbj8CwSeCW5at2INA98EcCcY7MVFdmXvk7Sb4DZhC\nm6qD\n'
but get the same result.
How can I configure my fixtures to work with attr_encrypted?
A solution that works is to replace all '\n' with '\\n' and use double quotes. This works:
one:
encryped_secret: "<%= Thing.encrypt_secret(SecureRandom.uuid).gsub(/\n/, '\\\\n') %>"
Is there a tidier way to do this?
I faced the same situation under Rails4 + attr_encrypted + fixture + Minitest environment, and here my workaround is.
In summary, I had the following steps:
write plain (= unencrypted) text fixture with a specific file extention (in my case, it is *.yml.noenc).
write rake-task to convert from the plain fixture (.yml.noenc) to encrypted fixture (.yml).
Let me explain the detail below.
For example, "Message" model has two attributes 'name' and 'body' which are required to be encrypted as follows:
class Message < ActiveRecord::Base
attr_encrypted :name, key: ...
attr_encrypted :body, key: ...
...
end
write test/fixtures/messages.yml.noenc as follows, which has plain name and body text:
msg1:
name: Hello
body: Hello, I am here...
msg2:
name: How are you
body: Good morning, ...
write like the following rake-task (e.g. lib/tasks/encrypt_fixture.rake) to convert messages.yml.noenc to messages.yml:
require 'active_record/fixtures'
src_yml = 'test/fixtures/messages.yml.noenc'
dest_yml = 'test/fixtures/messages.yml'
task 'test' => dest_yml
namespace :[MY_APP] do
desc "generate encrypted fixture"
file dest_yml => src_yml do |t|
require Rails.root + 'config/environment'
encrypted_hash = {}
for k, v in YAML.load(ERB.new(File.read(Rails.root + src_yml)).result) do
msg = Message.new(v.merge([ANY ADDITIONAL ATTRS]))
encrypted_hash[k] = {
'encrypted_name' => msg.encrypted_name,
'encrypted_name_iv' => msg.encrypted_name_iv,
'encrypted_body' => msg.encrypted_body,
'encrypted_body_iv' => msg.encrypted_body_iv,
[ANY ADDITIONAL KEY_N_VALUE]
}
end
File.open(Rails.root + t.name, 'w') do |f|
f.write(<<EOH)
#----------------------------------------------------------------------
# DO NOT MODIFY THIS FILE!!
#
# This file is generated from #{src_yml} by:
#
# (edit #{src_yml})
# $ rake [MY_APP]:generate_fixture, or
# $ rake
#----------------------------------------------------------------------
EOH
f.write(encrypted_hash.to_yaml)
end
end
end
Please substitute [MY_APP], [ANY ADDITIONAL ATTRS], and [ANY ADDITIONAL KEY_N_VALUE] to actual values.
Then, 'rake' or 'rake test' checks file dependency between messages.yml.noenc and messages.yml, and generate messages.yml when necessary before 'rake test'.

ruby net-sftp read file line by line

I am using ruby 2.0.0 and rails 4.0.0. I have something similar to this:
require 'net/sftp'
sftp = Net::SFTP.start('ftp.app.com','username', :password => 'password')
sftp.file.open("/path/to/remote/file.csv", "r") do |f|
puts f.gets
end
This opens the file on the FTP site, but it only puts the first line of the csv file. I need to read this file row by row, preferably ignoring the header.
How can I read the file row by row, without downloading the file locally?
I solved this by doing this:
data = sftp.download!("/path/to/remote/file.csv").split(/\r\n/)
data.each do |line|
puts line
end
The proper answer for this would actually be to use the file.eof? value.
The code would look like:
require 'net/sftp'
sftp = Net::SFTP.start('ftp.app.com','username', :password => 'password')
sftp.file.open("/path/to/remote/file.csv", "r") do |f|
while !f.eof?
puts f.gets
end
end
Documentation can be found here
In my case something like this worked:
data = sftp.download!("/path/to/remote/file.csv").split(/\n/).map{ |e| e.split(/,/).map{ |x| x.gsub(/"/, "")} }
data.each do |line|
puts line
end
Will also split each row of the .csv into different array columns and remove any excess of "". Note this is for mac where line breaks are \n.

How do I recursively flatten a YAML file into a JSON object where keys are dot separated strings?

For example if I have YAML file with
en:
questions:
new: 'New Question'
other:
recent: 'Recent'
old: 'Old'
This would end up as a json object like
{
'questions.new': 'New Question',
'questions.other.recent': 'Recent',
'questions.other.old': 'Old'
}
Since the question is about using YAML files for i18n on a Rails app, it's worth noting that the i18n gem provides a helper module I18n::Backend::Flatten that flattens translations exactly like this:
test.rb:
require 'yaml'
require 'json'
require 'i18n'
yaml = YAML.load <<YML
en:
questions:
new: 'New Question'
other:
recent: 'Recent'
old: 'Old'
YML
include I18n::Backend::Flatten
puts JSON.pretty_generate flatten_translations(nil, yaml, nil, false)
Output:
$ ruby test.rb
{
"en.questions.new": "New Question",
"en.questions.other.recent": "Recent",
"en.questions.other.old": "Old"
}
require 'yaml'
yml = %Q{
en:
questions:
new: 'New Question'
other:
recent: 'Recent'
old: 'Old'
}
yml = YAML.load(yml)
translations = {}
def process_hash(translations, current_key, hash)
hash.each do |new_key, value|
combined_key = [current_key, new_key].delete_if { |k| k.blank? }.join('.')
if value.is_a?(Hash)
process_hash(translations, combined_key, value)
else
translations[combined_key] = value
end
end
end
process_hash(translations, '', yml['en'])
p translations
#Ryan's recursive answer is the way to go, I just made it a little more Rubyish:
yml = YAML.load(yml)['en']
def flatten_hash(my_hash, parent=[])
my_hash.flat_map do |key, value|
case value
when Hash then flatten_hash( value, parent+[key] )
else [(parent+[key]).join('.'), value]
end
end
end
p flatten_hash(yml) #=> ["questions.new", "New Question", "questions.other.recent", "Recent", "questions.other.old", "Old"]
p Hash[*flatten_hash(yml)] #=> {"questions.new"=>"New Question", "questions.other.recent"=>"Recent", "questions.other.old"=>"Old"}
Then to get it into json format you just need to require 'json' and call the to_json method on the hash.

Resources