I have the below resque job that produces a csv file and sends it to a mailer. I want to validate that the csv file has data so I do not email blank files. For some reason, when I write a method outside of the perform method, it will not work. For example, the below code will print invalid when I know the csv file has data on the first line. If I uncomment the line below ensure it works properly, however I want to extract this checking of the file into a separate method. Is this correct?
class ReportJob
#queue = :report_job
def self.perform(application_id, current_user_id)
user = User.find(current_user_id)
client_application = Application.find(client_application_id)
transactions = application.transactions
file = Tempfile.open(["#{Rails.root}/tmp/", ".csv"]) do |csv|
begin
csv_file = CSV.new(csv)
csv_file << ["Application", "Price", "Tax"]
transactions.each do |transaction|
csv_file << [application.name, transaction.price, transaction.tax]
end
ensure
ReportJob.email_report(user.email, csv_file)
#ReportMailer.send_report(user.email, csv_file).deliver
csv_file.close(unlink=true)
end
end
end
def self.email_report(email, csv)
array = csv.to_a
if array[1].blank?
puts "invalid"
else
ReportMailer.send_report(email, csv).deliver
end
end
end
You should invoke your method as such:
ReportJob.email_report(email, csv)
Otherwise, get rid of the self in:
def self.email_report(email, csv)
# your implementation here.
end
and define your method as follows:
def email_report(email, csv)
# your implementation.
end
This is something that we call Class Methods and Instance Methods.
Related
I am trying to write a helper method that can download a CSV file from S3 storage, read the first few rows of the file and then save those first few rows to a new local file.
All is working well when I include the helper in the rails console and call the methods on the object, but when calling it in exactly the same way through the controller, the local file contains all of the rows from the S3 file, rather than just the first few.
My code, in the helper file (I've replaced AWS credentials with comments for the purpose of posting the question):
def download_file(data_source)
s3 = Aws::S3::Client.new(#API keys etc.)
File.open(data_source.file.data['id'], 'wb') do |file|
reap = s3.get_object({ bucket:#Bucket Name, key: 'store/' + data_source.file.data['id'] }, target: file)
end
end
def reduce_csv(filename)
data = CSV.open(filename, 'r') { |csv| csv.first(3) }
csv_string = CSV.generate do |csv|
data.each do |d|
csv << d
end
end
File.open('test.csv', 'wb') do |file|
file << csv_string
end
end
def make_small_data_source(data_source)
download_file(data_source)
reduce_csv(data_source.file.data['id'])
end
And in the controller:
if #data_source.save
make_small_data_source(#data_source)
Any ideas would be much appreciated!
I have this method in my models/images.rb model. I am starting with testing and having a hard time coming up with tests for it. Would appreciate your help.
def self.tags
t = "db/data.csv"
#arr = []
csvdata = CSV.read(t)
csvdata.shift
csvdata.each do |row|
row.each_with_index do |l, i|
unless l.nil?
#arr << l
end
end
end
#arr
end
First off a word of advice - CSV is probably the worst imaginable data format and is best avoided unless absolutely unavoidable - like if the client insists that manipulating data in MS Excel is a good idea (it is not).
If you have to use CSV don't use a method name like .tags which can confused for a regular ActiveRecord relation.
Testing methods that read from the file system can be quite difficult.
To start with you might want to alter the signature of the method so that you can pass a file path.
def self.tags(file = "db/data.csv")
# ...
end
That way you can pass a fixture file so that you can test it deterministically.
RSpec.describe Image do
describe "tags" do
let(:file) { Rails.root.join('spec', 'support', 'fixtures', 'tags.csv') }
it 'returns an array' do
expect(Image.tags(file)).to eq [ { foo: 'bar' }, { foo: 'baz' } ]
end
end
end
However your method is very ideosyncratic -
def self.tags
t = "db/data.csv"
#arr = []
self.tags makes it a class method yet you are declaring #arr as an instance variable.
Additionally Ruby's enumerable module provides so many methods for manipulating arrays that using an outer variable in a loop is not needed.
def self.tags(file = "db/data.csv")
csv_data = CSV.read(file)
csv_data.shift
csv_data.compact # removes nil elements
end
I am trying to learn how to get data via a screen scrape and then save it to a model. So far I can grab the data. I say this as if I do:
puts home_team
I get all the home teams returned
get_match.rb #grabbing the data
require 'open-uri'
require 'nokogiri'
module MatchGrabber::GetMatch
FIXTURE_URL = "http://www.bbc.co.uk/sport/football/premier-league/fixtures"
def get_fixtures
doc = Nokogiri::HTML(open(FIXTURE_URL))
home_team = doc.css(".team-home.teams").text
end
end
Then i want to update my model
match_fixtures.rb
module MatchFixtures
class MatchFixtures
include MatchGrabber::GetMatch
def perform
update_fixtures
end
private
def update_fixtures
Fixture.destroy_all
fixtures = get_fixtures
end
def update_db(matches)
matches.each do |match|
fixture = Fixture.new(
home_team: match.first
)
fixture.save
end
end
end
end
So the next step is where I am getting stuck. First of all I need to put the home_team results into an array?
Second part is I am passing matches through my update_db method but that's not correct, what do I pass through here, the results of the home_team from my update_fixtures method or the method itself?
To run the task I do:
namespace :grab do
task :fixtures => :environment do
MatchFixtures::MatchFixtures.new.perform
end
end
But nothing is saved, but that is to be expected.
Steep learning curve here and would appreciate a push in the right direction.
Calling css(".team-home.teams").text does not return the matching DOM elements as an array, but as a single string.
In order to obtain an array of elements, refactor get fixture into something like this:
get_teams
doc = Nokogiri::HTML(open(FIXTURE_URL))
doc.css(".team-home.teams").map { |el| el.text.strip }
end
This will return an array containing the text of the elements matching your selector, stripped out of blank and new line characters. At this point you can loop over the returned array and pass each team as an argument to your model's create method:
get_teams.each { |team| Fixture.create(home_team: team) }
You could just pass the array directly to the update method:
def update_fixtures
Fixture.destroy_all
update_db(get_fixtures)
end
def update_db(matches)
matches.each {|match| Fixture.create(home_team: match.first) }
end
Or do away with the method all together:
def update_fixtures
Fixture.destroy_all
get_fixtures.each {|match| Fixture.create(home_team: match.first) }
end
There is a table questions, and a data file questions.yml. Assume there is no 'Question' model.
'questions.yml' has some recodes dump from the table.
---
questions_001:
title: ttt1
content: ccc1
questions_002:
title: ttt2
content: ccc2
I want to load the data from the yml file, insert them to database. But I can't use rake db:fixtures:load, because it will treat the content as 'erb' template, which is not want I want
So I want to write another rake task, to load the data manually.
I can read the records by:
File.open("#{RAILS_ROOT}/db/fixtures/#{table_name}.yml", 'r') do |file|
YAML::load(file).each do |record|
# how to insert the record??
end
end
But I don't know how to insert them.
Edit:
I have tried:
Class.new(ActiveRecord::Base).create(record)
and
class Dummy < ActiveRecord::Base {}
Dummy.create(rcord)
But nothing inserted to database
Try this after loading the date from the yml file to records:
class Question < ActiveRecord::Base
# Question model just to import the yml file
end
records.each { |record| Question.create(record) }
You can simply create a model just for importing. You don't need to create the app/models/question.rb. Just write the code above in the script responsible for the importing.
UPDATE:
You can use the following function:
def create_class(class_name, superclass, &block)
klass = Class.new superclass, &block
Object.const_set class_name, klass
end
source
File.open("#{RAILS_ROOT}/db/fixtures/#{table_name}.yml", 'r') do |file|
YAML::load(file).each do |record|
model_name = table_name.singularize.camelize
create_class(model_name, ActiveRecod::Base) do
set_table_name table_name.to_sym
end
Kernel.const_get(model_name).create(record)
end
end
To use the connection directly you can use the following:
ActiveRecord::Base.connection.execute("YOUR SQL CODE")
Got it working thanks to #jigfox 's answer. Had to modify a bit for the full implementation now with Rails 4.
table_names = Dir.glob(Rails.root + 'app/models/**.rb').map { |s| Pathname.new(s).basename.to_s.gsub(/\.rb$/,'') }
table_names.each do |table_name|
table_name = table_name.pluralize
path = "#{Rails.root}/db/fixtures/#{table_name}.yml"
if File.exists?(path)
File.open(path, 'r') do |file|
y = YAML::load(file)
if !y.nil? and y
y.each do |record|
model_name = table_name.singularize.camelize
rec = record[1]
rec.tap { |hs| hs.delete("id") }
Kernel.const_get(model_name).create(rec)
end
end
end
end
end
This loads fixtures into the current RAILS_ENV, which, by default, is development.
$ rake db:fixtures:load
I'm developing a "script generator" to automatize some processes at work.
It has a Rails application running on a server that stores all data needed to make the script and generates the script itself at the end of the process.
The problem I am having is how to export the data from the ActiveRecord format to Plain Old Ruby Objects (POROs) so I can deal with them in my script with no database support and a pure-ruby implementation.
I thought about YAML, CSV or something like this to export the data but it would be a painful process to update these structures if the process changes. Is there a simpler way?
Ty!
By "update these structures if the process changes", do you mean changing the code that reads and writes the CSV or YAML data when the fields in the database change?
The following code writes and reads any AR object to/from CSV (requires the FasterCSV gem):
def load_from_csv(csv_filename, poro_class)
headers_read = []
first_record = true
num_headers = 0
transaction do
FCSV.foreach(csv_filename) do |row|
if first_record
headers_read = row
num_headers = headers_read.length
first_record = false
else
hash_values = {}
for col_index in 0...num_headers
hash_values[headers_read[col_index]] = row[col_index]
end
new_poro_obj = poro_class.new(hash_values) # assumes that your PORO has a constructor that accepts a hash. If not, you can do something like new_poro_obj.send(headers_read[col_index], row[col_index]) in the loop above
#work with your new_poro_obj
end
end
end
end
#objects is a list of ActiveRecord objects of the same class
def dump_to_csv(csv_filename, objects)
FCSV.open(csv_filename,'w') do |csv|
#get column names and write them as headers
col_names = objects[0].class.column_names()
csv << col_names
objects.each do |obj|
col_values = []
col_names.each do |col_name|
col_values.push obj[col_name]
end
csv << col_values
end
end
end