I am trying to do something simple. I would like to print the pure sql before its execution and then to print the response when the query finishes. I think that I should monkey patch one of these two methods but the sql queries of the application do not use them.
Any idea how can I do it?
I know that it sounds stupid, but then I will extend this logic.
require 'active_record/connection_adapters/postgresql_adapter'
class ActiveRecord::ConnectionAdapters::PostgreSQLAdapter
def query(sql, name = nil) #:nodoc:
File.open("file1", "a") { |f| f.write sql + "\n\n" }
log(sql, name) do
result_as_array #connection.async_exec(sql)
File.open("file1", "a") { |f| f.write "RESPONSE \n\n" }
end
end
def execute(sql, name = nil)
File.open("file2", "a") { |f| f.write sql + "\n\n" }
log(sql, name) do
#connection.async_exec(sql)
File.open("file2", "a") { |f| f.write "RESPONSE \n\n" }
end
end
end
My main idea is to handle execution methods at least for MySQL and PostgreSQL. I found that I can do it for MySQL this way:
require 'active_record/connection_adapters/mysql2_adapter'
module ActiveRecord
module ConnectionAdapters
class Mysql2Adapter < AbstractMysqlAdapter
def exec_query(sql, name = 'SQL', binds = [])
File.open("path2", "a") { |f| f.write sql + "\n" }
result = execute(sql, name)
ActiveRecord::Result.new(result.fields, result.to_a)
end
end
end
end
Maybe there is a way to handle this for all DB-s in one place?
Related
I have no idea where to start with testing this rake task. Do I need stubs? If so, how to use them? Any help would be appreciated. Thanks!
desc "Import CSV file"
task :import => [:environment] do
data = "db/data.csv"
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.where(name: c).first_or_create }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.where(id: row[0], f_name: row[1]).first_or_create
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.where(name: l.strip, model_1_id: cs[i].id).first_or_create
Model4.where(p_id: p.id, model_3_id: cl.id).first_or_create
}
end
end
end
end
Here's how I'd do it:
1) Happy-path test(s)
Rake tasks as such are a pain to test. Extract the body of the rake task into a class:
whatever.rake
desc "Import CSV file"
task :import => [:environment] do
CSVImporter.new.import "db/data.csv"
end
end
lib/csv_importer.rb
class CsvImporter
def import(data)
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.where(name: c).first_or_create }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.where(id: row[0], f_name: row[1]).first_or_create
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.where(name: l.strip, model_1_id: cs[i].id).first_or_create
Model4.where(p_id: p.id, model_3_id: cl.id).first_or_create
}
end
end
end
end
Now it's easy to write a test that calls CSVImporter.new.import on a test file (that's why import takes the file as a parameter instead of hardcoding it) and expects the results. If it's reasonable to import db/data.csv in the test environment, you can do that in a test if you want. You probably only need one test like this. No stubs are required.
2) Edge and error cases
There is a lot of logic here which, for simplicity and speed, you'll want to test without creating actual model objects. That is, yes, you'll want to stub. Model2.find_ls and FastImage.size are already easy to stub. Let's extract a method to make the other model calls easy to stub:
class CsvImporter
def import(data)
headers = CSV.open(data, 'r') { |csv| csv.first }
cs = headers[2..-1].map { |c| Model1.first_or_create_with(name: c) }
ls = Model2.find_ls
csv_contents = CSV.read(photos)
csv_contents.shift
csv_contents.each do |row|
p = Model2.first_or_create_with(id: row[0], f_name: row[1])
p_d = FastImage.size(p.file.url(:small))
p.update_attributes(dimensions: p_d)
row[2..-1].each_with_index do |ls, i|
unless ls.nil?
ls.split(',').each { |l|
cl = Model3.first_or_create_with(name: l.strip, model_1_id: cs[i].id)
Model4.first_or_create_with(p_id: p.id, model_3_id: cl.id)
}
end
end
end
end
app/models/concerns/active_record_extensions.rb
module ActiveRecordExtensions
def first_or_create_with(attributes)
where(attributes).first_or_create
end
end
and include the module in all of the models that need it.
Now it's easy to stub all of the model methods so you can write tests that simulate any database situation you like.
I'm trying to create a method that deletes files on an S3 instance, but I am getting a AWS::S3::Errors::NoSuchKey: No Such Key error when I try to call .head or .read on an object.
app/models/file_item.rb
def thumbnail
{
exists: thumbnailable?,
small: "http://#{bucket}.s3.amazonaws.com/images/#{id}/small_thumb.png",
large: "http://#{bucket}.s3.amazonaws.com/images/#{id}/large_thumb.png"
}
end
lib/adapters/amazons3/accessor.rb
module Adapters
module AmazonS3
class Accessor
S3_BUCKET = AWS::S3.new.buckets[ENV['AMAZON_BUCKET']]
...
def self.delete_file(thumbnail)
prefix_pattern = %r{http://[MY-S3-HOST]-[a-z]+.s3.amazonaws.com/}
small_path = thumbnail[:small].sub(prefix_pattern, '')
large_path = thumbnail[:large].sub(prefix_pattern, '')
small = S3_BUCKET.objects[small_path]
large = S3_BUCKET.objects[large_path]
binding.pry
S3_BUCKET.objects.delete([small, large])
end
end
end
end
example url1
"http://projectname-staging.s3.amazonaws.com/images/994/small_thumb.png"
example url2
"http://projectname-production.s3.amazonaws.com/images/994/large_thumb.png"
assuming awssdk v1 for ruby.
small = S3_BUCKET.objects[small_path]
does not actually get any objects.
from: https://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/Bucket.html
bucket.objects['key'] #=> makes no request, returns an S3Object
bucket.objects.each do |obj|
puts obj.key
end
so you would need to alter your code to something like:
to_delete = []
S3_BUCKET.objects[small_path].each do |obj|
to_delete << obj.key
end
S3_BUCKET.objects[large_path].each do |obj|
to_delete << obj.key
end
S3_BUCKET.objects.delete(to_delete)
just banged out the code, so the idea is there, you might need to correct/polish it a bit
I was able to come of with a kind of different solution thanks to your answer of #Mircea above.
def self.delete_file(thumbnail)
folder = thumbnail[:small].match(/(\d+)(?!.*\d)/)
to_delete = []
S3_BUCKET.objects.with_prefix("images/#{folder}").each do |thumb|
to_delete << thumb.key
end
# binding.pry
S3_BUCKET.objects.delete(to_delete)
end
We are trying to load fixtures for a group of models that uses a different database connection than ActiveRecord::Base (inheriting from Foo::Base in this example).
We've created this module that we include in ActiveSupport::TestCase and the specify a path to the .yml files and e.g. foo_fitures :all. This works great for the first test that runs. Fixture accessors are defined and records are found in the database. But for subsequent tests there are no records in the database.
module Foo::Fixtures
extend ActiveSupport::Concern
included do
setup :setup_foo_fixtures
teardown :teardown_foo_fixtures
class_attribute :foo_fixture_path
class_attribute :foo_fixture_table_names
self.foo_fixture_table_names = []
end
module ClassMethods
def foo_fixtures(*fixture_names)
if fixture_names.first == :all
fixture_names = Dir[foo_fixture_path.join("**/*.yml")].map { |f| File.basename(f, ".yml") }
else
fixture_names = fixture_names.flatten.map { |n| n.to_s }
end
self.foo_fixture_table_names |= fixture_names
require_fixture_classes(fixture_names)
setup_fixture_accessors(fixture_names)
end
end
def setup_foo_fixtures
#loaded_fixtures.merge!(load_foo_fixtures)
end
def teardown_foo_fixtures
Foo::Base.clear_active_connections!
end
private
def load_foo_fixtures
foo_classes = Foo::Base.subclasses.flat_map { |klass| klass.abstract_class ? klass.subclasses : klass }
class_names = foo_classes.each_with_object({}) do |klass, memo|
memo[klass.table_name.to_sym] = klass if klass.table_name.present? && foo_fixture_table_names.include?(klass.table_name)
end
foo_fixtures = ActiveRecord::Fixtures.create_fixtures(foo_fixture_path, foo_fixture_table_names, class_names) do
Foo::Base.connection
end
Hash[foo_fixtures.map { |f| [f.name, f] }]
end
end
Rails' fixture system is a bit convoluted, and I'm not able to figure out what we are missing to make sure that our extra fixtures are loaded.
OK. It looks like it might be the transactions that are removing our fixtures from the database. My guess is that the transaction has started before our code loads in the fixtures, so that's why they are there for the first test, but gone at the second.
So we changed strategy, and now we just hook into load_fixtures and fixtures. This works just fine.
module FooFixtures
module ClassMethods
def foo_fixture_classes
collect_subclasses = ->(k) { k.abstract_class ? k.subclasses.flat_map(&collect_subclasses) : k }
Foo::Base.subclasses.flat_map(&collect_subclasses)
end
def foo_fixture_path
Rails.root.join("test/foo_fixtures")
end
def foo_fixture_table_names
Dir[foo_fixture_path.join("**/*.yml")].map { |f| File.basename(f, ".yml") }
end
def fixtures(*fixture_names)
super
if fixture_names.first == :all
require_fixture_classes(foo_fixture_table_names)
setup_fixture_accessors(foo_fixture_table_names)
end
end
end
private
def load_fixtures
foo_fixture_path = self.class.foo_fixture_path
foo_fixture_table_names = self.class.foo_fixture_table_names
class_names = self.class.foo_fixture_classes.each_with_object({}) do |klass, memo|
memo[klass.table_name.to_sym] = klass if klass.table_name.present? && foo_fixture_table_names.include?(klass.table_name)
end
foo_fixtures = ActiveRecord::Fixtures.create_fixtures(foo_fixture_path, foo_fixture_table_names, class_names) do
Foo::Base.connection
end
super.merge(Hash[foo_fixtures.map { |f| [f.name, f] }])
end
end
class ActiveSupport::TestCase
extend FooFixtures::ClassMethods
prepend FooFixtures
self.foo_fixture_classes.each do |fixture_class|
set_fixture_class fixture_class.table_name.to_sym => fixture_class
end
...
end
I have a controller like this:
def download_link
#It starts a background process to handle all these things
temp_file = Tempfile.new 'temp_file'
temp_sqlite_db = SQLite3::Database.new temp_file.path
temp_sqlite_db.execute("CREATE TABLE inspection (id INTEGER NOT NULL,desc VARCHAR(255));")
inspections = Inspection.a_heavy_query_that_doesnt_worths_to_wait_so_much_for_a_reply
# Some code inserting records and creating tables, with execute() too
# more code, compressing the db and sending an email with a download link to the zip file
end
Now, I would like to know if there's a way to replace the execute() function and maybe create the tables and save records like inspection.create(something) . Thanks in advance
If anyone needs something similar, this was my implementation:
# config/initializers/sql_returner.rb
module ActiveRecord
class Base
def sql_insert
if attributes_with_quotes.empty?
connection.empty_insert_statement(self.class.table_name)
else
"INSERT INTO #{self.class.quoted_table_name} " +
"(#{quoted_column_names.join(', ')}) " +
"VALUES(#{attributes_with_quotes.values.join(', ')});"
end
end
def self.sql_create
"CREATE TABLE #{table_name} (" +
" #{ self.columns.collect{ |column|
column_sql = " #{ column.name } #{ sql_type column } "
column_sql << " PRIMARY KEY " if column.primary
column_sql << " NOT NULL " unless column.null
column_sql
}.join(', ') } );"
end
private
def self.sql_type column
case column.type
when 'datetime', 'string'
'TEXT'
else
column.type.to_s
end
end
end
end
Then, if I need to create tables and insert records, taking the same code of the question as example, I must to run:
def download_link
temp_file = Tempfile.new 'temp_file'
temp_sqlite_db = SQLite3::Database.new temp_file.path
temp_sqlite_db.execute(Inspection.sql_create)
inspections = Inspection.a_heavy_query_that_doesnt_worths_to_wait_so_much_for_a_reply
insert = ""
inspections.each{ |insp|
insert << insp.return_insert_sql
}
#.....
end
For the first method sql_insert I took as example the create method code of ActiveRecord. I know that maybe some kittens died coding this implementation, but at least for me it works.
Here's how some of my existing logging code with Log4r is working. As you can see in the WorkerX::a_method, any time that I log a message I want the class name and the calling method to be included (I don't want all the caller history or any other noise, which was my purpose behind LgrHelper).
class WorkerX
include LgrHelper
def initialize(args = {})
#logger = Lgr.new({:debug => args[:debug], :logger_type => 'WorkerX'})
end
def a_method
error_msg("some error went down here")
# This prints out: "WorkerX::a_method - some error went down here"
end
end
class Lgr
require 'log4r'
include Log4r
def initialize(args = {}) # args: debug boolean, logger type
#debug = args[:debug]
#logger_type = args[:logger_type]
#logger = Log4r::Logger.new(#logger_type)
format = Log4r::PatternFormatter.new(:pattern => "%l:\t%d - %m")
outputter = Log4r::StdoutOutputter.new('console', :formatter => format)
#logger.outputters = outputter
if #debug then
#logger.level = DEBUG
else
#logger.level = INFO
end
end
def debug(msg)
#logger.debug(msg)
end
def info(msg)
#logger.info(msg)
end
def warn(msg)
#logger.warn(msg)
end
def error(msg)
#logger.error(msg)
end
def level
#logger.level
end
end
module LgrHelper
# This module should only be included in a class that has a #logger instance variable, obviously.
protected
def info_msg(msg)
#logger.info(log_intro_msg(self.method_caller_name) + msg)
end
def debug_msg(msg)
#logger.debug(log_intro_msg(self.method_caller_name) + msg)
end
def warn_msg(msg)
#logger.warn(log_intro_msg(self.method_caller_name) + msg)
end
def error_msg(msg)
#logger.error(log_intro_msg(self.method_caller_name) + msg)
end
def log_intro_msg(method)
msg = class_name
msg += '::'
msg += method
msg += ' - '
msg
end
def class_name
self.class.name
end
def method_caller_name
if /`(.*)'/.match(caller[1]) then # caller.first
$1
else
nil
end
end
end
I really don't like this approach. I'd rather just use the existing #logger instance variable to print the message and be smart enough to know the context. How can this, or similar simpler approach, be done?
My environment is Rails 2.3.11 (for now!).
After posting my answer using extend, (see "EDIT", below), I thought I'd try using set_trace_func to keep a sort of stack trace like in the discussion I posted to. Here is my final solution; the set_trace_proc call would be put in an initializer or similar.
#!/usr/bin/env ruby
# Keep track of the classes that invoke each "call" event
# and the method they called as an array of arrays.
# The array is in the format: [calling_class, called_method]
set_trace_func proc { |event, file, line, id, bind, klass|
if event == "call"
Thread.current[:callstack] ||= []
Thread.current[:callstack].push [klass, id]
elsif event == "return"
Thread.current[:callstack].pop
end
}
class Lgr
require 'log4r'
include Log4r
def initialize(args = {}) # args: debug boolean, logger type
#debug = args[:debug]
#logger_type = args[:logger_type]
#logger = Log4r::Logger.new(#logger_type)
format = Log4r::PatternFormatter.new(:pattern => "%l:\t%d - %m")
outputter = Log4r::StdoutOutputter.new('console', :formatter => format)
#logger.outputters = outputter
if #debug then
#logger.level = DEBUG
else
#logger.level = INFO
end
end
def debug(msg)
#logger.debug(msg)
end
def info(msg)
#logger.info(msg)
end
def warn(msg)
#logger.warn(msg)
end
def error(msg)
#logger.error(msg)
end
def level
#logger.level
end
def invoker
Thread.current[:callstack] ||= []
( Thread.current[:callstack][-2] || ['Kernel', 'main'] )
end
end
class CallingMethodLogger < Lgr
[:info, :debug, :warn, :error].each do |meth|
define_method(meth) { |msg| super("#{invoker[0]}::#{invoker[1]} - #{msg}") }
end
end
class WorkerX
def initialize(args = {})
#logger = CallingMethodLogger.new({:debug => args[:debug], :logger_type => 'WorkerX'})
end
def a_method
#logger.error("some error went down here")
# This prints out: "WorkerX::a_method - some error went down here"
end
end
w = WorkerX.new
w.a_method
I don't know how much, if any, the calls to the proc will affect the performance of an application; if it ends up being a concern, perhaps something not as intelligent about the calling class (like my old answer, below) will work better.
[EDIT: What follows is my old answer, referenced above.]
How about using extend? Here's a quick-and-dirty script I put together from your code to test it out; I had to reorder things to avoid errors, but the code is the same with the exception of LgrHelper (which I renamed CallingMethodLogger) and the second line of WorkerX's initializer:
#!/usr/bin/env ruby
module CallingMethodLogger
def info(msg)
super("#{#logger_type}::#{method_caller_name} - " + msg)
end
def debug(msg)
super("#{#logger_type}::#{method_caller_name} - " + msg)
end
def warn(msg)
super("#{#logger_type}::#{method_caller_name} - " + msg)
end
def error(msg)
super("#{#logger_type}::#{method_caller_name} - " + msg)
end
def method_caller_name
if /`(.*)'/.match(caller[1]) then # caller.first
$1
else
nil
end
end
end
class Lgr
require 'log4r'
include Log4r
def initialize(args = {}) # args: debug boolean, logger type
#debug = args[:debug]
#logger_type = args[:logger_type]
#logger = Log4r::Logger.new(#logger_type)
format = Log4r::PatternFormatter.new(:pattern => "%l:\t%d - %m")
outputter = Log4r::StdoutOutputter.new('console', :formatter => format)
#logger.outputters = outputter
if #debug then
#logger.level = DEBUG
else
#logger.level = INFO
end
end
def debug(msg)
#logger.debug(msg)
end
def info(msg)
#logger.info(msg)
end
def warn(msg)
#logger.warn(msg)
end
def error(msg)
#logger.error(msg)
end
def level
#logger.level
end
end
class WorkerX
def initialize(args = {})
#logger = Lgr.new({:debug => args[:debug], :logger_type => 'WorkerX'})
#logger.extend CallingMethodLogger
end
def a_method
#logger.error("some error went down here")
# This prints out: "WorkerX::a_method - some error went down here"
end
end
w = WorkerX.new
w.a_method
The output is:
ERROR: 2011-07-24 20:01:40 - WorkerX::a_method - some error went down here
The downside is, via this method, the caller's class name isn't automatically figured out; it's explicit based on the #logger_type passed into the Lgr instance. However, you may be able to use another method to get the actual name of the class--perhaps something like the call_stack gem or using Kernel#set_trace_func--see this thread.