How do I print the require time of my files so that I can debug a slow application boot time.
I envision something like this:
$ rails s
$ app/models/user.rb took 300 milliseconds
$ ...
$ lib/connect.rb took 5000 milliseconds
Would I have to override Kernel#require and if so, how?
# ACK:
# modified from
# http://gist.github.com/264496 by github.com/eric
# http://gist.github.com/465293 by github.com/alexyoung
# USAGE:
# 1) > ruby -rrequire_tracking -e "require 'active_support'"
# 2) put in config/preinitializer.rb
# WHAT: Provides a simple overview of memory allocation occuring
# during a require.
#
# For a longer explanation, see post at:
# http://bitmonkey.net/post/308322913/tracking-initial-memory-usage-by-file-in-ruby
#
# NOTE: The numbers provided are of self + children, meaning the same will
# be attributed to multiple files at once.
#
# Also, memory that is no longer referenced (and would be freed) is
# still taken into account.
#
# It is intended to give a simple overview of allocations to track
# down gross offenders, not give an exact view of your memory usage.
require 'benchmark'
if GC.respond_to?(:enable_stats)
module RequireTracking
def require(*args)
start_allocated_size = GC.allocated_size
output = nil
benchmark = Benchmark.realtime do #Benchmark.measure
output = super
end
benchmark = (benchmark * 100000).to_i
first_caller = caller[0][40..-1].split(':')[0]
$my_require_stats << [args[0], benchmark, first_caller, (GC.allocated_size - start_allocated_size)]
end #def
end #module
else
module RequireTracking
def require(*args)
output = nil
benchmark = Benchmark.realtime do #Benchmark.measure
output = super
end
benchmark = (benchmark * 1000_00).to_i
first_caller = caller[0][40..-1].split(':')[0]
$my_require_stats << [path, benchmark, first_caller, 'NA']
end #def
end #module
puts "Warning: Not running with an interpreter with GC statistics"
end #if
module RequireTracking
$my_require_stats ||= []
$require_stats_top ||= 20
def numeric_thousands_indicators(number)
number.to_s.gsub(/(\d)(?=\d{3}+(?:\.|$))(\d{3}\..*)?/,'\1,\2')
end
def dump_require_benchmark_stats
File.open('require_trace.log.csv', 'w') do |out|
out << "\"path\",\"benchmark\",\"caller\",\"mem\"\n"
$my_require_stats.each do |path, benchmark, caller, mem|
out << "\"#{path}\",\"#{benchmark}\",\"#{caller}\",\"#{mem}\"\n"
end
end
end
def print_require_benchmark_stats_by_memory
puts " %40s %5s %5s " % [ 'File', 'KB', 'NanoSec' ]
puts " %40s %s" % [ '-------------', '--------' ]
$my_require_stats.sort_by {|v| v[3] }.slice(0, $require_stats_top).each do |path, benchmark, caller, mem|
puts "%40s %5s %5s " % [ path, numeric_thousands_indicators(mem / 1024), benchmark ]
end
end
def print_require_benchmark_stats_by_time
puts " %40s %5s %5s " % [ 'File', 'KB', 'NanoSec' ]
puts " %40s %s" % [ '-------------', '--------' ]
$my_require_stats.sort_by {|v| v[1] }.slice(0, $require_stats_top).each do |path, benchmark, caller, mem|
puts "%40s %5s %5s " % [ path, numeric_thousands_indicators(mem / 1024), benchmark ]
end
end
end #module
Object.send(:include, RequireTracking)
Kernel.send(:include, RequireTracking)
if GC.respond_to?(:enable_stats)
GC.enable_stats
GC.clear_stats
end
at_exit do
dump_require_benchmark_stats
puts "Memory used by file:"
print_require_benchmark_stats_by_memory
puts "Time required by file:"
print_require_benchmark_stats_by_time
end
Related
So I am working on some stupid simulation of mining cryptocoins - just for fun.
But I have run into an issue I want to command the client to start using a function but I am not sure how to do it, my code is bellow for Server.rb and Client.rb
Client
require 'socket'
s = TCPSocket.new 'localhost', 2626
while line = s.gets
puts line
end
s.close
Server.rb
#!/bin/ruby
require "socket"
require 'securerandom'
PORT = 2626
server_pool = TCPServer.open(PORT)
sndc_block_time = 200 # Size of block
sndc_block_time = 10 # Time to brake block 1,1 out of 200, 200
sndc_coin_blocks= 126^2 # Ammount of available blocks
sndc_coin_balance = 0
sub_blocks = 0 # Sub blocks user wants to mine
addr = SecureRandom.hex # generate a random hexadecimal address
# Notify message shown when user registers a new address
disclaimer = "Welcome to the SendCoin Network!
A new address has been registered and saved
to your computer desktop!"
def handle_connection(client)
puts "New client! #{client} \n\n"
end
puts "Listening on #{PORT}. Press CTRL+C to cancel."
while client = server_pool.accept
if Thread.new { handle_connection(client) }
if sndc_coin_blocks < 126^2 / 4
sndc_coin_blocks + 126^2 / 4 # Add a quarter of the original size
end
if Dir['../MyAddress/*.addr'].any? == true
# Statrs Mining
client.puts "Starting to mine.."
sleep(5)
client.puts "Current blocks: " + sndc_coin_blocks.to_s
client.puts "Block Size: " + sndc_coin_balance.to_s
client.puts "Approximate time to mine 1 sub-block: " + sndc_block_time.to_s
client.puts "Searching for block.."
sleep(3)
if sndc_block_time != 10
sndc_block_time = 10
end
client.puts "Started mining..."
elsif Dir['../addresses/*.addr'].any? == false
File.open("../addresses/"+addr+".addr", "w") do |f| # Create file
f.write(sndc_coin_balance.to_s)
end
client.puts "New address generated: " + addr.to_s + " , you may realunch the app now and enter your address!"
# Start Mining
puts "> File 'address.addr' has been generated for#{client}"
end
end
end
So essentially I want after this line of code:
client.puts "Started mining..."
.. to command the client to (client.rb) to start using a function.
Originally posted as
https://github.com/Mange/roadie-rails/issues/75
We are seeing performance issue for our daily email jobs
By using NewRelic custom instrumentation,
we found out that most time is spent in calling Roadies
Screenshot of our NewRelic data for an example worker:
The integration code:
# frozen_string_literal: true
require "rails"
require "action_controller"
require "contracts"
require "memoist"
require "roadie"
require "roadie-rails"
require "new_relic/agent/method_tracer"
module Shared::MailerMixins
module WithRoadieIntegration
# I don't want to include the constants into the class as well
module Concern
def self.included(base)
base.extend ClassMethods
end
include ::NewRelic::Agent::MethodTracer
def mail(*args, &block)
super.tap do |m|
options = roadie_options
next unless options
trace_execution_scoped(
[
[
"WithRoadieIntegration",
"Roadie::Rails::MailInliner.new(m, options).execute",
].join("/"),
],
) do
Roadie::Rails::MailInliner.new(m, options).execute
end
end
end
private
def roadie_options
::Rails.application.config.roadie.tap do |options|
options.asset_providers = [UserAssetsProvider.new]
options.external_asset_providers = [UserAssetsProvider.new]
options.keep_uninlinable_css = false
options.url_options = url_options.slice(*[
:host,
:port,
:path,
:protocol,
:scheme,
])
end
end
add_method_tracer(
:roadie_options,
"WithRoadieIntegration/roadie_options",
)
end
class UserAssetsProvider
extend(
::Memoist,
)
include(
::Contracts::Core,
::Contracts::Builtin,
)
include ::NewRelic::Agent::MethodTracer
ABSOLUTE_ASSET_PATH_REGEXP = /\A#{Regexp.escape("//")}.+#{Regexp.escape("/assets/")}/i
Contract String => Maybe[Roadie::Stylesheet]
def find_stylesheet(name)
return nil unless file_exists?(name)
Roadie::Stylesheet.new("whatever", stylesheet_content(name))
end
add_method_tracer(
:find_stylesheet,
"UserAssetsProvider/find_stylesheet",
)
Contract String => Roadie::Stylesheet
def find_stylesheet!(name)
stylesheet = find_stylesheet(name)
if stylesheet.nil?
raise Roadie::CssNotFound.new(
name,
"does not exists",
self,
)
end
stylesheet
end
add_method_tracer(
:find_stylesheet!,
"UserAssetsProvider/find_stylesheet!",
)
private
def file_exists?(name)
if assets_precompiled?
File.exists?(local_file_path(name))
else
sprockets_asset(name)
end
end
memoize :file_exists?
# If on-the-fly asset compilation is disabled, we must be precompiling assets.
def assets_precompiled?
!Rails.configuration.assets.compile
rescue
false
end
def local_file_path(name)
asset_path = asset_path(name)
if asset_path.match(ABSOLUTE_ASSET_PATH_REGEXP)
asset_path.gsub!(ABSOLUTE_ASSET_PATH_REGEXP, "assets/")
end
File.join(Rails.public_path, asset_path)
end
memoize :local_file_path
add_method_tracer(
:local_file_path,
"UserAssetsProvider/local_file_path",
)
def sprockets_asset(name)
asset_path = asset_path(name)
if asset_path.match(ABSOLUTE_ASSET_PATH_REGEXP)
asset_path.gsub!(ABSOLUTE_ASSET_PATH_REGEXP, "")
end
# Strange thing is since rails 4.2
# name is passed in like
# `/assets/mailer-a9c96bd713d0b091297b82053ccd9155b933c00a53595812d755825d1747f42d.css`
# Before any processing
# And since `sprockets_asset` is used for preview
# We just "fix" the name by removing the
#
# Regexp taken from gem `asset_sync`
# https://github.com/AssetSync/asset_sync/blob/v1.2.1/lib/asset_sync/storage.rb#L142
#
# Modified to match what we need here (we need `.css` suffix)
if asset_path =~ /-[0-9a-fA-F]{32,}\.css$/
asset_path.gsub!(/-[0-9a-fA-F]{32,}\.css$/, ".css")
end
Rails.application.assets.find_asset(asset_path)
end
add_method_tracer(
:sprockets_asset,
"UserAssetsProvider/sprockets_asset",
)
def asset_path(name)
name.gsub(%r{^[/]?assets/}, "")
end
Contract String => String
def stylesheet_content(name)
if assets_precompiled?
File.read(local_file_path(name))
else
# This will compile and return the asset
sprockets_asset(name).to_s
end.strip
end
memoize :stylesheet_content
add_method_tracer(
:stylesheet_content,
"UserAssetsProvider/stylesheet_content",
)
end
end
end
I would like to report my own findings
With NewRelic data, we think most of the time is spent on
Roadies::Inliner/selector_elements => Roadie::Inliner/elements_matching_selector
And it seems a stylesheet with more style rules will make the style inlining takes longer
Benchmark code will be something like:
# frozen_string_literal: true
require "benchmark/ips"
class TestMailer < ::ActionMailer::Base
def show(benchmark_file_path:)
return mail(
from: "somewhere#test.com",
to: ["somewhere#test.com"],
subject: "some subject",
# This is trying to workaround a strange bug in `mail` gem
# https://github.com/mikel/mail/issues/912#issuecomment-156186383
content_type: "text/html",
) do |format|
format.html do
render(
file: benchmark_file_path,
layout: false,
)
end
end
end
end
Benchmark.ips do |x|
x.warmup = 5
x.time = 60
options = Roadie::Rails::Options.new(
# Use your own provider or use built-in providers
# I use a custom provider which can be used inside a rails app,
# See https://github.com/Mange/roadie for built-in providers
#
# options.asset_providers = [UserAssetsProvider.new]
# options.external_asset_providers = [UserAssetsProvider.new]
options.keep_uninlinable_css = false
)
# Need to prepare html_file yourself with
# different stylesheet tag pointing to two different stylesheet files
x.report("fat") do
message = ::TestMailer.
show(
benchmark_file_path: "benchmark-fat-stylesheet.html",
).message.tap do |m|
Roadie::Rails::MailInliner.new(m, options).execute
end
if message.body.to_s =~ /stylesheet/
raise "stylesheet not processed"
end
end
x.report("slim") do
message = ::TestMailer.
show(
benchmark_file_path: "benchmark-slim-stylesheet.html",
).message
if message.body.to_s =~ /stylesheet/
raise "stylesheet not processed"
end
end
# Compare the iterations per second of the various reports!
x.compare!
end
I've created a ruby script that executes fine if I run it from Console.
The script fetches some information from various websites and saves it to my database table.
However, when I want to turn the code into a rake task, the code still runs, but it does not save any new records. I don't get any errors from the rake either.
# Add your own tasks in files placed in lib/tasks ending in .rake,
# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.
require File.expand_path('../config/application', __FILE__)
Rails.application.load_tasks
require './crawler2.rb'
task :default => [:crawler]
task :crawler do
### ###
require 'rubygems'
require 'nokogiri'
require 'open-uri'
start = Time.now
$a = 0
sites = ["http://www.nytimes.com","http://www.news.com"]
for $a in 0..sites.size-1
url = sites[$a]
$i = 75
$error = 0
avoid_these_links = ["/tv", "//www.facebook.com/"]
doc = Nokogiri::HTML(open(url))
links = doc.css("a")
hrefs = links.map {|link| link.attribute('href').to_s}.uniq.sort.delete_if {|href| href.empty?}.delete_if {|href| avoid_these_links.any? { |w| href =~ /#{w}/ }}.delete_if {|href| href.size < 10 }
#puts hrefs.length
#puts hrefs
for $i in 0..hrefs.length
begin
#puts hrefs[60] #for debugging)
#file = open(url)
#doc = Nokogiri::HTML(file) do
if hrefs[$i].downcase().include? "http://"
doc = Nokogiri::HTML(open(hrefs[$i]))
else
doc = Nokogiri::HTML(open(url+hrefs[$i]))
end
image = doc.at('meta[property="og:image"]')['content']
title = doc.at('meta[property="og:title"]')['content']
article_url = doc.at('meta[property="og:url"]')['content']
description = doc.at('meta[property="og:description"]')['content']
category = doc.at('meta[name="keywords"]')['content']
newspaper_id = 1
puts "\n"
puts $i
#puts "Image: " + image
#puts "Title: " + title
#puts "Url: " + article_url
#puts "Description: " + description
puts "Catory: " + category
Article.create({
:headline => title,
:caption => description,
:thumbnail_url => image,
:category_id => 3,
:status => true,
:journalist_id => 2,
:newspaper_id => newspaper_id,
:from_crawler => true,
:description => description,
:original_url => article_url}) unless Article.exists?(original_url: article_url)
$i +=1
#puts $i #for debugging
rescue
#puts "Error here: " + url+hrefs[$i] if $i < hrefs.length
$i +=1 # do_something_* again, with the next i
$error +=1
end
end
puts "Page: " + url
puts "Articles: " + hrefs.length.to_s
puts "Errors: " + $error.to_s
$a +=1
end
finish = Time.now
diff = ((finish - start)/60).to_s
puts diff + " Minutes"
### ###
end
The code executes fine, if I save the file as crawler.rb and open it in Console by doing --> " load './crawler2.rb' ". When I use the exact same code in a rake task, I get no new records.
I figured out what was wrong.
I need to remove:
require './crawler2.rb'
task :default => [:crawler]
and instead edit the following:
task :crawler => :environment do
Now the crawler runs every ten minutes with a bit of help from Heroku scheduler :-)
Thanks for the help guys - and sorry for the bad formatting. Hope this answer may help others.
My ruby application uses about 97 %CPU which eventually gets killed. The program is reading files from the folder and if a file name exists in the database, it skips it and checks another file. While executing this procedure, application usually gets killed.
COMMAND %CPU
ruby 96.5
Even if I insert almost all files and try to lunch an application again (because it was killed), system tends to kill it even sooner. How can I decrease the %CPU?
task :process_data, [:data_directory] => :environment do |_task, args|
# add data to a database
saver = CsvToSqlSaver.new
saver.fill_files_names
Dir.foreach(args.data_directory) do |filename|
# if not present in records already we read it
Base.logger.info "> Found #{filename}."
next if saver.files_names.to_s.include?(filename) ||
!filename.include?('csv')
Base.logger.info "> Reading #{filename}."
begin
saver.generate_db_rows_from_csv_file(args.data_directory, filename)
# handle Malformed .csv exception
rescue CSV::ArgumentError, CSV::MalformedCSVError => e
Base.logger.info e.message
next
end # we continue csv file loop?
unless integrator.insert_data_to_database
Base.logger.info '> No new data saved.'
end
end
end
This is fill_files_names:
def fill_files_names
#files_names = []
files_names = MyFilesTable.select(:filename).distinct
files_names.each do |row|
#files_names.push(row[:filename])
end
end
This is Base:
class Base
class << self
attr_accessor :logger
end
#logger ||= Logger.new(STDERR)
end
This is generate_db_rows_from_csv_file
def generate_db_rows_from_csv_file(directory, filename)
#incoming_data = []
CSV.foreach("#{directory}/#{filename}",
headers: true, quote_char: "\x00") do |csv_record|
# if invalid record, go further
next if record_invalid?(csv_record)
generate_row_in_the_database(csv_record, filename)
end
end
My locale file has become unwieldy with a bunch of nested keys. Is there a way to get a list of all available locale keys, or all locale keys from a single locale file?
For eg.
en:
car:
honda:
civic:
name: 'Civic'
description: 'Entry Level Sedan'
ferrari:
la_ferrari:
name: 'La Ferrari'
description: 'Supercar'
This locale should return the list of keys, which in this case is
['en.car.honda.civic.name', 'en.car.honda.civic.description',
'en.ferrari.la_ferrari.name', 'en.car.ferrari.la_ferrari.name.description']
Is there a Rails (I18n) helper to do this?
The other way is to iterate over the parsed YAML.
To get an array of available locales:
I18n.available_locales
I recommend avoiding putting multiple locales in one YAML file. If you need to do so for some processing-related reason, you can always concatenate the files on the fly with, e.g., your *NIX shell:
...to a file
cat my_app/config/locales/*.yml >> locales.yml
...or to another processs
cat my_app/config/locales/*.yml | command_that_takes_stdin -
This is a script I've written when I had to deal with this. Working great for me.
#! /usr/bin/env ruby
require 'yaml'
filename = if ARGV.length == 1
ARGV[0]
elsif ARGV.length == 0
"/path/to/project/config/locales/new.yml"
end
unless filename
puts "Usage: flat_print.rb filename"
exit(1)
end
hash = YAML.load_file(filename)
hash = hash[hash.keys.first]
def recurse(obj, current_path = [], &block)
if obj.is_a?(String)
path = current_path.join('.')
yield [path, obj]
elsif obj.is_a?(Hash)
obj.each do |k, v|
recurse(v, current_path + [k], &block)
end
end
end
recurse(hash) do |path, value|
puts path
end
I do not pretend that this is a uniqe right solution, but this code works for me.
# config/initializers/i18n.rb
module I18n
class << self
def get_keys(hsh = nil, parent = nil, ary = [])
hsh = YAML.load_file("config/locales/en.yml") unless hsh
keys = hsh.keys
keys.each do |key|
if hsh.fetch(key).is_a?(Hash)
get_keys(hsh.fetch(key), "#{parent}.#{key}", ary)
else
keys.each do |another|
ary << "#{parent}.#{another}"[1..-1]
end
end
end
ary.uniq
end
end
end
Result
[14] pry(main)> I18n.get_keys
=> ["en.car.honda.civic.name", "en.car.honda.civic.description", "en.car.ferrari.la_ferrari.name", "en.car.ferrari.la_ferrari.description", "en.car.suzuki.escudo.name", "en.car.suzuki.escudo.description"]
My en.yml
en:
car:
honda:
civic:
name: 'Civic'
description: 'Entry Level Sedan'
ferrari:
la_ferrari:
name: 'La Ferrari'
description: 'Supercar'
suzuki:
escudo:
name: 'Escudo'
description: 'SUV'