I have an application in Rails that I run on Heroku (apprx 1 000 page views a day). I have been experiencing frequent crashes of the application since the launch last week.
Looking into New Relic it seems like the Dynos memory usage is constantly increasing without ever going down in memory usage. Basically, it builds up during a couple of hours and then end in request timeouts, which seems likely.
Thus, I believe the problem with the app crashing is due to a memory leak.
My app (presenttips . com) is a gift website where I have features like "random gift", "gift of the day" and "banners". These I load in the application controller like this:
before_filter :global_setup
def global_setup
# Create random gift
rand_gift = []
rand_gift << Gift.where(:gift_status_id => 1) #=> Accepted
#random_gift = rand_gift[0][rand(rand_gift[0].size) - 1]
rand_gift = nil
#nbr_of_active_gifts = (Gift.where(:gift_status_id => 1).count / 100 ).round * 100
#toplist = Gift.where(:gift_status_id => 1).order("week_click DESC").limit(20)
#banners = Banner.where("first_date <= '" + Time.now.to_date.to_s + "'").where("last_date >= '" + Time.now.to_date.to_s + "'").order("first_date ASC")
advertise_here = []
(#banners.count..4).each do |i|
advertise_here[i] = Banner.new(:advertiser => "Presenttips.com", :banner_image => "annons.jpg", :url => advertise_path)
end
#banners << advertise_here.compact
#banners = #banners.flatten
#page_categories = PageCategory.order(:prio_rank)
if Rails.env.production?
#random_sql = "RANDOM()"
#meta_robots_block = false
#analytics_block = false
else
#meta_robots_block = true
#analytics_block = true
#random_sql = "RAND()"
end
gift_from_daily = DailyGift.where(:publish_date => Time.now.to_date).first
gift_from_daily = DailyGift.create(:publish_date => Time.now.to_date, :gift_id => #random_gift.id) if gift_from_daily.blank?
#daily_gift = Gift.find(gift_from_daily.gift_id)
#head_categories = Category.order(:name).where(:parent_id => nil)
todays_date = Time.now.to_date.to_s
#season = Season.where("'" + todays_date + "' >= date_start ", "'" + todays_date + "' <= date_end" ).first
#season_theme = #season.css
#logo = 'logo.png'
#logo = 'seasons/logo_christmas.png' if #season.css.eql?('theme_christmas.css')
end
so that I can use them in the app globabally (gift of the day, for example, is always presenet in the right column).
I guess this is not great considering memory usage though.
My questions:
Is this likely to cause the memory build-up?
What would be a smarter way to do this, in that case?
I remove almost all of these variables and it still didn't help. I will assume the application controller was not causing the memory problem.
Related
Good afternoon!
I have a function where she does the bank reconciliation with a .txt of the bank with the system, where she shows the number of the launch whose value is exactly equal to the one launched in the system, the problem is that there is 1 debit in the .txt that corresponds to ha several launches in the system of the same client id, in .txt I do not have the client id but the sum of the values launched corresponds exactly to the debit, is there any way to do this query? ... I am trying to use the in the variable sum_lancamentos having for me returns the ids but is generating synthase error. and I believe that all this function could be improved, I just don’t know how, I’m a new RoR and I’m still getting used to good practices.
Any help is very life and thank you in advance!
def conciliacao
#conciliacao = session[:conciliacao_file]
comparacao = {}
#conciliacao.each do |key, line|
data = line[:data]
valor = line[:valor].to_f.round(2)
if line[:mov] == "D"
despesa = true
else
despesa = false
end
lancamentos = Lancamento.ativos.pagos.where(conta_id: params[:conta_id], despesa: despesa).where("lancamentos.data_pagamento BETWEEN '#{data.to_date.beginning_of_day.to_s(:db)}' AND '#{data.to_date.end_of_day.to_s(:db)}'").where(["cast(lancamentos.valor AS NUMERIC(15,2) ) = :value or
cast(lancamentos.valor_pago AS NUMERIC(15,2)) = :value ",
{ value: line[:valor] }])
unless lancamentos.blank?
lancamentos.each do |lancamento|
#puts line
#conciliacao[key][:lancamentos] = "#{lancamento.id}"
#conciliacao[key][:status] = 1
#conciliacao[key][:color] = "green lighten-4"
end
else
sum_lancamentos = Lancamento.ativos.pagos.group(:cliente_id).where(conta_id: params[:conta_id], despesa: despesa).where("lancamentos.data_pagamento BETWEEN '#{data.to_date.beginning_of_day.to_s(:db)}' AND '#{data.to_date.end_of_day.to_s(:db)}'").having(["sum(cast(lancamentos.valor AS NUMERIC(15,2) )) = :value or sum(cast(lancamentos.valor_pago AS NUMERIC(15,2))) = :value ", { value: line[:valor] }])
unless sumlancamentos.blank?
#conciliacao[key][:lancamentos] = "#{sum_lancamentos.ids}"
#conciliacao[key][:status] = 1
#conciliacao[key][:color] = "green lighten-4"
end
end
end
session.delete(:conciliacao_file)
end
I have a rails background process (using Sidekiq and Redis) that parses XML files and later makes modifications to it.
The background process works as intended but stays in processing and does not complete when the XML is really big. My two assumptions for this are:
My background process is storing large amounts of text from the XML into arrays old_texts & new_texts and it is causing issues
My background process is timing out
The problem occurs on both my development machine (& staging).
I'm not sure how to debug this problem. I don't think posting my code will help but I'll do it in case you need an idea of what I'm doing:
old_texts, new_texts = [], []
xml_no_includs = ["pctHeight","pctWidth","posOffset","delText","delInstrText","instrText"]
search = '//w:document//w:body//w:p'
ancestors_excluds = ['//mc:Fallback', '//w:tbl', '//wps:txbx', '//v:textbox']
old_texts, new_texts = get_texts(old_texts, new_texts, XML, xml_no_includs, ancestors_excluds, search)
search_param = '//w:document//w:body//w:p'
ancestors_excluds = ['//mc:Fallback', '//w:tbl', '//wps:txbx', '//v:textbox']
replace_texts(old_texts, new_texts, XML, search_param, ancestors_excluds)
-
def replace_texts(old_texts, new_texts, XML, search_param, ancestors_excluds)
text_params = './/text()[not(ancestor::wp14:pctHeight or ancestor::wp14:pctWidth or ancestor::wp:posOffset or ancestor::w:instrText or ancestor::w:delText or ancestor::w:delInstrText)]'
inc = 0
old_texts.each_with_index do |old_text, index|
accum_string = ''
double_break = false
XML.search(search_param).drop(inc).each do |line|
inc += 1
temp = true
line.search(text_params).each do |p|
temp2 = true
ancestors_excluds.each do |param|
temp2 = false if p.ancestors(param).present?
end
if temp2 == true
if accum_string.blank? && !p.content.blank?
accum_string += p.content
p.content = new_texts[index]
else
accum_string += p.content unless accum_string.blank?
p.content = ''
end
if accum_string.strip == old_text.strip
double_break = true
break
end
end
end
break if double_break == true
end
end
end
-
def get_texts(old_texts, new_texts, XML, xml_no_includs, ancestors_excluds, search_param)
XML.xpath(search_param).each do |p|
text = ''
temp = true
p.search('text()').each do |p2|
temp2 = true
temp2 = false if xml_no_includs.include?(p2.parent.name)
ancestors_excluds.each do |param|
temp2 = false if p2.ancestors(param).present?
end
text += p2.text if temp2 == true
end
unless text.blank?
old_texts.append(text)
new_texts.append(text.gsub(/(.)./, '\1*') )
end
end
old_texts.reject!(&:blank?)
new_texts.reject!(&:blank?)
return old_texts, new_texts
end
I ended up refactoring my code to manage each element that was originally going to be pushed into the array. Didn't speed it up more, but at least the background task completes after a few hours.
I'm iterating over a collection of updates (which include additions, updates and deletions). I'm currently looping out over these updates, checking to see whether it exists. If it does, updating the information. If it doesn't, creating a record.
This seems to be taking an unpleasurable amount of time. I'd say 1 every 2 seconds, maybe slightly faster. But I've got 60,000 left to process and it's going to take 109 hours (give or take) to complete. Which isn't acceptable.
Is there anyway to perform this more efficiently?
Update.all.each do |u|
product = Product.find(:all, :conditions => ["product_codes = ?", u.product_codes])
if u.update_type == "Delete"
daap = DeletedProduct.new
daap.product_codes = u.product_codes
daap.date_deleted = timestamp
daap.save
if product.count == 1
product.first.destroy
puts "Product #{ u.product_codes } deleted"
u.destroy
#delete = #delete + 1
elsif product.count > 1
product.each do |pro|
pro.destroy
puts "Product #{ u.product_codes } deleted (UPDATE #{ u.id})"
#delete = #delete + 1
end
u.destroy
else
#notfound.push(u.product_codes)
u.destroy
#delete = #delete + 1
end
elsif u.update_type == "AddOrUpdate"
if product.count == 1
p = Product.first
p.data = u.data
p.last_updated = timestamp
p.save
puts "Product Updated #{ p.product_codes }"
u.destroy
#update = #update + 1
else
p = Product.new
p.product_codes = u.product_codes
p.data = u.data
p.last_updated = timestamp
p.save
puts "Product Added #{ p.product_codes }"
u.destroy
#delete = #delete + 1
end
end
end
EDIT: It's the product lookup that takes so long. Is there anyway to speed up this part of the process? 'product = Product.find(...'
I'm working on allowing clients to view analytics per day, week, month, in a period of time, grouped by hours or days or months, etc... All of that is based on the created_at attribute.
Is there any gem out there that already does this? Something like:
Posts.analytics(:by => :day, :period => :this_week, :column => :created_at)
Would return:
{
'2012-06-19' => 14,
'2012-06-20' => 0, // Empty rows padding support*
'2012-06-21' => 3
}
I'm trying to make it from scratch but it seems like a lot of unecessary work if there's already a gem to do the job.
Update
I tried to make an analytics module that gets included into all models for easy analytics generation, But it's really unreliable, Sometimed i get more days than i need, and it's really messy, Could anyone collaborate and rewrite/improve on this:
# Usage:
# include Analytics::Timeline
# Model.timeline(:period => :last_24_hours, :time_by => :hour)
module Analytics
module Timeline
def self.included(base)
base.class_eval {
def self.timeline(*filters)
filters = filters[0]
period = filters[:period] || :this_week
time_by = filters[:time_by] || :days
date_column = filters[:date_column] || :created_at
# Named periods conventions
period_range = case period
when :last_12_hours
[Time.now-12.hours, Time.now]
when :last_24_hours
[Time.now-24.hours, Time.now]
when :last_7_days
[Time.now-7.days, Time.now]
when :last_30_days
[Time.now-30.days, Time.now]
when :this_week
[Time.now.beginning_of_week, Time.now.end_of_week]
when :past_week
[(Time.now - 1.week).beginning_of_week, (Time.now - 1.week).end_of_week]
when :this_month
[Time.now.beginning_of_month, Time.now.end_of_month]
when :past_month
[(Time.now-1.month).beginning_of_month, (Time.now - 1.month).end_of_month]
when :this_year
[Time.now.beginning_of_year, Time.now.end_of_year]
end
period_range = period if period.kind_of?(Array)
period_range = [period, Time.now] if period.is_a?(String)
# determine the SQL group method
group_column = case time_by
when :months
time_suffix = "-01 00:00:00"
records = where("#{table_name}.#{date_column} > ? AND #{table_name}.#{date_column} <= ?", period_range[0].to_date, period_range[1].to_date)
"DATE_FORMAT(#{table_name}.#{date_column.to_s}, '%Y-%m')"
when :days
time_suffix = " 00:00:00"
records = where("#{table_name}.#{date_column} > ? AND #{table_name}.#{date_column} <= ?", period_range[0].to_date, period_range[1].to_date)
"DATE(#{table_name}.#{date_column.to_s})"
when :hours
time_suffix = ":00:00"
records = where("#{table_name}.#{date_column} > ? AND #{table_name}.#{date_column} <= ?", period_range[0], period_range[1])
"DATE_FORMAT(#{table_name}.#{date_column.to_s}, '%Y-%m-%d %H')"
when :minutes
time_suffix = ":00"
records = where("#{table_name}.#{date_column} > ? AND #{table_name}.#{date_column} <= ?", period_range[0], period_range[1])
"DATE_FORMAT(#{table_name}.#{date_column.to_s}, '%Y-%m-%d %H:%M')"
end
# Get counts per cycle
records = records.group(group_column).select("*, count(*) AS series_count, #{group_column} AS series_time")
series = {}
# Generate placeholder series
time_table = { :days => 60*60*24, :hours => 60*60, :minutes => 60, :seconds => 0 }
if time_by == :months
ticks = 12 * (period_range[1].year - period_range[0].year) + (period_range[1].month + 1) - period_range[0].month
else
ticks = (period_range[1] - period_range[0] + 1) / time_table[time_by]
end
ticks.to_i.times do |i|
time = period_range[1]-i.send(time_by)
time = case time_by
when :minutes
time.change(:sec => 0)
when :hours
time.change(:min => 0)
when :days
time.change(:hour => 0)
when :months
time.change(:day => 1, :hour => 0)
end
series[time.to_s(:db)] = 0
end
# Merge real counts with placeholder series
to_merge = {}
records.each do |r|
to_merge[r.series_time.to_s+time_suffix] = r.series_count
end
series.merge!(to_merge)
end
}
end
end
end
The ActiveRecord statistics gem seems like it could be really useful to you.
If the statistics gem doesn't help, the admin_data gem has some analytics built in. Check out the demo. Use of the entire admin system might be overkill but you could at least try to browse the source to mimic the analytics feature.
I have a some code like this users_controller.rb, I need to print line no 30 in development.log
line#29 def selectrole
line#30 #userrole = RolesUser.find(:all, :conditions =>["p.user_id = ? and p.status = ? ",session[:user_id], params['status']])
line#31 logger.debug print_line(30)
line#32 end
Can I see 30th line in my development.log like this
#userrole = RoleUser.find(:all, :conditions => ["p.user_id = ? and p.status = ? ", 1234, 'Active'])
What is the approach to write "print_line" function? Here is the my print_line code?
def print_line(file_name, line)
counter = 1
printline = "-------- NO SUCH LINE --------"
File.open(file_name, "r") do |infile|
while (line_text = infile.gets)
if counter == line
printline = "#{counter} :: " + line_text
break
end
counter += 1
end
end
printline
end
from this function I am getting like this
#userrole = RolesUser.find(:all, :conditions =>["p.user_id = ? and p.status = ? ",session[:user_id], params['status']])
Is their any way to find and replace the variables with their respective values?
Assuming that you're primarily interested in knowing the contents of your :conditions, why not just do something like this:
def selectrole
conditions = ["p.user_id = ? and p.status = ? ",session[:user_id], params['status']]
logger.debug(conditions)
#userrole = RolesUser.find(:all, :conditions => conditions)
end