After a process why Memory continuously stay up in Ruby? - ruby-on-rails

Question

I'm using Rails 4 with Ruby 2.1.0. I have a code which taking about 2 GB of memory to complete process, but problem is after finish process...why is memory usage still up?
I also assign nil to variable after used which save 300 MB but still large amount of memory using by Ruby process.
Here is sample code to reproduce the problem:
def download_data(params, options = {})
$file_name = ''
portfolio_names = Portfolio.get_names
building_names = Building.get_names
tenant_names = Tenant.get_names
meter_names = Meter.get_meter_names
#portfolio_info, #building_info, #tenant_info, #meter_id, #from_date, #to_date = params[:portfolio_name], params[:building_name], params[:tenant_name], params[:meter_id], params[:from_date], params[:to_date]
#from_date = DateTime.parse(#from_date).to_i
#to_date = DateTime.parse(#to_date).to_i
#building_f_name = building_names["#{#building_info}"]
unless params[:fields].nil?
#fields = params[:fields].split('_')
#columns = []
#fields.each do |f|
if f.length > 0
#columns << MeterData.new.show_real_value_of_meter_data_field_revert(f).to_sym
end
end
#columns << 'date_time'.to_sym
#columns << 'tenant'.to_sym
#columns << 'meter_id'.to_sym
else
#columns = [:id, :date_time, :w3pht, :whintvlrec, :w__3phmaxavgdmd, :tenant, :meter_id]
end
#### database fetch ###########
#meter_data = MeterData.find_by_sql("SELECT tenant, meter_id, SUM(van) as van, SUM(vbn) as vbn,
SUM(vcn) as vcn, SUM(ia) as ia, SUM(ib) as ib, SUM(ic) as ic,
SUM(w3pht) as w3pht, SUM(pf3pht) as pf3pht, SUM(f) as f,
SUM(whrec) as whrec, SUM(whtot) as whtot, SUM(varhrec) as varhrec,
SUM(varhtot) as varhtot, SUM(whintvlrec) as whintvlrec,
SUM(whintvldel) as whintvldel, SUM(w__phavg) as w__phavg,
SUM(var__3phavg) as var__3phavg, SUM(w_3phavg) as w_3phavg,
SUM(var_3phavg) as var_3phavg, SUM(phai) as phai, SUM(phbi) as phbi,
SUM(phci) as phci, SUM(w__3phmaxavgdmd) as w__3phmaxavgdmd,
MAX(w__3phmaxavgdmd) as w__3phmaxavgdmd_max, SUM(var__3phmaxavgdmd) as var__3phmaxavgdmd,
SUM(w_3phmaxavgdmd) as w_3phmaxavgdmd, SUM(var_3phmaxavgdmd) as var_3phmaxavgdmd,
date_time FROM `meter_data` WHERE (`date_time_i` >= #{#from_date} AND `date_time_i` <= #{#to_date}
AND building = '#{#building_info}') GROUP by tenant, meter_id, date_time
ORDER BY `meter_data`.`date_time_i` ASC")
sleep 0.5
#meter_sum = MeterData.find_by_sql("SELECT meter_id, SUM(van) as van, SUM(vbn) as vbn, SUM(vcn) as vcn,
SUM(ia) as ia, SUM(ib) as ib, SUM(ic) as ic, SUM(w3pht) as w3pht,
SUM(pf3pht) as pf3pht, SUM(f) as f, SUM(whrec) as whrec,
SUM(whtot) as whtot, SUM(varhrec) as varhrec, SUM(varhtot) as varhtot,
SUM(whintvlrec) as whintvlrec, SUM(whintvldel) as whintvldel,
SUM(w__phavg) as w__phavg, SUM(var__3phavg) as var__3phavg,
SUM(w_3phavg) as w_3phavg, SUM(var_3phavg) as var_3phavg,
SUM(phai) as phai, SUM(phbi) as phbi, SUM(phci) as phci,
SUM(w__3phmaxavgdmd) as w__3phmaxavgdmd, MAX(w__3phmaxavgdmd) as w__3phmaxavgdmd_max,
SUM(var__3phmaxavgdmd) as var__3phmaxavgdmd, SUM(w_3phmaxavgdmd) as w_3phmaxavgdmd,
SUM(var_3phmaxavgdmd) as var_3phmaxavgdmd, date_time FROM `meter_data`
WHERE (`date_time_i` >= #{#from_date} AND `date_time_i` <= #{#to_date} AND building = '#{#building_info}')
GROUP by meter_id ORDER BY `meter_data`.`date_time_i` ASC")
#meter_data_max = MeterData.find_by_sql("SELECT tenant, meter_id, MAX(w__3phmaxavgdmd) as w__3phmaxavgdmd,
date_time FROM `meter_data` WHERE (`date_time_i` >= #{#from_date} AND `date_time_i` <= #{#to_date}
AND building = '#{#building_info}') GROUP by tenant, meter_id
ORDER BY `meter_data`.`date_time_i` ASC")
sleep 0.5
#uniq_meter_id = MeterData.select(:meter_id).where("`date_time_i` >= #{#from_date} AND `date_time_i` <= #{#to_date} AND building = '#{#building_info}'").uniq(:meter_id)
#### database fetch ###########
p = Axlsx::Package.new
wb = p.workbook
wb.styles do |s|
styles_hash = AxlsxStylesHash.get s
wb.add_worksheet(:name => "Building data details") do |sheet|
$row_num = 0
meter_extra = MeterExtra.new
columns_ = []; #columns.each { |s| columns_ << "#{s}_".to_s }
sheet.add_row []
$row_num += 1
sheet.add_row ["Building report of #{#building_f_name} #{#from_date} to #{#to_date}"], :style => styles_hash[:heading_cell]
$row_num += 1
sheet.add_row []
$row_num += 1
sheet.add_row []
$row_num += 1
sheet.merge_cells("A2:Q2")
new_columns_ = []
new_columns_ << "Date & Time" if columns_.include?('date_time_')
new_columns_ << "Building" if columns_.include?('building_')
new_columns_ << "Tenant" if columns_.include?('tenant_')
new_columns_ << "Meter ID" if columns_.include?('meter_id_')
new_columns_ << "Meter Name" if columns_.include?('meter_id_')
new_columns_ << "van" if columns_.include?('van_')
new_columns_ << "vbn" if columns_.include?('vbn_')
new_columns_ << "vcn" if columns_.include?('vcn_')
new_columns_ << "ia" if columns_.include?('ia_')
new_columns_ << "ib" if columns_.include?('ib_')
new_columns_ << "ic" if columns_.include?('ic_')
new_columns_ << "w3pht" if columns_.include?('w3pht_')
new_columns_ << "pf3pht" if columns_.include?('pf3pht_')
new_columns_ << "f" if columns_.include?('f_')
new_columns_ << "whrec" if columns_.include?('whrec_')
new_columns_ << "whtot" if columns_.include?('whtot_')
new_columns_ << "varhrec" if columns_.include?('varhrec_')
new_columns_ << "varhtot" if columns_.include?('varhtot_')
new_columns_ << "whintvlrec" if columns_.include?('whintvlrec_')
new_columns_ << "whintvldel" if columns_.include?('whintvldel_')
new_columns_ << "w+phavg" if columns_.include?('w__phavg_')
new_columns_ << "var+3phavg" if columns_.include?('var__3phavg_')
new_columns_ << "w-3phavg" if columns_.include?('w_3phavg_')
new_columns_ << "var-3phavg" if columns_.include?('var_3phavg_')
new_columns_ << "phai" if columns_.include?('phai_')
new_columns_ << "phbi" if columns_.include?('phbi_')
new_columns_ << "phci" if columns_.include?('phci_')
new_columns_ << "w+3phmaxavgdmd" if columns_.include?('w__3phmaxavgdmd_')
new_columns_ << "var+3phmaxavgdmd" if columns_.include?('var__3phmaxavgdmd_')
new_columns_ << "w-3phmaxavgdmd" if columns_.include?('w_3phmaxavgdmd_')
new_columns_ << "var-3phmaxavgdmd" if columns_.include?('var_3phmaxavgdmd_')
sheet.add_row new_columns_, :style => styles_hash[:green_bold_border_cell]
$row_num += 1
t = 1; #meter_data.each do |m|
new_columns_data = []
if columns_.include?('date_time_')
m_date_time = m['date_time']
new_columns_data << ( m_date_time.nil? ? '--' : m_date_time)
end
if columns_.include?('building_')
m_building = m_building
new_columns_data << ( m_building.nil? ? '--' : building_names["#{m_building}"])
end
if columns_.include?('tenant_')
m_tenant = m['tenant']
new_columns_data << ( m_tenant.nil? ? '--' : tenant_names["#{m_tenant}"])
end
if columns_.include?('meter_id_')
m_meter_id = m['meter_id']
new_columns_data << ( m_meter_id.nil? ? '--' : m_meter_id)
end
if columns_.include?('meter_id_')
m_meter_id = m['meter_id']
new_columns_data << ( meter_extra.get_meter_name(m_meter_id.nil? ? '--' : m_meter_id) )
end
if columns_.include?('van_')
m_van = m.van
new_columns_data << ( m_van.nil? ? '--' : m_van )
end
if columns_.include?('vbn_')
m_vbn = m.vbn
new_columns_data << ( m_vbn.nil? ? '--' : m_vbn )
end
if columns_.include?('vcn_')
m_vcn = m.vcn
new_columns_data << ( m_vcn.nil? ? '--' : m_vcn )
end
if columns_.include?('ia_')
m_ia = m.ia
new_columns_data << ( m_ia.nil? ? '--' : m_ia )
end
if columns_.include?('ib_')
m_ib = m.ib
new_columns_data << ( m_ib.nil? ? '--' : m_ib )
end
if columns_.include?('ic_')
m_ic = m.ic
new_columns_data << ( m_ic.nil? ? '--' : m_ic )
end
if columns_.include?('w3pht_')
m_w3pht = m.w3pht
new_columns_data << ( m_w3pht.nil? ? '--' : m_w3pht )
end
if columns_.include?('pf3pht_')
m_pf3pht = m.pf3pht
new_columns_data << ( m_pf3pht.nil? ? '--' : m_pf3pht )
end
if columns_.include?('f_')
m_f = m.f
new_columns_data << ( m_f.nil? ? '--' : m_f )
end
if columns_.include?('whrec_')
m_whrec = m.whrec
new_columns_data << ( m_whrec.nil? ? '--' : m_whrec )
end
if columns_.include?('whtot_')
m_whtot = m.whtot
new_columns_data << ( m_whtot.nil? ? '--' : m_whtot )
end
if columns_.include?('varhrec_')
m_varhrec = m.varhrec
new_columns_data << ( m_varhrec.nil? ? '--' : m_varhrec )
end
if columns_.include?('varhtot_')
m_varhtot = m.varhtot
new_columns_data << ( m_varhtot.nil? ? '--' : m_varhtot )
end
if columns_.include?('whintvlrec_')
m_whintvlrec = m.whintvlrec
new_columns_data << ( m_whintvlrec.nil? ? '--' : m_whintvlrec )
end
if columns_.include?('whintvldel_')
m_whintvldel = m.whintvldel
new_columns_data << ( m_whintvldel.nil? ? '--' : m_whintvldel )
end
if columns_.include?('w__phavg_')
m_w__phavg = m.w__phavg
new_columns_data << ( m_w__phavg.nil? ? '--' : m_w__phavg )
end
if columns_.include?('var__3phavg_')
m_var__3phavg = m.var__3phavg
new_columns_data << ( m_var__3phavg.nil? ? '--' : m_var__3phavg )
end
if columns_.include?('w_3phavg_')
m_w_3phavg = m.w_3phavg
new_columns_data << ( m_w_3phavg.nil? ? '--' : m_w_3phavg )
end
if columns_.include?('var_3phavg_')
m_var_3phavg = m.var_3phavg
new_columns_data << ( m_var_3phavg.nil? ? '--' : m_var_3phavg )
end
if columns_.include?('phai_')
m_phai = m.phai
new_columns_data << ( m_phai.nil? ? '--' : m_phai )
end
if columns_.include?('phbi_')
m_phbi = m.phbi
new_columns_data << ( m_phbi.nil? ? '--' : m_phbi )
end
if columns_.include?('phci_')
m_phci = m.phci
new_columns_data << ( m_phci.nil? ? '--' : m_phci )
end
if columns_.include?('w__3phmaxavgdmd_')
m_w__3phmaxavgdmd = m.w__3phmaxavgdmd
new_columns_data << ( m_w__3phmaxavgdmd.nil? ? '--' : m_w__3phmaxavgdmd )
end
if columns_.include?('var__3phmaxavgdmd_')
m_var__3phmaxavgdmd = m.var__3phmaxavgdmd
new_columns_data << ( m_var__3phmaxavgdmd.nil? ? '--' : m_var__3phmaxavgdmd )
end
if columns_.include?('w_3phmaxavgdmd_')
m_w_3phmaxavgdmd = m.w_3phmaxavgdmd
new_columns_data << (m_w_3phmaxavgdmd.nil? ? '--' : m_w_3phmaxavgdmd )
end
if columns_.include?('var_3phmaxavgdmd_')
m_var_3phmaxavgdmd = m.var_3phmaxavgdmd
new_columns_data << ( m_var_3phmaxavgdmd.nil? ? '--' : m_var_3phmaxavgdmd )
end
if t == 1
sheet.add_row new_columns_data , :style => styles_hash[:simple_green_cell], :widths=>[20]
else
sheet.add_row new_columns_data, :style => styles_hash[:simple_white_cell], :widths=>[20]
end
$row_num += 1
t = 0 if t == 2
t += 1
puts $row_num
m = nil
end
#meter_data = nil
## logo ##
ReportBillLogo.put sheet, $row_num
sheet = nil;
end
## summary calculation ##
wb.add_worksheet(:name => "Summary") do |sheet|
$row_num = 0
sheet.add_row []
$row_num += 1
sheet.add_row ["Summary report of #{#building_f_name} #{#from_date} to #{#to_date}"], :style => styles_hash[:heading_cell]
$row_num += 1
sheet.add_row []
$row_num += 1
sheet.add_row []
$row_num += 1
sheet.merge_cells("A1:Q1")
columns_ = []; #columns.each { |s| columns_ << "#{s}_".to_s }
new_columns_ = []
new_columns_ << "Meter ID"
new_columns_ << "Meter Name"
new_columns_ << "Max KW"
sheet.add_row new_columns_, :style => styles_hash[:green_bold_border_cell]
$row_num += 1
t = 0
#meter_data_max.each do |m|
new_columns_data = []
new_columns_data << m.meter_id
new_columns_data << meter_names["#{m.meter_id}"]
new_columns_data << ( m.w__3phmaxavgdmd.nil? ? '' : m.w__3phmaxavgdmd )
if t == 1
sheet.add_row new_columns_data, :style => styles_hash[:simple_green_cell], :widths=>[25]
else
sheet.add_row new_columns_data, :style => styles_hash[:simple_white_cell], :widths=>[25]
end
$row_num += 1
t = 0 if t == 2
t += 1
m = nil;
end
## logo ##
ReportBillLogo.put sheet, $row_num
sheet = nil
end
## summary calculation ##
## summary calculation ##
wb.add_worksheet(:name => "Energy consumed") do |sheet|
$row_num = 0
sheet.add_row []
$row_num += 1
sheet.add_row ["Energy consumed report of #{#building_f_name} #{#from_date} to #{#to_date}"], :style => styles_hash[:heading_cell]
$row_num += 1
sheet.add_row []
$row_num += 1
sheet.add_row []
$row_num += 1
sheet.merge_cells("A2:Q2")
engery = []
#uniq_meter_id.each do |f|
start = MeterData.find_by_sql("select whtot from meter_data where `date_time_i` >= #{#from_date} AND meter_id = '#{f.meter_id}' ORDER BY date_time_i ASC limit 1")
eand = MeterData.find_by_sql("select whtot from meter_data where `date_time_i` <= #{#from_date} AND meter_id = '#{f.meter_id}' ORDER BY date_time_i DESC limit 1")
sleep 0.009
begin
engery << (eand.last.whtot.to_i - start.first.whtot.to_i)
rescue
engery << '--'
end
start = nil; eand = nil;
f = nil;
end
columns_ = []; #columns.each { |s| columns_ << "#{s}_".to_s }
new_columns_ = []
new_columns_ << "Meter ID"
new_columns_ << "Meter Name"
new_columns_ << "Subtraction result"
new_columns_ << "Aggregate result"
sheet.add_row new_columns_, :style => styles_hash[:green_bold_border_cell]
$row_num += 1
flage = 0
t = 0
#meter_sum.each do |m|
new_columns_data = []
new_columns_data << m.meter_id
new_columns_data << meter_names["#{m.meter_id}"]
new_columns_data << engery[flage]
new_columns_data << m.whintvlrec
if t == 1
sheet.add_row new_columns_data, :style => styles_hash[:simple_green_cell], :widths=>[25]
else
sheet.add_row new_columns_data, :style => styles_hash[:simple_white_cell], :widths=>[25]
end
$row_num += 1
flage += 1
t = 0 if t == 2
t += 1
m = nil
end
## logo ##
ReportBillLogo.put sheet, $row_num
sheet = nil;
end
end
$file_name = "tmp/#{#building_info} . #{#from_date} - #{#to_date} _ #{Time.new.to_i}.xlsx"
p.serialize($file_name)
p = nil; wb = nil;
#meter_data = nil; #meter_data_max = nil; #meter_sum = nil;
return $file_name
end
# total records = 5, 00, 000

Answer 1

When ruby process is running out of memory it access a huge bit of memory from the system called heap slab. This big chunk is then internally divided into many small bits, which holds your variables and your code. When you assign nil to some variable, garbage collector will mark this small bit as empty, but the whole slab is never returned to the system until process is terminated.

Answer 2

I'm not exactly sure if this is actually the problem that you're having, but there's a known issue with Ruby 2.1.x's generational garbage collection where allocated objects in memory that are supposed to be "short-lived" end up being accidentally promoted to "long-lived" instead, causing memory to bloat because the garbage collector doesn't run frequently enough to clean up these objects:
Koichi Sasada, the author of the generational GC in Ruby 2.1:
“Some ‘short-lived’ young objects will be promoted to ‘old-gen’ accidentally….if such ‘short-lived’ objects consume huge memory we need to free such objects.”
A current workaround that is being mentioned a lot right now in the Ruby community is to change a default Ruby setting in order to tell it to garbage collect more often:
Expect memory doubling with Ruby 2.1.1. Not happy with that? You have 2 options:
Tune it down by reducing RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR. At 1 your memory consumption will be on par with 2.0. It ships with the 2 default.
Option 2, wait for a future release of Ruby, this will be fixed in 2.2 maybe even patched a bit more in 2.1.2 . See:
https://bugs.ruby-lang.org/issues/9607 and
http://vimeo.com/89491942 and
https://speakerdeck.com/samsaffron/why-ruby-2-dot-1-excites-me