NoMemoryError: failed to allocate memory - Ruby on Rails - ruby-on-rails

Question

I generated a simple script that on other occasions has worked for me, but this is because the amount of information in the loop generates a NoMemoryError error.
I have 16 GB of memory and also a lot of virtual memory available. When I perform the test, the RAM memory is completely filled.
The script is:
require 'rest-client'
require 'json'
require 'open-uri'
require 'csv'
def self.qos7705egressdiscard_summary
xml = File.read("#{Rails.root}/public/Discard_qos7705egress.xml")
data = RestClient.post("http://10.140.255.1:8080/xmlapi/invoke", xml,{:"Content-Type" => 'application/soap+xml'})
data_parsed = Hash.from_xml(data)
return data_parsed
end
def self.samqos7705egressdiscardtotal_table
tabletotal = Hash.new
data_stats = qos7705egressdiscard_summary['Envelope']['Body']['findResponse']['result']['service.SapEgrQosQueueStatsLogRecord']
data_stats.map do |qosdiscard_device|
#devicetotal = qosdiscard_device["monitoredObjectSiteName"]
#servicetotal = qosdiscard_device["monitoredObjectPointer"]
#porttotal = qosdiscard_device["displayedName"]
#queueIdtotal = qosdiscard_device["queueId"]
#discardinproftotal = qosdiscard_device["droppedInProfOctets"].to_i
#discardoutproftotal = qosdiscard_device["droppedOutProfOctets"].to_i
time_unixtotal = (qosdiscard_device["timeCaptured"]).to_i/1000
#timeCapturedtotal = Time.at(time_unixtotal).strftime("%B %e, %Y at %I:%M %p")
#discardtotal = #discardinproftotal + #discardoutproftotal
#device_int_stats_total = (#devicetotal+#porttotal+#queueIdtotal).to_s
hash = Hash[devicetotal: #devicetotal, servicetotal: #servicetotal, porttotal: #porttotal, queueIdtotal: #queueIdtotal, discardtotal: #discardtotal, device_int_stats_total: #device_int_stats_total, timeCapturedtotal: #timeCapturedtotal, time_unixtotal: time_unixtotal]
tabletotal << hash
#tabletotal.write(hash)
end
end
The exact error is:
NoMemoryError: failed to allocate memory
from /usr/local/lib/ruby/2.2.0/irb/inspector.rb:108:in `inspect'
from /usr/local/lib/ruby/2.2.0/irb/inspector.rb:108:in `block in <module:IRB>'
from /usr/local/lib/ruby/2.2.0/irb/inspector.rb:101:in `call'
from /usr/local/lib/ruby/2.2.0/irb/inspector.rb:101:in `inspect_value'
from /usr/local/lib/ruby/2.2.0/irb/context.rb:383:in `inspect_last_value'
from /usr/local/lib/ruby/2.2.0/irb.rb:661:in `output_value'
from /usr/local/lib/ruby/2.2.0/irb.rb:490:in `block (2 levels) in eval_input'
from /usr/local/lib/ruby/2.2.0/irb.rb:623:in `signal_status'
from /usr/local/lib/ruby/2.2.0/irb.rb:486:in `block in eval_input'
from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:245:in `block (2 levels) in each_top_level_statement'
from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:231:in `loop'
from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:231:in `block in each_top_level_statement'
from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:230:in `catch'
from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:230:in `each_top_level_statement'
from /usr/local/lib/ruby/2.2.0/irb.rb:485:in `eval_input'
from /usr/local/lib/ruby/2.2.0/irb.rb:395:in `block in start'
from /usr/local/lib/ruby/2.2.0/irb.rb:394:in `catch'
from /usr/local/lib/ruby/2.2.0/irb.rb:394:in `start'
from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands/console.rb:110:in `start'
from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands/console.rb:9:in `start'
from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands/commands_tasks.rb:68:in `console'
from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands.rb:17:in `<top (required)>'
from bin/rails:4:in `require'
from bin/rails:4:in `<main>'Maybe IRB bug!
On line 25 I added this tabletotal.write (hash) to be able to write it on disk and not in memory but I got the following error:
NoMethodError: undefined method `write' for {}:Hash
What is the problem here? Also how can I fix it?

Answer 1

Start by making it less cr*p:
def extract_data(input)
{
devicetotal: input["monitoredObjectSiteName"],
servicetotal: input["monitoredObjectPointer"],
porttotal: input["displayedName"],
queueIdtotal: input["queueId"],
discardinproftotal: input["droppedInProfOctets"].to_i,
discardoutproftotal: input["droppedOutProfOctets"].to_i,
time_unixtotal: input["timeCaptured"].to_i/1000
}.tap do |h|
h[:timeCapturedtotal] = Time.at(h[:time_unixtotal]).strftime("%B %e, %Y at %I:%M %p"),
h[:discardtotal] = h[:discardinproftotal] + h[:discardoutproftotal]
h[:device_int_stats_total] =(h[:devicetotal]+h[:porttotal]+h[:queueIdtotal]).to_s
end
end
This method is really easy to test since you just insert input and write assertions about the output.
If you want to map and apply this to the input array you would do:
data_stats.map(&:extract_data)
Your code tries to output a hash but uses the shovel operator like on a array. You need to decide if the appropiate output is an array or a hash.
I need all those variables because I use them in a html in table
format.
This won't work - the instance variables will only contain the values from the last element as they get overwritten in each iteration.
You instead need to iterate over an array of hashes or objects in the view:
<% #stats.each do |s| %>
<tr>
<td><%= s[:devicetotal] %></td>
# ...
</tr>
<% end %>
If you really want to use instance varibles you need to create a object instance for each element:
class DataStat
include ActiveModel::AttributeAssignment
attr_accessor :devicetotal, :servicetotal # ...
def self.from_import(input)
self.new(
devicetotal: input["monitoredObjectSiteName"],
servicetotal: input["monitoredObjectPointer"],
# ...
)
end
end
#stats = data_stats.map { |ds| DataStat.from_import(ds) }
You also need to deal with the issue that you´re running out of memory since you are just slop converting the whole XML document into a Ruby hash. You need to parse it with Nokogiri instead and extract what you actually need.