Problem with large rails background process not completing - ruby-on-rails

Question

I have a rails background process (using Sidekiq and Redis) that parses XML files and later makes modifications to it.
The background process works as intended but stays in processing and does not complete when the XML is really big. My two assumptions for this are:
My background process is storing large amounts of text from the XML into arrays old_texts & new_texts and it is causing issues
My background process is timing out
The problem occurs on both my development machine (& staging).
I'm not sure how to debug this problem. I don't think posting my code will help but I'll do it in case you need an idea of what I'm doing:
old_texts, new_texts = [], []
xml_no_includs = ["pctHeight","pctWidth","posOffset","delText","delInstrText","instrText"]
search = '//w:document//w:body//w:p'
ancestors_excluds = ['//mc:Fallback', '//w:tbl', '//wps:txbx', '//v:textbox']
old_texts, new_texts = get_texts(old_texts, new_texts, XML, xml_no_includs, ancestors_excluds, search)
search_param = '//w:document//w:body//w:p'
ancestors_excluds = ['//mc:Fallback', '//w:tbl', '//wps:txbx', '//v:textbox']
replace_texts(old_texts, new_texts, XML, search_param, ancestors_excluds)
-
def replace_texts(old_texts, new_texts, XML, search_param, ancestors_excluds)
text_params = './/text()[not(ancestor::wp14:pctHeight or ancestor::wp14:pctWidth or ancestor::wp:posOffset or ancestor::w:instrText or ancestor::w:delText or ancestor::w:delInstrText)]'
inc = 0
old_texts.each_with_index do |old_text, index|
accum_string = ''
double_break = false
XML.search(search_param).drop(inc).each do |line|
inc += 1
temp = true
line.search(text_params).each do |p|
temp2 = true
ancestors_excluds.each do |param|
temp2 = false if p.ancestors(param).present?
end
if temp2 == true
if accum_string.blank? && !p.content.blank?
accum_string += p.content
p.content = new_texts[index]
else
accum_string += p.content unless accum_string.blank?
p.content = ''
end
if accum_string.strip == old_text.strip
double_break = true
break
end
end
end
break if double_break == true
end
end
end
-
def get_texts(old_texts, new_texts, XML, xml_no_includs, ancestors_excluds, search_param)
XML.xpath(search_param).each do |p|
text = ''
temp = true
p.search('text()').each do |p2|
temp2 = true
temp2 = false if xml_no_includs.include?(p2.parent.name)
ancestors_excluds.each do |param|
temp2 = false if p2.ancestors(param).present?
end
text += p2.text if temp2 == true
end
unless text.blank?
old_texts.append(text)
new_texts.append(text.gsub(/(.)./, '\1*') )
end
end
old_texts.reject!(&:blank?)
new_texts.reject!(&:blank?)
return old_texts, new_texts
end

Answer 1

I ended up refactoring my code to manage each element that was originally going to be pushed into the array. Didn't speed it up more, but at least the background task completes after a few hours.