Best way to populate db with large ruby/json files? - ruby-on-rails

Question

Let's say I have a ruby or json file consisting of hashes and range anywhere from 14-20MB (300K lines unminified). I've created a rake task which loops through each hash and creates an AR object based on the values in each hash.
Unfortunately due to the file's size I get stack level too deep errors each time I run the task. The only way I've actually gotten the script to run is by splitting the file into smaller ones. Although this works, it gets extremely tedious to split the file and repeat the task over and over. Are there any good alternatives for loading/running large files?
Rake Task
namespace :db do
task populate: :environment do
$restaurants.each_with_index do |r, index|
uri = URI(r[:website])
restaurant = Restaurant.find_or_create_by(name: r[:name], website: "#{uri.scheme}://#{uri.host}")
restaurant.cuisines = r[:cuisines].map { |c| Cuisine.find_or_create_by(name: c) }
location = Location.create(
restaurant: restaurant,
city_id: 1,
address: r[:address],
latitude: r[:latitude],
longitude: r[:longitude],
phone_number: r[:phone_number]
)
r[:hours].each do |h|
Hour.create(
location: location,
day: Date::DAYNAMES.index(h[:day]),
opens: h[:opens],
closes: h[:closes]
)
end
menu_group = MenuGroup.create(
restaurant: restaurant,
locations: [location],
address: r[:address]
)
r[:menus].each do |m|
menu = Menu.create(
menu_group: menu_group,
position: m[:position],
name: m[:name]
)
m[:sections].each do |s|
section = Section.create(
menu: menu,
position: s[:position],
name: s[:name]
)
s[:dishes].each do |d|
tag = Tag.find_or_create_by(
name: d[:name].downcase.strip
)
Dish.find_or_create_by(
restaurant: restaurant,
sections: [section],
tags: [tag],
name: d[:name],
description: d[:description]
)
end
end
end
puts "#{index + 1} of #{$restaurants.size} completed"
end
end
end
Error
rake aborted!
SystemStackError: stack level too deep
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:12:in`to_binary'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:12:in`input_to_storage'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:37:in`fetch'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/compile_cache/iseq.rb:37:in`load_iseq'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:283:in `block in require'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:249:in `load_dependency'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:283:in `require'
/Users/user/app/lib/tasks/populate.rake:1:in `<main>'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `load'
/usr/local/lib/ruby/gems/2.5.0/gems/bootsnap-1.3.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `load'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:277:in `block in load'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:249:in `load_dependency'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies.rb:277:in `load'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:650:in `block in run_tasks_blocks'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:650:in `each'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:650:in `run_tasks_blocks'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/application.rb:515:in `run_tasks_blocks'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:459:in `load_tasks'
/Users/user/app/Rakefile:6:in `<top (required)>'
/usr/local/lib/ruby/gems/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
(See full trace by running task with --trace)

Answer 1

I would use something like Sidekiq to break the jobs in to workers that can run concurrently.
For example:
$restaurants.each_with_index do |r, index|
RestaurantParser.perform_async(r, index)
end
In RestaurantParser perform the steps you would normally take.
As long as restaurants don't rely on other restaurants already existing in the DB you can run workers concurrently to speed up the process.