What makes heroku response time too much slow - ruby-on-rails

Question

I am making a rails application to crawl the flight information from specific website. This app can be found here https://vemaybay.herokuapp.com/.
It only took around 4-5 seconds to response locally, but it took 15-20 seconds when running on heroku.
Is there anyway to speed up this response time?
I have already changed the free to hobby dyno type to avoid DB spin-up costs but I believe DB connection and query is not the root cause.
Is it related to the hosting problem? So can think about buying a host.
Below is my example code:
FlightService
def crawl(from, to, date)
return if flight_not_available?(from, to)
begin
selected_day = date.day - 1
browser = ::Ferrum::Browser.new
browser.headers.set({ "User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" })
browser.goto("https://www.abay.vn/")
browser.at_css("input#cphMain_ctl00_btnSearch").click
browser.back
browser.execute("document.getElementById('cphMain_ctl00_txtFrom').setAttribute('value','#{from}')")
browser.execute("document.getElementById('cphMain_ctl00_txtTo').setAttribute('value','#{to}')")
browser.execute("document.getElementById('cphMain_ctl00_cboDepartureDay').selectedIndex = #{selected_day}")
browser.at_css("input#cphMain_ctl00_btnSearch").click
# browser.execute("document.querySelectorAll('a.linkViewFlightDetail').forEach(btn=> btn.click())")
sleep(1)
body = Nokogiri::HTML(browser.body)
flight_numbers = body.css("table.f-result > tbody > tr.i-result > td.f-number").map(&:text)
depart_times = body.css("table.f-result > tbody > tr.i-result > td.f-time").map { |i| i.text.split(" - ").first }
arrival_times = body.css("table.f-result > tbody > tr.i-result > td.f-time").map { |i| i.text.split(" - ").second }
base_prices = body.css("table.f-result > tbody > tr.i-result > td.f-price").map(&:text)
prices = base_prices
store_flight(flight_numbers, from, to, date, depart_times, arrival_times, base_prices, prices)
browser.quit
rescue StandardError => e
Rails.logger.error e.message
fail_with_message(e.message)
browser.quit
end
end
Then in my controller i just call the crawl method to fetch data:
service = FlightService.new(from: #from, to: #to, departure_date: #departure_date, return_date: #return_date)
service.crawl_go_flights
#go_flights = service.go_flights

Answer 1

I would try to add NewRelic Heroku add-on, it will show you what takes the most time, most likely it will be your Ruby code doing HTTP requests in a controller action to crawl a page.
Heroku tends to be slower than running code on your own development machine because Heroku resources are shared across users unless you bought expensive M/L dynos.
Without you sharing the code for crawling we don't know much how it work and where is the bottleneck. Do you crawl the single page or many pages (then this might be slow).
You can try moving crawl logic to the background worker, for instance, use Sidekiq gem. You could crawl page from time to time and store results in your DB then your controller action would only ask for data from your DB instead of crawling page every time. You can also use a rake task every 10 minutes defined in Heroku Scheduler to crawl page instead of Sidekiq (this might be faster to do). I don't know if having data up to date every 10 minutes is good enough for your use case. You need to pick a tech solution for your business use case needs. With Sidekiq you could run jobs more often by starting them every 1 minute using clockwork gem.