getting errors when scrapping data with kimurai ruby on rails - ruby-on-rails

Question

FATAL -- web_scrapper_spider: Spider: stopped: {:spider_name=>"web_scrapper_spider", :status=>:failed, :error=>"#<NameError: uninitialized constant URI::HTTP\n\n raise InvalidUrlError, "Requested url is invalid: #{url}" unless URI.parse(url).kind_of?(URI::HTTP)\n >", :environment=>"development", :start_time=>2022-11-17 19:03:32.418101772 +0530, :stop_time=>2022-11-17 19:03:32.420534612 +0530, :running_time=>"0s", :visits=>{:requests=>0, :responses=>0}, :items=>{:sent=>0, :processed=>0}, :events=>{:requests_errors=>{}, :drop_items_errors=>{}, :custom=>{}}}
~/.rvm/gems/ruby-3.1.2/gems/kimurai-1.4.0/lib/kimurai/base.rb:194:in `request_to': uninitialized constant URI::HTTP (NameError)
raise InvalidUrlError, "Requested url is invalid: #{url}" unless URI.parse(url).kind_of?(URI::HTTP)
^^^^^^
from ~/.rvm/gems/ruby-3.1.2/gems/kimurai-1.4.0/lib/kimurai/base.rb:128:in `block in crawl!'
require 'open-uri'
require 'nokogiri'
require 'kimurai'
class WebScrapper < Kimurai::Base
#name = "web_scrapper_spider"
#engine = :mechanize
#start_urls = ["https://metaruby.com/"]
#config = {
user_agent: "Chrome/68.0.3440.84"
}
def parse(response, url:, data: {})
blogs = []
response.xpath("//table[#class='topic-list']//tbody//tr").each do |tr|
scrapped_data = {
title: tr.at('td[1]//span').text,
category: tr.at('td[1]//div//span').text,
date: tr.at('td[3]').text.strip
}
blogs << scrapped_data
save_to "results.json", scrapped_data.as_json, format: :json
end
end
end

Answer 1

From the logs posted in the question, it is clear that you are using Ruby version 3.1.2.
The gem Kimura that you are using is not maintained since 2020 and hence it is not compatible with Ruby 3.x due to the separation of positional and keyword arguments from 3.0.
Refer to Kimura issue #66 for more details and updates on the reported issue.
For now, you can use the gem Kimura with earlier Ruby versions.
I tried your code with Ruby 2.x and found no issues with the code.