Testing whether a Ruby object is in fact waiting - ruby-on-rails

I have a system, comprised of a BoundedQueue class, a Producer class that pushes items into a BoundedQueue object and a Consumer class that takes items out of the BoundedQueue object, the Producer and Consumer exist on separate threads. So that items aren't lost when they're pushed to a full queue I use a condition variable and a mutex to tell the Producer to wait until the queue has a space.
I need to create a test case that checks that the Producer is waiting when the queue is full, I'm not sure if I'm being ditsy or not but I just can't think of how to do this properly.
BoundedQueue class:
class BoundedQueue
attr_reader :count
def initialize(size)
#mutex = Mutex.new
#rep = Array.new(size) if size > 0
#size = size
#back = size-1
#front = 0
#count = 0
#condvar = ConditionVariable.new
end
def isEmpty?
#count == 0
end
def isFull?
#count == #size
end
def put(item)
#mutex.synchronize do
if item != nil
while isFull?
#condvar.wait(#mutex)
end
#back += 1
#back = 0 if #back >= #size
#rep[#back] = item
#count += 1
end
end
end
def get
#mutex.synchronize do
result = nil
if (!isEmpty?)
result = #rep[#front]
#rep[#front] = nil
#front += 1
#front = 0 if #front >= #size
#count -= 1
#condvar.signal
end
result
end
end
end
Producer class:
class Producer
def initialize(id, no_items, queue)
#id = "Producer#{id}"
#no_items = no_items
#queue = queue
end
def produce
while #no_items != 0
#queue.put("Item #{#no_items} from #{#id} ")
puts "#{#id} putting item #{#no_items} into the queue "
#no_items -= 1
end
end
end
Consumer class:
class Consumer
def initialize(queue)
#queue = queue
end
def consume
while !#queue.isEmpty?
puts "#{#queue.get} consumed from the queue"
sleep(Random.rand(10))
end
end
end
The test case so far:
require "./boundedQueue.rb"
require "./producer.rb"
require "./consumer.rb"
mutex = Mutex.new
cv = ConditionVariable.new
test_queue = BoundedQueue.new(5)
puts "Creating the producer"
producerOne = Producer.new(0,7,test_queue)
puts "Creating consumer"
consumer = Consumer.new(test_queue)
puts "Creating threads"
a = Thread.new{
producerOne.produce
}
b = Thread.new{
consumer.consume
}
puts "Joining threads"
b.join
a.join

Related

How to implement a Redis Store for Money/Open-Exchange-Rate-Bank Ruby on Rails

I spent days searching the whole internet and couldn't find any implementation. I, therefore, implemented one and will like to share.
class RedisRateStore
INDEX_KEY_SEPARATOR = '_TO_'.freeze
# Using second db of the redis instance
# because sidekiq uses the first db
REDIS_DATABASE = 1
# Using Hash to store rates data
REDIS_STORE_KEY = 'rates'
def initialize
conn_url = "#{Rails.application.credentials.redis_server}/#{REDIS_DATABASE}"
#connection = Redis.new(url: conn_url)
end
def add_rate(iso_from, iso_to, rate)
#connection.hset(REDIS_STORE_KEY, rate_key_for(iso_from, iso_to), rate)
end
def get_rate(iso_from, iso_to)
#connection.hget(REDIS_STORE_KEY, rate_key_for(iso_from, iso_to))
end
def each_rate
rates = #connection.hgetall(REDIS_STORE_KEY)
return to_enum(:each_rate) unless block_given?
rates.each do |key, rate|
iso_from, iso_to = key.split(INDEX_KEY_SEPARATOR)
yield iso_from, iso_to, rate
end
end
def transaction
yield
end
private
def rate_key_for(iso_from, iso_to)
[iso_from, iso_to].join(INDEX_KEY_SEPARATOR).upcase
end
end
# config/initializers/open-exchange-rate.rb
# frozen_string_literal: true
require 'money/bank/open_exchange_rates_bank'
Rails.application.config.to_prepare do
oxr = Money::Bank::OpenExchangeRatesBank.new(RedisRateStore.new)
oxr.app_id = Rails.application.credentials.oxr_app_id
oxr.cache = 'db/rates.json'
oxr.ttl_in_seconds = 3600
oxr.prettyprint = false
Money.default_bank = oxr
end

Scraping data in rails using thread

I am doing scraping to fetch the data from the website to my database in rails.I am fetching the 32000 record with this script there isn't any issue but i want to fetch the data faster so i apply the thread in my rake task but then there is a issue while running the rake task some of the data is fetching then the rake task getting aborted.
I am not aware of what to do task if any help can be done i am really grateful . Here is my rake task code for the scraping.
task scratch_to_database: :environment do
time2 = Time.now
puts "Current Time : " + time2.inspect
client = Mechanize.new
giftcard_types=Giftcard.card_types
find_all_merchant=Merchant.all.pluck(:id, :name).to_h
#first index page of the merchant
index_page = client.get('https://www.twitter.com//')
document_page_index = Nokogiri::HTML::Document.parse(index_page.body)
#set all merchant is deteled true
# set_merchant_as_deleted = Merchant.update_all(is_deleted: true) if Merchant.exists?
# set_giftcard_as_deleted = Giftcard.update_all(is_deleted: true) if Giftcard.exists?
update_all_merchant_record = []
update_all_giftcard_record = []
threads = []
#Merchant inner page pagination loop
page_no_merchant = document_page_index.css('.pagination.pagination-centered ul li:nth-last-child(2) a').text.to_i
1.upto(page_no_merchant) do |page_number|
threads << Thread.new do
client.get("https://www.twitter.com/buy-gift-cards?page=#{page_number}") do |page|
document = Nokogiri::HTML::Document.parse(page.body)
#Generate the name of the merchant and image of the merchant loop
document.css('.product-source').each do |item|
merchant_name= item.children.css('.name').text.gsub("Gift Cards", "")
href = item.css('a').first.attr('href')
image_url=item.children.css('.img img').attr('data-src').text.strip
#image url to parse the url of the image
image_url=URI.parse(image_url)
#saving the record of the merchant
# #merchant=Merchant.create(name: merchant_name , image_url:image_url)
if find_all_merchant.has_value?(merchant_name)
puts "this if"
merchant_id=find_all_merchant.key(merchant_name)
puts merchant_id
else
#merchant= Merchant.create(name: merchant_name , image_url:image_url)
update_all_merchant_record << #merchant.id
merchant_id=#merchant.id
end
# #merchant.update_attribute(:is_deleted, false)
#set all giftcard is deteled true
# set_giftcard_as_deleted = Giftcard.where(merchant_id: #merchant.id).update_all(is_deleted: true) if Giftcard.where(merchant_id: #merchant.id).exists?
#first page of the giftcard details page
first_page = client.get("https://www.twitter.com#{href}")
document_page = Nokogiri::HTML::Document.parse(first_page.body)
page_no = document_page.css('.pagination.pagination-centered ul li:nth-last-child(2) a').text.to_i
hrefextra =document_page.css('.dropdown-menu li a').last.attr('href')
#generate the giftcard details loop with the pagination
# update_all_record = []
find_all_giftcard=Giftcard.where(merchant_id:merchant_id).pluck(:row_id)
puts merchant_name
# puts find_all_giftcard.inspect
card_page = client.get("https://www.twitter.com#{hrefextra}")
document_page = Nokogiri::HTML::Document.parse(card_page.body)
#table details to generate the details of the giftcard with price ,per_off and final value of the giftcard
document_page.xpath('//table/tbody/tr[#class="toggle-details"]').collect do |row|
type1=[]
row_id = row.attr("id").to_i
row.at("td[2] ul").children.each do |typeli|
type = typeli.text.strip if typeli.text.strip.length != 0
type1 << type if typeli.text.strip.length != 0
end
value = row.at('td[3]').text.strip
value = value.to_s.tr('$', '').to_f
per_discount = row.at('td[4]').text.strip
per_discount = per_discount.to_s.tr('%', '').to_f
final_price = row.at('td[5] strong').text.strip
final_price = final_price.to_s.tr('$', '').to_f
type1.each do |type|
if find_all_giftcard.include?(row_id)
update_all_giftcard_record<<row_id
puts "exists"
else
puts "new"
#giftcard= Giftcard.create(card_type: giftcard_types.values_at(type.to_sym)[0], card_value:value, per_off:per_discount, card_price: final_price, merchant_id: merchant_id , row_id: row_id )
update_all_giftcard_record << #giftcard.row_id
end
end
#saving the record of the giftcard
# #giftcard=Giftcard.create(card_type:1, card_value:value, per_off:per_discount, card_price: final_price, merchant_id: #merchant.id , gift_card_type: type1)
end
# Giftcard.where(:id =>update_all_record).update_all(:is_deleted => false)
#delete all giftcard which is not present
# giftcard_deleted = Giftcard.where(:is_deleted => true,:merchant_id => #merchant.id).destroy_all if Giftcard.where(merchant_id: #merchant.id).exists?
time2 = Time.now
puts "Current Time : " + time2.inspect
end
end
end
end
threads.each(&:join)
puts "-------"
puts threads
# merchant_deleted = Merchant.where(:is_deleted => true).destroy_all if Merchant.exists?
merchant_deleted = Merchant.where('id NOT IN (?)',update_all_merchant_record).destroy_all if Merchant.exists?
giftcard_deleted = Giftcard.where('row_id NOT IN (?)',update_all_giftcard_record).destroy_all if Giftcard.exists?
end
end
Error i am receiving:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use
Each thread requires a separate connection to your database. You need to increase the connection pool size that your application can use in your database.yml file.
But your database should also be capable of handling the incoming connections. If you are using mysql you can check this by running select ##MAX_CONNECTIONS on your console.

Having trouble with how classes work

I'm testing this code for a class assignment to make sure it works.
I don't quite understand code yet. Maybe give me some pointers.
require 'minitest/autorun'
require './human'
require './coffee'
## This assignment made more sense when I wrote it in the morning.
class CaffeineTest < MiniTest::Test
def test_humans_tend_to_be_sleepy
tyler = Human.new "Tyler"
assert tyler.alertness < 0.1
end
def test_humans_need_coffee
randy = Human.new "Randy"
refute randy.has_coffee?
assert randy.needs_coffee?
end
def test_humans_can_drink_coffee
sherri = Human.new "Sherri"
tsmf = Coffee.new "Triple Shot Mocha Frappuccino"
assert tsmf.full?
sherri.buy tsmf
sherri.drink!
assert_in_epsilon sherri.alertness, 0.33, 0.1
refute tsmf.full?
refute tsmf.empty?
end
def test_humans_can_drink_all_the_coffee
trevor = Human.new "Trevor"
tsmf = Coffee.new "Triple Shot Mocha Frappuccino"
trevor.buy tsmf
3.times { trevor.drink! }
assert tsmf.empty?
assert trevor.alertness > 0.9
end
end
Here's what I've added to it.
class Human
def initialize(name)
#name = name
#alertness = 0.0
#flavor = Coffee.new(name)
#coffee = 0
end
def alertness
#alertness
end
def has_coffee?()
false
end
def needs_coffee?()
true
end
def drink!()
#alertness += 0.33
#coffee -= 0.33
end
def empty?
if #coffee <= 0
true
else
false
end
end
def buy(type)
#flavor
#coffee += 1 #add tsmf to some kind of array or some shit
end
end
class Coffee
def initialize(drink)
#coffee = 0
end
def full?
if #coffee >= 0
true
else
false
end
end
def has_coffee?
false
end
end
I keep getting failures on empty? and full?. Please guide me a little here.
require 'minitest/autorun'
## This assignment made more sense when I wrote it in the morning.
class CaffeineTest < MiniTest::Test
def test_humans_tend_to_be_sleepy
tyler = Human.new "Tyler"
assert tyler.alertness < 0.1
end
def test_humans_need_coffee
randy = Human.new "Randy"
refute randy.has_coffee?
assert randy.needs_coffee?
end
def test_humans_can_drink_coffee
sherri = Human.new "Sherri"
tsmf = Coffee.new "Triple Shot Mocha Frappuccino"
assert tsmf.full?
sherri.buy tsmf
sherri.drink!
assert_in_epsilon sherri.alertness, 0.33, 0.1
refute sherri.empty?
end
def test_humans_can_drink_all_the_coffee
trevor = Human.new "Trevor"
tsmf = Coffee.new "Triple Shot Mocha Frappuccino"
trevor.buy tsmf
3.times { trevor.drink! }
assert tsmf.empty?
assert trevor.alertness > 0.9
end
end
class Human
def initialize(name)
#name = name
#alertness = 0.0
#flavor = Coffee.new(name)
#coffee = 0
end
def alertness
#alertness
end
def has_coffee?()
false
end
def needs_coffee?()
true
end
def drink!()
#alertness += 0.33
#coffee -= 0.33
end
def empty?
if #coffee <= 0
true
else
false
end
end
def buy(type)
#flavor
#coffee += 1 #add tsmf to some kind of array or some shit
end
end
class Coffee
def initialize(drink)
#coffee = 0
end
def full?
if #coffee >= 0
true
else
false
end
end
def has_coffee?
false
end
def empty?
if #coffee <= 0
true
else
false
end
end
end

Storing associations in variables

Hey I am trying to make a method polymorphic so that it can access different models associations, but treat them the same
module Unitable
# Adds new units without duplication
def add_units(type, units = {})
if type.downcase == "army"
relationships = self.members
elsif type.downcase == "character"
relationships = self.owns
end
units.each do |name, amount|
unit = Unit.find_by(name: name)
if relationships.where(unit_id: unit.id).first.nil?
relationships.create(unit_id: unit.id, amount: amount) #<--- This is where the error occurs
else
relationship = relationships.find_by(unit_id: unit.id)
original_amount = relationship.amount
new_amount = amount + original_amount
relationship.update_attribute(:amount, new_amount)
end
end
end
end
This throws
ERROR: current transaction is aborted, commands ignored until end of transaction block
This is the root method
def gen_starting_units
credit = 9
#transaction do
units = Unit.tagged_with(["#{race}", "1"]).sample(2)
while credit > 0
unit = units.sample
if unit.cost <= credit
self.army.add_units("army", unit.name => 1)
credit -= unit.cost
end
end
#end
end

A ruby script and a rails model: Same code - different behaviour

I have coded the following ruby script:
require 'open-uri'
require 'Nokogiri'
require 'anemone'
class JobFox
attr_accessor :company_url,
:jobs_page,
:max_words,
:jobs_part,
:jobs_container,
:element_score,
:max_score,
:jobs
def calc_element_score(element)
self.element_score += (element['class'].to_s.scan(/job|career|position|opening/).count + element['id'].to_s.scan(/job|career|position|opening/).count) * 100
self.element_score += element.to_s.scan(/job|career|position|opening/).count * 5
element.css('a').each do |a|
self.element_score += a.to_s.scan(/job|career|position|opening/).count * 7
end
element.css('li').each do |li|
self.element_score += li.to_s.scan(/job|career|position|opening/).count * 5
end
element.css('h').each do |h|
self.element_score += h.to_s.scan(/job|career|position|opening/).count * 3
end
if self.element_score > self.max_score
self.max_score = self.element_score
self.jobs_part = element
end
if element.children.count == 0
self.element_score = 0
end
end
end
fox = JobFox.new
fox.company_url = 'http://www.website.com'
fox.max_words = 0
fox.jobs = []
# CRAWL THE WEBSITE TO FIND THE JOBS LINK
Anemone.crawl(fox.company_url, :depth_limit => 3) do |anemone|
anemone.on_pages_like(/job|jobs|career|careers|team|about/) do |page|
begin
puts "SCANNING: " + page.url.to_s
# SCAN THE HTML AND FIND THE OCCURENCES OF THE WORD "JOB"
source_html = open(page.url).read
job_occurences = source_html.scan(/job|jobs|work|position/).count
# IF MORE OCCURENCES THAN BEFORE, WE KEEP THE PAGE URL
if job_occurences > fox.max_words
fox.max_words = job_occurences
fox.jobs_page = page.url
end
rescue Exception => e
puts e
end
end
end
fox.jobs_container = Nokogiri::HTML(open(fox.jobs_page))
fox.element_score = fox.max_score = 0
fox.jobs_container.css('div, section').each do |container|
container.traverse do |element|
fox.calc_element_score(element)
end
end
fox.jobs_part.traverse do |element|
element.css('a').each do |job|
fox.jobs << job.text
end
end
# REMOVE POSSIBLE DUPLICATE ENTRIES
fox.jobs = fox.jobs.uniq
puts fox.jobs
and I am trying to port it to a rails application - not as a script/task but as a model function:
require 'anemone'
require 'open-uri'
require 'Nokogiri'
class Company < ActiveRecord::Base
has_many :jobs
accepts_nested_attributes_for :jobs
# CALCULATE THE RELATEDNESS OF EACH HTML ELEMENT
def calculate_element_score(element)
#jobs_expression = '/job|career|position|opening/'
#element_score += (element['class'].to_s.scan(#jobs_expression).count + element['id'].to_s.scan(#jobs_expression).count) * 100
#element_score += element.to_s.scan(#jobs_expression).count * 5
element.css('a').each do |a|
#element_score += a.to_s.scan(#jobs_expression).count * 7
end
element.css('li').each do |li|
#element_score += li.to_s.scan(#jobs_expression).count * 5
end
element.css('h').each do |h|
#element_score += h.to_s.scan(#jobs_expression).count * 3
end
if #element_score > #max_score
#max_score = #element_score
#jobs_part = element
end
if element.children.count == 0
#element_score = 0
end
end
# CRAWL THE WEBSITE TO FIND THE JOBS PAGE
def find_jobs_page
max_words = 0
Anemone.crawl(self.website, :depth_limit => 3) do |anemone|
anemone.on_pages_like(/job|jobs|career|careers|team|about/) do |page|
begin
# SCAN THE HTML AND FIND OCCURENCES OF RELEVANT WORDS
source_html = open(page.url).read
job_occurences = source_html.scan(/job|jobs|work|position/).count
# IF MORE OCCURENCES THAN BEFORE, KEEP THE PAGE URL
if job_occurences > max_words
max_words = job_occurences
self.jobs_page = page.url
end
rescue Exception => e
puts e
end
end
end
end
# FIND THE CONTAINER THAT HAS THE JOB LISTINGS
def find_jobs_container
jobs_container = Nokogiri::HTML(open(self.jobs_page))
#element_score = #max_score = 0
#jobs_expression = '/job|career|position|opening/'
jobs_container.css('div, section').each do |container|
container.traverse do |element|
self.calculate_element_score(element)
end
end
end
# ADD THE JOBS FROM THE PAGE TO THE COMPANY ASSOCIATION
def extract_jobs
#jobs_part.traverse do |element|
element.css('a').each do |job|
j = JOBS.new()
j.title = job.text
j.url = job
self.jobs << j
end
end
end
# THE METHOD TO FIND ALL THE JOBS FOR A COMPANY
def find_jobs
self.find_jobs_page
self.find_jobs_container
self.extract_jobs
end
end
Everything works just fine apart from the calculate_element_score method - #elements_score is always 0. Have I understood something entirely wrong regarding global variables?

Resources