I'm using thinking_sphinx and am delta indexing a model.
The delta index works but there is small bug. When I create a new product it is index. However, when I update that product it is not getting index right away. I have to update or create a new product before that old updated product is indexed.
Not quite sure where to start.
My recommendation would be to use delayed_delta indexing instead of straight delta indexing (which can be slow and if you have a few updates in a few seconds, can cause you all kinds of problems).
It takes two steps:
Change your define_index block to have a set_property :delta => :delayed
Create a short script to make sure the delayed indexing jobs get run. Here's the one I use:
#!/usr/bin/env ruby
## this script is for making sure and delayed_jobs get run
## it is used by thinking sphinx
require File.dirname(__FILE__) + '/../config/environment'
# you can also put the definition of this in config/environments/*.rb so it's different for test, production and development
JobRunnerPidFile = "#{RAILS_ROOT}/tmp/pids/job_runner.pid"
if File.exists?(JobRunnerPidFile)
old_pid = File.read(JobRunnerPidFile).to_i
begin
if Process.getpgid(old_pid) > 0
# still running, let's exit silently...
exit(0)
end
rescue
# looks like nothing is running, so let's carry on
end
end
File.open(JobRunnerPidFile, "w") {|f| f.write "#{$$}\n" }
Delayed::Worker.new.start
You can run that script from cron every 5 minutes (it'll only run one instance) or if you have a monitoring service (e.g., monit) you can have it make sure it's running.
Make sure to restart that script when ever you deploy a new version of your code.
Related
I have an API which uses a Service, in which I have used Ruby thread to reduce the response time of the API. I have tried to share the context using the following example. It was working fine with Rails 4, ruby 2.2.1
Now, we have upgraded rails to 5.2.3 and ruby 2.6.5. After which service has stopped working. I can call the service from Console, it works fine. But with API call, service becomes unresponsive once it reaches CurrencyConverter.new. Any Idea what can be the issue?
class ParallelTest
def initialize
puts "Initialized"
end
def perform
# Our sample set of currencies
currencies = ['ARS','AUD','CAD','CNY','DEM','EUR','GBP','HKD','ILS','INR','USD','XAG','XAU']
# Create an array to keep track of threads
threads = []
currencies.each do |currency|
# Keep track of the child processes as you spawn them
threads << Thread.new do
puts currency
CurrencyConverter.new(currency).print
end
end
# Join on the child processes to allow them to finish
threads.each do |thread|
thread.join
end
{ success: true }
end
end
class CurrencyConverter
def initialize(params)
#curr = params
end
def print
puts #curr
end
end
If I remove the CurrencyConverter.new(currency), then everything works fine. CurrencyConverter is a service object that I have.
Found the Issue
Thanks to #anothermh for this link
https://guides.rubyonrails.org/threading_and_code_execution.html#wrapping-application-code
https://guides.rubyonrails.org/threading_and_code_execution.html#load-interlock
As per the blog, When one thread is performing an autoload by evaluating the class definition from the appropriate file, it is important no other thread encounters a reference to the partially-defined constant.
Only one thread may load or unload at a time, and to do either, it must wait until no other threads are running application code. If a thread is waiting to perform a load, it doesn't prevent other threads from loading (in fact, they'll cooperate, and each perform their queued load in turn, before all resuming running together).
This can be resolved by permitting concurrent loads.
https://guides.rubyonrails.org/threading_and_code_execution.html#permit-concurrent-loads
Rails.application.executor.wrap do
urls.each do |currency|
threads << Thread.new do
CurrencyConverter.new(currency)
puts currency
end
ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
threads.map(&:join)
end
end
end
Thank you everybody for your time, I appreciate.
Don't re-invent the wheel and use Sidekiq instead. 😉
From the project's page:
Simple, efficient background processing for Ruby.
Sidekiq uses threads to handle many jobs at the same time in the same process. It does not require Rails but will integrate tightly with Rails to make background processing dead simple.
With 400+ contributors, and 10k+ starts on Github, they have build a solid parallel job execution process that is production ready, and easy to setup.
Have a look at their Getting Started to see it by yourself.
I'm trying to understand how to use whenever properly, or if I'm even using it for the right thing. I've created a job:
class ScheduleSendNotificationsJob < ActiveJob::Base
queue_as :notification_emails
def perform(*args)
user_ids = User.
joins(:receipts).
where(receipts: {is_read: false}).
select('DISTINCT users.id').
map(&:id)
user_ids.each do |user_id|
SendNotificationsJob.create(id: user_id)
Rails.logger.info "Scheduled a job to send notifications to user #{user_id}"
end
end
end
I'd like to perform this job ever day at a set time. The job polls to see if there are any outstanding notifications, batches them, and then sends them to users so that a user can get one email with a bunch of notifications instead of a bunch of emails with one notification per email. I tried doing this with Delayed Job, but it seems it's not designed to schedule something on a recurring basis. So now I'm trying to do it with the whenever gem, but I can't seem to figure out how to set it up properly.
This is what I have in my config/schedule.rb file:
every 1.minute do
runner ScheduleSendNotifications.create
end
When I run whenever -i in the console I get the following:
Lorenzs-MacBook-Pro:Heartbeat-pods lorenzsell$ whenever -i
config/schedule.rb:13:in `block in initialize': uninitialized constant Whenever::JobList::ScheduleSendNotifications (NameError)
What am I doing wrong here? Should I be using something else? I'm just learning ruby and rails so any help is very much appreciated. Thank you.
The whenever gem takes a string as the argument to the runner function. Whenever doesn't actually load the Rails environment so it doesn't know about your ScheduleSendNotifications class.
The code below should get the crontab set up correctly to run your job.
every 1.minute do
runner "ScheduleSendNotifications.create"
end
From your project directory run whenever -w to set up the crontab file. Run crontab -l to view the written crontab file. Every minute the system will execute your Rails runner. From there you may need to debug your ScheduleSendNotifications.create code if something isn't working.
In my application, I need to execute a task every 5 minutes. The task itself is quite lean, so that's not the problem: the point is that loading the Rails environment uses almost all the CPU of the server. Right now, I'm using whenever with a code like this in my schedule.rb:
every 5.minutes do
runner Task.perform
end
But the cost is too big. Is there anyway to execute this kind of tasks in a preloaded environment? Or any other better solution?
Consider using a background process (many gems, notably delayed job) and implementing a simple "cron" in your code with a polling loop something like
def perform
last = Time.now
while true
if last.sec % (5 * 60) != 0
do_task
last = Time.now
end
end
end
Warning: this example has more holes than a slice of swiss cheese. Write something better :-)
You could use a cron job or set it up to run in a bash script. These would have the advantage of not being tied to your application.
I want to add a revision counter to my rails app.
Not the number of commits necessary but the number of live pushes/deployments for example.
I'm using github as my remote repo.
Any suggestions?
Thanks
There's not one magic solution.
But basically, you should execute some code every time you deploy your application that increments the number of deployments by one.
One solution would be to create a capistrano task which would increment this.
namespace :deploy do
desc "Increments the number of deployments"
task :increment do
Config.find_by_key('deployments').update('value = value + 1'
end
end
It will take the uplet "deployments" in a config database (which you have to implement, this way or an other).
And in your capistrano recipes, you add the following :
after "deploy", "deploy:increment"
Every time you deploy your application, the deployment value in the config model will be updated by one.
This is only one example of a possible implementation. You might want to store the number of deployments somewhere else.
The main idea is to have the code executed every time you deploy.
>rails -v
Rails 1.2.6
>ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32]
When I run a test fixture (that tests a rails model class) like this, it takes 20-30 secs to start executing these tests (show the "Loaded suite..."). What gives?
>ruby test\unit\category_test.rb
require File.dirname(__FILE__) + '/../test_helper'
class CategoryTest < Test::Unit::TestCase
def setup
Category.delete_all
end
def test_create
obCategoryEntry = Category.new({:name=>'Apparel'})
assert obCategoryEntry.save, obCategoryEntry.errors.full_messages.join(', ')
assert_equal 1, Category.count
assert_not_nil Category.find(:all, :conditions=>"name='Apparel'")
end
#.. 1 more test here
end
This one is Rails using a MySql DB with no fixtures. This time it clocked 30secs+ to startup.
Take a look at this Rails Test Server.
A quote from the author:
"Every time you run a test in a Rails
application, the whole environment is
loaded, including libraries that don’t
change between two consecutive runs.
That can take a considerable amount of
time. What if we could load the
environment once, and only reload the
changing parts before each run?
Introducing RailsTestServing.
With RailsTestServing, the run time of
a single test file has gone from 8
seconds down to .2 of a second on my
computer. That’s a x40 speed
improvement. Now, I don’t think twice
before hitting ⌘R in TextMate. It
feels liberating!"
(This was featured on the Rails Envy Podcast this past week which is where I found this.)
When starting any tests, Rails first loads any fixtures you have (in test/fixtures) and recreates the database with them.
20-30 seconds sounds very slow though. Do you have a lot of fixtures that need to be loaded before your tests run, or is your database running slow?
Ruby's gem tool follows a path discovery algorithm which, apparently, is not Windows (as I see from your ruby -v) friendly.
You can get a clear picture if you trace, for example, a Rails application loading with ProcMon. Every (I really mean every) require starts a scan over all directories in Ruby's path plus all gem directories. A typical require takes 20 ms on an average machine. Since Rails makes hundreds of requires, those 20 ms easily sum up to seconds every time you launch the Rails environment. Take in the time to initialize the fixtures in the database and you get a better idea of why it takes so much time to just begin running the test-cases.
Perhaps because of each file-system architecture and implementation (path caching etc.), this is less of a problem in Linux than in Windows. I don't know who you should blame, though. It looks like the NTFS file-system could be improved with a better path caching implementation, but clearly the gem tool could implement the caching itself and have its performance not so dependent on the platform.
It seems like Test::Unit is the simplest, but also one of the slowest ways to do unit testing with Ruby. One of alternatives is ZenTest.
Test unit startup isn't particularly slow, and nowhere near 20 seconds.
(11:39) ~/tmp $ cat test_unit.rb
require 'test/unit'
class MyTest < Test::Unit::TestCase
def test_test
assert_equal("this", "that")
end
end
(11:39) ~/tmp $ time ruby test_unit.rb
Loaded suite test_unit
Started
F
Finished in 0.007338 seconds.
1) Failure:
test_test(MyTest) [test_unit.rb:4]:
<"this"> expected but was
<"that">.
1 tests, 1 assertions, 1 failures, 0 errors
real 0m0.041s
user 0m0.027s
sys 0m0.012s
It's probably something you're doing in your tests. Are you doing anything complicated? Setting up a database? Retrieving something from the internet?
Complete shot in the dark, but the majority of the time I see long startup times on things, it is usually due to some sort of reverse DNS lookup happening with some TCP socket communication somewhere along the way.
Try adding:
require 'socket'
Socket.do_not_reverse_lookup = true
at the top of your test file after your other require line.
What does your test_helper.rb look like? Are you using instantiated fixtures?
self.use_instantiated_fixtures = true
[edit]
If this is set to true try setting it to false.