I'm using delta indexing for my Thinking Sphinx indexes in my Rails project. In my machine (Mac OS X) it's working fine. I change a record and it immediately finds it. On the servers (Debian) it doesn't.
I did run a sql query for delta = true and the I've got the expected recently changed records, so that part is working. In the log/searchd.query.log I see the proper request:
[Fri Oct 22 10:25:29.193 2010] 0.000 sec [all/3/rel 0 (0,20)] [customer_core,customer_delta] Jonas4
Any ideas what else could it be?
Thanks.
I'll answer here, even though you've posted to the support list as well...
Which user is running the TS rake tasks? And which user owns the Rails site on your server? They should be the same.
Also: are you using Passenger? If so, you'll want to make sure the bin_path setting is set in your config/sphinx.yml file. The documentation runs through both points.
Let me know if neither of these things help matters.
Related
I have a Rails 5.2 app that's configured to use the new much touted feature of recyclable cache keys.
I can confirm the setting is enabled in the console:
Rails.application.config.active_record.cache_versioning
=> true
ActiveRecord::Base.cache_versioning
=> true
BlogPost.cache_versioning
=> true
With this setting, blog_post.cache_key now returns a stable string, because the cache_version is actually stored inside the cache entry (as this article details):
blog_post.cache_key
=> "blog_posts/10317"
blog_post.cache_version
=> "20190417193345000000"
But the problem is, even tough everything works as expected in the console, I can't seem to see this working watching the server logs, because it keeps generating cache_keys that contain the cache_version:
In my view:
<% cache(['blog_post_list_item_v2', blog_post, I18n.locale, browser.device.mobile?]) do %>
...
<% end %>
In the server logs:
Rendered blog/blog_posts/_blog_post_list_item.html.erb (2.5ms) [cache miss]
Read fragment views/blog/blog_posts/_blog_post_list_item:0bdff42a9193ea497e5ed4a9cc2f51e8/blog_post_list_item_v2/blog_posts/10317-20190417193345000000/pt-br/ (0.5ms)
As you see, the cache key should be .../blog_posts/10317/, but it actually contains the timestamp.
After debugging through the Rails code, I could confirm that the key was actually stable. What gets printed in the server log includes the version for debugging purposes only, but the key being stored on your cache doesn't actually contain the version.
The version is stored instead within the serialized object in the cache, which is an instance of ActiveSupport::Cache::Entry and contains an attr_reader :version. So, if you're like me, you'd assume that the cache (for instance, raw HTML) was stored directly in memcached, but it actually is stored in the value attribute of that ActiveSupport::Cache::Entry (which also has the version attribute if you have cache_versioning turned on), and that entire object is saved serialized into the cache.
If you want to confirm it yourself, you can check your own memcached realtime log. If you're on a mac, first stop it (I'm assuming it was installed with homebrew) with brew services stop memcached, start it on the foreground with verbose mode with memcached -vv and take a look at the keys requested by rails. After you finish your study, brew services start memcached will re-enable memcached as a daemon.
Also, if you are migrating from the old way (without recyclable cache keys), you should wipe your cache first with Rails.cache.clear in the console. Remember to do that in production as well.
If you want to understand more about how this works, a good read is https://dzone.com/articles/cache-invalidation-complexity-rails-52-and-dalli-c, but debugging through the rails code with binding.pry was what got things clear to me.
In a nutshell, it's a very brilliant implementation in my opinion, and the cache recycling just makes it so much better (the article quotes DHH saying that 'We went from only being able to keep 18 hours of caching to, I believe, 3 weeks. It was the single biggest performance boost that Basecamp 3 has ever seen.')
We are experiencing a strange problem on a Rails application on Heroku. Juste after migrate from Rails 3.2.17 to Rails 4.0.3 our postgresql server show an infinite increase of memory usage, then it returns the following error on every request :
ERROR: out of memory
DETAIL: Failed on request of size xxx
Juste after releasing the application with rails 4, postgresql memory start to increase.
As you can see in the screenshot below, It increase from 500 MO to more than 3,5Go in 3 hours
Simultaneously, commit per second doubled. It passed from 120 commit per second :
to 280 commit per second :
It is worth noting that when we restart the application, memory go down to a normal value of 600 Mo before going up to more than 3 Go few hours later (then every sql request show the 'out of memory' error). It is like if killing ActiveRecord connections were releasing memory on postgresql server.
We may well have a memory leak somewhere.
However :
It was working very well with Rails 3.2. Maybe this problem is a conjunction between changes we made to adapt our code to Rails 4 and Rails 4 code itself.
Ihe increase of the number of commit per second juste after Rails 4 upgrade seems very odd.
Our stack is :
Heroku, x2 dynos
Postgresql, Ika plan on heroku
Unicorn, 3 workers per instance
Rails 4.0.3
Redis Cache.
Noteworthy Gems : Delayed jobs (4.0.0), Active Admin (on master branch), Comfortable Mexican Sofa (1.11.2)
Nothing seems really fancy in our code.
Our postgresql config is :
work_mem : 100MB
shared_buffers : 1464MB
max_connections : 500
maintenance_work_mem : 64MB
Did someone ever experienced such a behaviour when switching to Rails 4 ? I am looking for idea to reproduce as well.
All help is very welcome.
Thanks in advance.
I don't know what is better : answer my question or update it ... so I choose to answer. Please let me know if it's better to update
We finally find out the problem. Since version 3.1, Rails added prepared statements on simple request like User.find(id). Version 4.0, added prepared statements to requests on associations (has_many, belongs_to, has_one).
For exemple following code :
class User
has_many :adresses
end
user.addresses
generate request
SELECT "addresses".* FROM "addresses" WHERE "addresses"."user_id" = $1 [["user_id", 1]]
The problem is that Rails only add prepared statement variables for foreign keys (here user_id). If you use custom sql request like
user.addresses.where("moved_at < ?", Time.now - 3.month)
it will not add a variable to the prepared statements for moved_at. So it generate a prepared statements every time the request is called. Rails handle prepared statements with a pool of max size 1000.
However, postgresql prepared statements are not shared across connection, so in one or two hours each connection has 1000 prepared statements. Some of them are very big. This lead to very high memory consumption on postgreqsl server.
I’m using Rails4 with postgres as my database – because I’m using quite a lot of postgres–specific features, my schema is stored in .sql format (I’m not sure if it changes anything, but just in case…)
Currently, when I’m running tests, my console output is flooded with following logs:
SET
Time: 2.822 ms
CREATE EXTENSION
Time: 11.300 ms
(...)
COMMENT
CREATE TABLE
Time: 1.532 ms
(...)
INSERT 0 1
Time: 0.326 ms
I traced which task adds them, and they are created in db:structure:load task.
Is there any way to suppress them?
I added min_messages: warning to my database.yml config, but unfortunately it didn’t change anything.
Ok, I found it – the root of that cause was slightly different.
Couple of days ago, I stumbled upon Thoughtbots’ post about improving postgres command–line experience – there is link to nice psqlrc settings, which I (after cosmetic modifications) applied on my machine. Apparently, the very last line (\unset QUIET) was causing this. After commenting out that, it works like a charm. Thank you all for your help!
I would very much like to use the mongo geoNear command as discussed here.
Here is the command I entered in my rails console with the accompanying error message.
MongoMapper.database.command({ 'geoNear' => "trips", 'near' => [45,45]})
Mongo::OperationFailure: Database command 'geoNear' failed:
(errmsg: 'more than 1 geo indexes :('; ok: '0.0').
I can not make sense of the error message, it is supposedly impossible to have more than 1 geo index and I am certain that I have only created one.
Based on this stackoverflow question I believe I am wording the query correctly. Does anyone understand that error message? How would I go about destroying and recreating my indexes?
I am using rails 3.1 with mongodb v2.0 and the mongo ruby gem v1.5.1.
I really asked this too soon, maybe I should delete it? Somehow there were in fact too many geo indexes, because deleting the index and recreating it fixed the problem.
MongoMapper.database.collection('trips').drop_indexes
Trip.ensure_index [[:route, '2d']]
I am having a problem with Thinking sphinx on webfaction -- I have no probelem when I do it locally on osx.
I search
>> ThinkingSphinx.search 'raymond'
and my response is this:
=> []
any ideas?
thx
sg
If you have not already, you should run rake ts:index
If you would like your search to update every time you make a change, I would suggest looking into Delta Indexes also.
Are you using AwesomePrint by any chance?
I am and that is what was causing my problem.
You can see the bug report here:
https://github.com/freelancing-god/thinking-sphinx/issues/257