Elasticsearch - No value specified for terms query - ruby-on-rails

Calling all elasticsearch experts:
Environment:
elasticsearch (6.8.3)
rails 5.1.7
Linux
We changed our association from our user model to the person model from a has_one to a has_many. We did not however remove the person_id column as this can cause other issues (e.g. running code from another branch that still tries to use it)
It seems no matter what we do to the index definitions for ElasticSearch, if any null values exist in person_id (which they do now with all newly added records) we get
"No value specified for terms query"
I have confirmed this by populating that column with an integer value resulting in no error occurring.
I have removed person_id from any indexes I even removed any reference to the peoples table in the ES index declaration but it appears that as soon as ES sees there is an association to a Person model and it sees the person_id column, it tries to use it.
Besides writing an arbitrary value in the person_id column, does anyone else have a better approach? We DO NEED to search fields in the associated Person model (peoples table) but since it's a has_many, it shouldn't be using the person_id column which is no longer relevant.
We also considered reversing the search from Person to User but that would require a LOT of recoding.
Is it possible a newer version of ES will fix it? We'd probably have to upgrade the ES server as well if we update the gem right?
Here is the association:
has_many :people, :class_name => Person.to_s # I believe it's this class name that's causting it to try to use the person_id column
Here is the index definition
def as_indexed_json(_options = {})
as_json(
:only => %i[id email portal_id],
:include => {
:portal => { :only => %i[name slug] },
:people => { :methods => %i[first last full_name], :only => %i[first last full_name] },
:events => { :only => [:id] },
:account_brokerage_firms => { :only => [:id] },
:brokerage_firms => { :only => [:id] },
:roles => { :methods => [:role_name], :only => [:role_name] }
}
)
end
settings AUTOCOMPLETE_SETTINGS do
mapping :dynamic => 'false' do
indexes :id, :type => 'long'
indexes :email, :type => 'text', :analyzer => 'email'
indexes :portal_id, :type => 'long'
indexes :people do
indexes :first, :type => 'text', :analyzer => 'autocomplete'
indexes :last, :type => 'text', :analyzer => 'autocomplete'
indexes :full_name, :type => 'text', :analyzer => 'autocomplete'
end
indexes :portal do
indexes :name, :type => 'text', :analyzer => 'autocomplete'
indexes :slug, :type => 'text', :analyzer => 'autocomplete'
end
indexes :events do
indexes :id, :type => 'long'
end
indexes :account_brokerage_firms do
indexes :id, :type => 'long'
end
indexes :brokerage_firms do
indexes :id, :type => 'long'
end
indexes :roles do
indexes :role_name, :type => 'text', :analyzer => 'autocomplete'
end
end
end
# end of elastic search settings
# this function assumes there is only one user role between a user and an account

Related

Tire gem: index a Rails parent/child relationship in Elasticsearch

Having a tough time wrapping my head around Tire's syntax, that of Elasticsearch and how they map together.
I have successfully indexed PDFs in a Rails app via Tire. But I need to break the full reports down into individual pages so queries can be more granular. It's easy enough to split the PDFs into individual pages and add them to a Page model that belongs_to the full Report model. What I'm struggling with is how to set up the mapping and where?!? I'd like to take advantage of Elasticsearch's Parent Field mapping so I can realize this ultimate goal.
Hoping someone can set me straight.
Report model (this is working for me if I index an entire PDF as the :attachment):
class Report < ActiveRecord::Base
include Tire::Model::Search
include Tire::Model::Callbacks
has_many :pages, :dependent => :destroy
attr_accessible :filename, :title
tire.mapping do
indexes :id, :type =>'integer'
indexes :title
indexes :attachment, :type => 'attachment',
:fields => {
:content_type => { :store => 'yes' },
:author => { :store => 'yes' },
:title => { :store => 'yes' },
:attachment => { :term_vector => 'with_positions_offsets', :store => 'yes' },
:date => { :store => 'yes' }
}
end
...
end
Page model:
class Page < ActiveRecord::Base
include Tire::Model::Search
include Tire::Model::Callbacks
belongs_to :report
attr_accessible :filename, :page_num,
tire.mapping do
indexes :id, :type => 'integer'
indexes :page_num, :type => 'integer'
indexes :report_id, :type => 'integer' ###<== how is this associated with _parent?
indexes :attachment, :type => 'attachment',
:fields => {
...
...
end
end

Thinking Sphinx order if attribute exists else ignore

If have multiple indexed models that have a verification level which need to be sorted by their verification level.
But I also have a couple of models without a verification level.
ThinkingSphinx.search #search.q, :order => 'class_crc ASC, #relevance DESC, verification_level DESC', :page => page, :limit => limit, :classes => #search.types_as_models
The problem is I can rank by verification_level on models that have that attribute but it errors on models without a verification_level
index ground_core,ground_delta,user_core,user_delta: sort-by attribute 'verification_level' not found
Example player define_index with verification_level
define_index do
indexes [first_name, last_name], :as => :name, :sortable => true
where "state = 1"
has created_at, updated_at, verification_level
set_property :delta => ThinkingSphinx::Deltas::ResqueDelta
end
Example User define_index without verification_level
define_index do
indexes [first_name, last_name], :as => :name, :sortable => true
has created_at, updated_at
set_property :delta => ThinkingSphinx::Deltas::ResqueDelta
end
Is there a way to sort when it is on a model and ignore when it isnt?
My solution was to fake a verification level on models which didn't require it by adding it in the define_index block
has '0', :as => :verification_level, :type => :integer

How do I set the default analyzer for elastic search with tire?

I have been experimenting with elasticsearch lately with ruby on rails. I am having trouble getting my data indexed so I can search for items with both plural, and non-plural keywords.
Tire will allow me to assign an analyzer per mapping attribute:
mapping do
indexes title, analyzer: 'snowball'
indexes body, analyzer: 'snowball'
end
Now, say I have a keyword in the title of 'tests'
if I do a search with the attribute in the query:
http://localhost:9200/myindex/mymapping/_search?q=title:test
It will work.
However, if I do a general search without specifying the attribute like so:
http://localhost:9200/myindex/mymapping/_search?q=test
It will not find the document.
How do I specify that I want the default analyzer to be 'snowball' so I don't have to specify the attribute I want to search on?
p.s. I am using the Tire Gem. So please answer as best as you can taking that into account.
It's possible to change default analyzer using index settings:
require 'rubygems'
require 'tire'
Tire.index 'articles' do
delete
create :settings => {
:index => {
:analysis => {
:analyzer => {
:default => {
:type => 'snowball'
}
}
}
}
},
:mappings => {
:article => {
:properties => {
:title => { :type => 'string', :analyzer => 'snowball'},
:body => { :type => 'string', :analyzer => 'snowball'}
}
}
}
store :title => 'Tests', :body => "Plural"
store :title => 'Test', :body => "Singular"
refresh
end
It's easiest to set this in the elasticsearch.yml settings file:
index.number_of_shards: 2
index.number_of_replicas: 0
index.analysis.analyzer.default.type: snowball

How to sort the field (in the template) depending on their weight?

I use thinking_sphinx for search, and i'm trying to get the output table in which fields are grouped according to weight(from set_property :field_weights)
This define_block
define_index do
indexes status, title, content, manager, note, start_date, end_date
has created_at, updated_at, parent_id, organization_id
has user_id, :as => :user_id, :type => :integer
has '7', :as => :model_order, :type => :integer
set_property :field_weights => {
:title => 1,
:start_date => 2,
:user_id => 7
}
set_property :delta => true
end
How to sort the field (in the template) depending on their weight?
I'm not sure if I understand the Q, but it seems that you are trying to read the configuration of the field_weights from the view...
I don't know if thinking_sphinx has a method to do that, but what I would do is save that configuration elsewere and use it where you need it...
class YourModel
FieldWeights = {
:title => 1,
:start_date => 2,
:user_id => 7
}
# more code here....
define_index do
indexes status, title, content, manager, note, start_date, end_date
has created_at, updated_at, parent_id, organization_id
has user_id, :as => :user_id, :type => :integer
has '7', :as => :model_order, :type => :integer
set_property :field_weights => YourModel::FieldWeights
set_property :delta => true
end
end
Then in your view you can use YourModel::FieldWeights to display the fields in order of their weights, for example:
YourModel::FieldWeight.invert.sort

Thinking Sphinx with Rails - Delta indexing seems to work fine for one model but not for the other

I have 2 models User and Discussion. I have defined the indices for the models as below:
For the User model:
define_index do
indexes email
indexes first_name
indexes last_name, :sortable => true
indexes groups(:name), :as => :group_names
has "IF(email_confirmed = true and status = 'approved', true, false)", :as => :approved_user, :type => :boolean
has "IF(email_confirmed = true and (status = 'approved' or status='blocked'), true, false)", :as => :approved_or_blocked_user, :type => :boolean
has points, :type => :integer
has created_at, :type => :datetime
has user(:id)
set_property :delta => true
end
For the Discussion model:
define_index do
indexes title
indexes description
indexes category(:title), :as => :category_title
indexes tags(:title), :as => :tag_title
has "IF(publish_to_blog = true AND sticky = false, true, false)", :as => :publish_to_main, :type => :boolean
has created_at
has updated_at, :type => :datetime
has recent_activity_at, :type => :datetime
has views_count, :type => :integer
has featured
has publish_to_blog
has sticky
set_property :delta => true
end
I have added a delta column to both tables as per the documentation. My problem is that delta indexing works only for the Discussion model and not for the User model. For ex: When I update the 'title' of a discussion, I can see the thinking sphinx is rotating the indices etc. (as is evident from the logs). But when I update the 'first_name' or the 'last_name' of a user, nothing happens.
The User model also has a has_many :through association through a model called GroupsUser. I have setup a after_save on the GroupsUser as follows:
def set_user_delta_flag
user.delta = true
user.save
end
Even this doesn't seem to trigger delta indexing on the User model. A similar setup for the Discussion model works perfectly! Can anyone tell me why this is happening?
Alright I figured it out. The problem was with the attribute definition:
has user(:id)
When I removed it from the definition and ran a rebuild, everything seemed to work perfectly.

Resources