Elasticsearch nested query for auto-complete - ruby-on-rails
I want to make sure I'm thinking about this in the right way. I'm trying to use Elasticsearch for an auto-complete for nested items. I have a list, and a list has many items. I want to use ES to return matching item names, weighing them more strongly if they're present in the current list by passing both the list name, and item name to search in Elasticsearch.
I could in theory simply index Items separately and search them that way, but I'd rather search them through the List document so that I can control relevancy.
I can't figure out how to return an item that nearly matches what's put in. Here's my setup... (Rails using Elasticsearch-Rails and Elasticsearch-Model)
The Index Mappings:
settings :index => { :number_of_shards => 1 } do
mapping :dynamic => 'false' do
indexes :private, :type => 'boolean'
indexes :name, :type => 'string'
indexes :slug, :type => 'string'
indexes :bookmarks_count , :type => 'integer'
indexes :item_names, :type => 'string'
indexes :up_count, :type => 'integer'
indexes :permalink, :type => 'string'
indexes :sub_text, :type => 'string'
indexes :title_text, :type => 'string'
indexes :creator_name, :type => 'string'
indexes :creator_avatar, :type => 'string'
indexes :cover_image, :type => 'string'
indexes :items, :type => "nested" do
indexes :name
indexes :description
indexes :image_url
indexes :link
end
end
end
The JSON:
def as_indexed_json(options = {})
as_json(:include => [:items, :tags],
:methods => [:permalink, :sub_text, :title_text, :icon_url, :item_names, :creator_name, :creator_avatar, :cover_image]
)
end
Here's the method I call for the search:
def item_typeahead_search(list_name, search_query, page = 1, per = 5)
wildcarded_query = "*#{search_query}*"
::List.search(item_typeahead_querystring(list_name, wildcarded_query)).per(per).page(page)
end
def item_typeahead_querystring(list_name, query_string)
{
:query => {
:bool => {
:should => [
{ :match => { :name => list_name }},
{
:nested => {
:path => "items",
:score_mode => "max",
:query => {
:bool => {
:must => [
{ :match => { "items.name" => query_string }}
]
}}}}
]
}}}
end
Here's a sample query...
results = List.item_typeahead_search("try me", "crazy glue")
Here are the results...
=> #<Elasticsearch::Model::Response::Result:0x007fb8f25c5208
#result=
{"_index"=>"lists",
"_type"=>"list",
"_id"=>"54504855f29a589a2700003b",
"_score"=>8.652843,
"_source"=>
{"_id"=>"54504855f29a589a2700003b",
"bookmarks_count"=>0,
"carousel"=>nil,
"cid"=>"5j18j8aor",
"content_source_name"=>nil,
"content_source_url"=>nil,
"created_at"=>"2014-10-29T01:52:21Z",
"description"=>nil,
"down_count"=>0,
"down_voters"=>[],
"intralist_id"=>"545057f7692a270e03000266",
"items"=>
[{"_id"=>"545048d5f29a589a27000044",
"created_at"=>"2014-10-29T01:54:29Z",
"description"=>nil,
"down_count"=>0,
"down_voters"=>[],
"image_small_url"=>nil,
"image_thumb_url"=>nil,
"image_url"=>nil,
"link"=>nil,
"name"=>"a swiss army knife",
"order"=>nil,
"picture"=>nil,
"up_count"=>0,
"up_voters"=>[],
"updated_at"=>"2014-10-29T01:54:29Z",
"vote_count"=>0},
{"_id"=>"545048d5f29a589a27000045",
"created_at"=>"2014-10-29T01:54:29Z",
"description"=>nil,
"down_count"=>0,
"down_voters"=>[],
"image_small_url"=>nil,
"image_thumb_url"=>nil,
"image_url"=>nil,
"link"=>nil,
"name"=>"duct tape",
"order"=>nil,
"picture"=>nil,
"up_count"=>0,
"up_voters"=>[],
"updated_at"=>"2014-10-29T01:54:29Z",
"vote_count"=>0},
{"_id"=>"545048d5f29a589a27000046",
"created_at"=>"2014-10-29T01:54:29Z",
"description"=>nil,
"down_count"=>0,
"down_voters"=>[],
"image_small_url"=>nil,
"image_thumb_url"=>nil,
"image_url"=>nil,
"link"=>nil,
"name"=>"Crazy glue",
"order"=>nil,
"picture"=>nil,
"up_count"=>0,
"up_voters"=>[],
"updated_at"=>"2014-10-29T01:54:29Z",
"vote_count"=>0},
{"_id"=>"545048d5f29a589a27000047",
"created_at"=>"2014-10-29T01:54:29Z",
"description"=>nil,
"down_count"=>0,
"down_voters"=>[],
"image_small_url"=>nil,
"image_thumb_url"=>nil,
"image_url"=>nil,
"link"=>nil,
"name"=>"a nut",
"order"=>nil,
"picture"=>nil,
"up_count"=>0,
"up_voters"=>[],
"updated_at"=>"2014-10-29T01:54:29Z",
"vote_count"=>0}],
"name"=>"try me",
"parent_list_creator"=>nil,
"parent_list_id"=>nil,
"private"=>false,
"promoted"=>false,
"slug"=>"things-for-fixing-anything",
"up_count"=>0,
"up_voters"=>[],
"updated_at"=>"2014-10-29T02:59:03Z",
"user_id"=>"543809c4b64c402d6a000003",
"vote_count"=>0,
"permalink"=>"/lists/things-for-fixing-anything",
"sub_text"=>"List - created by: CreepyTimes",
"title_text"=>"try me",
"icon_url"=>"",
"item_names"=>"a swiss army knife duct tape Crazy glue a nut",
"creator_name"=>"CreepyTimes",
"creator_avatar"=>"https://blahblah.com/uploads/profile/image/543809c4b64c402d6a000004/thumb__old_",
"cover_image"=>nil,
"tags"=>[]}}>
So, I see "crazy glue" as an item in the list, but it's one of many - not exactly something I can use out of the box here for Auto-complete purposes when someone starts typing an Item name
Is there a way to do what I'm trying to do using nested queries, filters or something? Relatively new to Elasticsearch, so I could use some help on the end solution. If I'm not thinking about this the right way and should simply index Items, I can do that too, but just curious if there's a way to make this work!
EDIT - this is the query going to Elasticsearch:
#<Elasticsearch::Model::Searching::SearchRequest:0x007fc555032218
#definition=
{:index=>"lists",
:type=>"list",
:body=>
{:query=>
{:bool=>
{:should=>
[{:match=>{:name=>"try me"}},
{:nested=>{:path=>"items", :score_mode=>"max", :query=>{:bool=>{:must=>[{:match=>{"items.name"=>"*crazy glue*"}}]}}}}]}}},
:size=>5,
:from=>0},
#klass=[PROXY] List,
#options={}>>
Related
Rails 4 - PG Search not returning exact match with tags
On rails 4 with the acts as taggable gem. My search is currently not returning exact matches first. It seems to be like the tags aren't being weighted properly. When I get rid of the :associated_against => { :tags => {:name => 'D'}} exact matches are returned first. Has anyone ran into this issue before? Any suggestions? Here is my search scope: pg_search_scope :search, :against => { :specific => 'A', :title => 'B', :aka => 'B'}, :associated_against => { :tags => {:name => 'D'}}, :using => { dmetaphone: {}, tsearch: { dictionary: 'english' }, trigram: {:threshold => 0.3} }, ignoring: :accents
Can you post the rest of your code in the controller, etc. I have the following in my app: # tools.rb include PgSearch pg_search_scope :search_including_tags, :against => [:description, :barcode], :associated_against => {:tags => [:name] } Then in my controller to search through I have: #tools_controller.rb def index if params[:search] #tools = Tool.where("(barcode) LIKE (?)", "%#{params[:search]}") elsif params[:tag] #tools = Tool.tagged_with(params[:tag]) elsif params[:id] #tool = Tool.find(params[:id]) else #tools = Tool.all #tool = Tool.first end end and finally for my search controller def new #tools = Tool.search_including_tags(params[:query]) end Hope this helps. Can't really say much without seeing all of the code. But I ended up using this which worked: :associated_against => {:tags => [:name] }
exclude 'pencil sharpener' from the results if the client searched for 'pencil'
I am using tire and we face a search result problem. We are searching for 'pencil'. 'red pencil' OK 'electronic pencil sharpener' NOT OK should not be included in the result set. This is the tire settings on the model: settings :analysis => { :analyzer => { :my_analyzer => { "tokenizer" => "lowercase", # "filter" => ["synonym", "porterStem", "phonetic"] "filter" => ["synonym", "porterStem"] } }, :filter => { :synonym => { "type" => "synonym", "synonyms_path" => "#{Synonym.path}" } } } do mapping do indexes :commodity_code indexes :commodity_name indexes :long_description, analyzer: 'my_analyzer' indexes :short_description, boost: 10, analyzer: 'my_analyzer'
The query electronic pencil sharpener will be translated to electronic OR pencil OR sharpener by default. If you want to exclude documents containing sharpener, use a query like this: electronic OR pencil NOT sharpener or +electronic +pencil -sharpener. Have look at bool and match queries to express conditions like these in the Query DSL; https://github.com/karmi/tire/tree/master/test/integration
Elasticsearch term AND range filter using tire
I am trying build a search function in rails based on elasticsearch+tire enabling search for Persons with filtering for associated Objects and their Values. A Person has_many Objects, and an Object has_many Values. I have managed to get the filtering on the object name (params[:object]) to work, but not for object+value. How should I construct the range filter for the values and the mapping so that the value is dependent on the object? Person controller mapping do indexes :objects do indexes :_id indexes :object_values do indexes :value end end indexes :name, type: 'string', analyzer: 'snowball' end def self.search(params) tire.search do query do boolean do must { string params[:query]} if params[:query].present? end end filter :term, {"objects._id" => params[:object]} if params[:object].present? filter :range, “objects.object_values.value” => {from: params[:value] } if params[:value].present? end end def to_indexed_json { :name => name, :objects => objects.map { |o| { :_type => 'object', :_id => o.id, :object_values => o.object_values.map {|ov| { :_type => 'object_value', :_id => ov.id, :value => ov.value } }, } } }.to_json end
Use gt or gte rather than from to specify the lower bounds of your range filter :range, “objects.object_values.value” => {from: params[:value] }
Tire does not find partial word (search on 2 fields)
What I want to do: I have a model 'Item' with 2 fields I want elasticsearch to search on: title and description. I want the search to find partial words, ex: bicycl should match against bicycle, bicycles, etc... Current situation: The search only shows perfect matches Here is what I have right now in my Item model: include Tire::Model::Search include Tire::Model::Callbacks class << self def search_index Tire.index(Item.index_name) end end settings :analysis => { :filter => { :my_ngram => { "type" => "nGram", "max_gram" => 10, "min_gram" => 3 } }, :analyzer => { :my_analyzer => { "type" => "custom", "tokenizer" => "standard", "filter" => ["my_ngram"] } } } do mapping do indexes :title, boost: 10, analyzer: 'my_analyzer' indexes :description, boost: 5, analyzer: 'my_analyzer' end end def self.search(query_string) tire.search(load: true) do if query_string.present? query do string query_string, default_operator: "AND" end end end end
When you do... string query_string, default_operator: "AND" ... you're actually searching the magic _all field. I'm pretty sure that you need to specifically search for the field analyzed with the ngram filter for this to work. should { string "title:#{query_string}", default_operator: "OR" } should { string "description:#{query_string}", default_operator: "OR" } for instance.
How to show foreign key relationship data in jqgrid?
I know how to access foreign key attributes in a scaffold index view. I can simply refer to the attributes using dot notation such as property.que.name. Given the following models: class Priority < ActiveRecord::Base belongs_to :que ... end class Que < ActiveRecord::Base has_many :priorities ... end In the index view, I can do something like this to get the name value: <td><%=h priority.que ? priority.que.name : "" %></td> How do I do this in the jqgrid? I tried this but the jqgrid comes back empty: Priorities Controller: #priorities = Priority.find(:all, :order => "position", :conditions => "multitenant_team_id = " + current_user.team.id.to_s ) do if params[:_search] == "true" id =~ "%#{params[:id]}%" if params[:id].present? issue_id =~ "%#{params[:issue_id]}%" if params[:issue_id].present? que =~ "%#{params[:que]}%" if params[:que].present? customer =~ "%#{params[:customer]}%" if params[:customer].present? title =~ "%#{params[:title]}%" if params[:title].present? reporting_source =~ "%#{params[:reporting_source]}%" if params[:reporting_source].present? priority =~ "%#{params[:priority]}%" if params[:priority].present? product =~ "%#{params[:product]}%" if params[:product].present? current_owner =~ "%#{params[:current_owner]}%" if params[:current_owner].present? end paginate :page => params[:page], :per_page => params[:rows] order_by "#{params[:sidx]} #{params[:sord]}" end if request.xhr? end respond_to do |format| format.html # index.html.erb format.json { render :json => #priorities.to_jqgrid_json( [:id, :issue_id, :que.name, :customer, :title, :reporting_source, :priority, :product, :current_owner], params[:page], params[:rows], #priorities.total_entries)} format.xml { render :xml => #priorities } end end Index View: <%= jqgrid("Priorities", "priorities", "/priorities", [ {:field => "id", :label => "ID", :width => 35, :resizable => false}, {:field => "issue_id", :label => "Issue Id"}, {:field => "que", :label => "Queue"}, {:field => "customer", :label => "Customer"}, {:field => "title", :label => "Title"}, {:field => "reporting_source", :label => "Reporting Source"}, {:field => "priority", :label => "Priority"}, {:field => "product", :label => "Product"}, {:field => "current_owner", :label => "Current Owner"} ], { :rows_per_page => 12, :height => 450 } )%> If I specify que instead of que.name, I get the data back in the grid but the Queue field shows a "#" symbol so I suspect the .to_jqgrid_json call doesn't like my syntax. Has anyone tried this before? I hope so.
I fixed my problem. I ended up changing my find to a find_by_sql so I could do a left outer join on the ques table. I think there were a couple of issues. I think the *to_jqgrid_json* had problems with null foreign key values and I couldn't figure out how to get at the Que.name any other way. I'm using SQLServer so I had to use isnull(ques.name, '') to convert the null to empty space. So I replaced my find as follows: #priorities = Priority.find_by_sql ["select priorities.id id, issue_id, isnull(ques.name,' ') queue_name, customer, title, reporting_source, priority, product, current_owner from priorities left outer join ques on priorities.que_id = ques.id where priorities.multitenant_team_id = ? order by issue_id", current_user.team.id.to_s] This introduced another problem in that find_by_sql returns an array which breaks the #priorities.total_entries call. So I had to replace it with array.count. format.json { render :json => #priorities.to_jqgrid_json( [:id, :issue_id, :queue_name, :customer, :title, :reporting_source, :priority, :product, :current_owner], params[:page], params[:rows], #priorities.count)} My grid looks great! Edit My grid LOOKS great but it doesn't paginate or sort. Back to the drawing board. :(
Okay, I think I fixed it for real this time. #priorities = Priority.find(:all, :select => "priorities.id, priorities.issue_id, priorities.customer, priorities.title, priorities.reporting_source, priorities.priority, priorities.product, priorities.current_owner, priorities.position, isnull(ques.name,' ') queue_name", :joins => "LEFT OUTER JOIN ques ON ques.id = priorities.que_id", :order => "priorities.position", :conditions => "priorities.multitenant_team_id = " + current_user.team.id.to_s ) do I had know idea I could specify joins like this. This keeps the resultset in a format the 2dc_jqgrid plugin likes. Sorting, pagination and searching all work now. Now my grid looks good and actually works.