Rails 4, elasticsearch-rails - ruby-on-rails

I'm looking for some advice on the best way forward with my app which i have began to integrate elasticsearch for the first time. Im a bit of a beginner in rails but keen to dive in so forgive any glaring errors!
I followed a tutorial http://www.sitepoint.com/full-text-search-rails-elasticsearch/ and have also implemented some additional elasticsearch dsl features from reading documentation etc.. im just not convinced i'm there yet. (I certainly need to move out of the model, as currently most sits in the Product active record model.)
I am trying to implement a search on the Product model with ability to partial word search, fuzzy search (misspellings). From what I understand, I am able to set my own analyzers and filters for the elasticsearch, which I have done and currently reside in the Product model. I would like to move these to a more sensible location too, once I have established if indeed I am actually doing this correctly. I do get results when i search but im including things like deleting the index, creating a new index with mapping all in the end of the product model, if what i have below is not "the correct way", what better way is there than what i have to 1, implement elastic search using rails 2, seperate concerns more efficiently.
thanks and much appreciated
CODE:
lib/tasks/elasticsearch.rake:
require 'elasticsearch/rails/tasks/import'
View:
<%= form_tag search_index_path, class: 'search', method: :get do %>
<%= text_field_tag :query, params[:query], autocomplete: :off, placeholder: 'Search', class: 'search' %>
<% end %>
Gems i used:
gem 'elasticsearch-model', git: 'git://github.com/elasticsearch/elasticsearch-rails.git'
gem 'elasticsearch-rails', git: 'git://github.com/elasticsearch/elasticsearch-rails.git'
Search Controller:
class SearchController < ApplicationController
def index
if params[:query].nil?
#products = []
else
#products = Product.search(params[:query])
end
end
end
Product Model:
require 'elasticsearch/model'
class Product < ActiveRecord::Base
# ElasticSearch
include Elasticsearch::Model
include Elasticsearch::Model::Callbacks
settings index: {
number_of_shards: 1,
analysis: {
filter: {
trigrams_filter: {
type: 'ngram',
min_gram: 2,
max_gram: 10
},
content_filter: {
type: 'ngram',
min_gram: 4,
max_gram: 20
}
},
analyzer: {
index_trigrams_analyzer: {
type: 'custom',
tokenizer: 'standard',
filter: ['lowercase', 'trigrams_filter']
},
search_trigrams_analyzer: {
type: 'custom',
tokenizer: 'whitespace',
filter: ['lowercase']
},
english: {
tokenizer: 'standard',
filter: ['standard', 'lowercase', 'content_filter']
}
}
}
} do
mappings dynamic: 'false' do
indexes :name, index_analyzer: 'index_trigrams_analyzer', search_analyzer: 'search_trigrams_analyzer'
indexes :description, index_analyzer: 'english', search_analyzer: 'english'
indexes :manufacturer_name, index_analyzer: 'english', search_analyzer: 'english'
indexes :type_name, analyzer: 'snowball'
end
end
# Gem Plugins
acts_as_taggable
has_ancestry
has_paper_trail
## -99,6 +146,33 ## def all_sizes
product_attributes.where(key: 'Size').map(&:value).join(',').split(',')
end
def self.search(query)
__elasticsearch__.search(
{
query: {
query_string: {
query: query,
fuzziness: 2,
default_operator: "AND",
fields: ['name^10', 'description', 'manufacturer_name', 'type_name']
}
},
highlight: {
pre_tags: ['<em>'],
post_tags: ['</em>'],
fields: {
name: {},
description: {}
}
}
}
)
end
def as_indexed_json(options={})
as_json(methods: [:manufacturer_name, :type_name])
end
end
# Delete the previous products index in Elasticsearch
Product.__elasticsearch__.client.indices.delete index: Product.index_name rescue nil
# Create the new index with the new mapping
Product.__elasticsearch__.client.indices.create \
index: Product.index_name,
body: { settings: Product.settings.to_hash, mappings: Product.mappings.to_hash }
# Index all article records from the DB to Elasticsearch
Product.import(force: true)
end

If you are using elasticsearch for searching,then i will recommend gem 'chewy' with elasticsearch server.
For more information just go to the links provided below.
for chewy:
https://github.com/toptal/chewy
Integrate chewy with elasticsearch:
http://www.toptal.com/ruby-on-rails/elasticsearch-for-ruby-on-rails-an-introduction-to-chewy
Thanks

I can recommend searchkick:
https://github.com/ankane/searchkick
Several apps in production running with searchkick and it's easy to use.
Also check out the documentation of searchkick where a search for products is described in detail with facets, suggestions, etc.

Related

Wish the search method for elasticsearch is working on rails

■The environment
MacOS
RailsServer
Ruby 2.4.1
Ruby on Rails 5.1.7
MySQL
Elasticsearch 7.10.2-SNAPSHOT
kuromoji
Gems
elasticsearch (7.4.0)
elasticsearch-api (7.4.0)
elasticsearch-model (7.1.0 80822d6)
elasticsearch-rails (7.1.0 80822d6)
elasticsearch-transport (7.4.0)
■My wish
Hi I'm japanese.
My wish is that read faster on database to use elasticsearch on rails.
Now i'm trouble with search method on rails.
I will explain the current settings.
First, I think the record on elastic search already input on following below.
curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open es_project_development Fpu_adXWT9Gtw7KZTh0aDw 1 1 6804 0 3.7mb 3.7mb
Second, the setting of index and mapping for elasticsearch are the following below.
/models/concerns/project_searchable.rb
module ProjectSearchable
extend ActiveSupport::Concern
included do
include Elasticsearch::Model
index_name "es_project_#{Rails.env}"
settings do
mappings dynamic: 'false' do
indexes :id, type: 'integer'
indexes :title, type: 'text', analyzer: 'kuromoji'
indexes :contents, type: 'text', analyzer: 'kuromoji'
indexes :industry, type: 'text'
...
def as_indexed_json(*)
attributes
.symbolize_keys
.slice(:id, :title, :contents, :industry, ...)
end
end
class_methods do
def create_index!
client = __elasticsearch__.client
client.indices.delete index: self.index_name rescue nil
client.indices.create(index: self.index_name,
body: {
settings: self.settings.to_hash,
mappings: self.mappings.to_hash
})
end
def es_search(query)
__elasticsearch__.search({
query: {
multi_match: {
fields: %w(id title contents industry ...),
type: 'cross_fields',
query: query,
operator: 'and'
}
}
})
end
end
end
Third, the view and the controller are the following below.
/views/top/index.html.slim
= form_tag search_path, {:method=>"get"}
table border="0"
tr
td
= text_field_tag 'keyword[name]', nil, class: 'write'
td
input.push type="submit" value=""
...
/controllers/projects_controller.rb
def search
...
#keyword = params.dig('keyword', 'name')
params = ''
connection = '?'
...
if #keyword.present?
params = "#{params}#{connection}keyword=#{#keyword}"
connection = '&'
end
...
Last, ProjectSearchable is included in models/project.rb.
/models/project.rb
class Project < ApplicationRecord
include ProjectSearchable
...
■The problem
On rails console, i typed command
Project.es_search('Game')
Response was following.
※Game is included in the record
#<Elasticsearch::Model::Response::Response:0x007fc1a85621a8
#klass=[PROXY] Project (call 'Project.connection' to establish a connection),
#search=
#<Elasticsearch::Model::Searching::SearchRequest:0x007fc1a8562248
#definition=
{:index=>"es_project_development",
:type=>nil,
:body=>
{:query=>
{:multi_match=>
{:fields=>
["id",
"title",
"contents",
"industry",
"required",
...
"comment"],
:type=>"cross_fields",
:query=>"Game",
:operator=>"and"}}}},
#klass=[PROXY] Project (call 'Project.connection' to establish a connection),
#options={}>>
I think elasticsearch doesn't work.
My opinion of this problem is in search method for elasticsearch.
But i don't have enough knowledge about that.
I really need your help.
Thank you.

Query nested model with multiple plucks

I was just wondering if the following is possible: I have model with nested associative models. I want to be able to render json: on current_user.reports.minned and have it eager_load the plucked values from each model. How can I accomplish this?
Here I use only 2 models as an example. In reality, the solution needs to work for n+1 nested models.
Does not work:
class Report
has_many :templates
def minned
self.pluck(:id, :title)
self.templates = templates.minned
end
end
class Template
belongs_to :report
def minned
self.pluck(:id, :name, :sections, :columns)
end
end
....
# reports.minned.limit(limit).offset(offset)
# This should return something like:
[{
'id': 0,
'title': 'Rep',
'templates': [{
'id': 0,
'name': 'Temp'
'sections': [],
'columns': []
}]
},
{
'id': 1,
'title': 'Rep 1',
'templates': [{
'id': 0,
'name': 'Temp',
'sections': [],
'columns': []
},
{
'id': 1,
'name': 'Temp 1',
'sections': [],
'columns': []
}]
}]
Thanks for any help.
Edit:
I will add that I found a way to do this by overriding as_json for each model, but this applies the plucking to all requests. I need to have control over which requests give what pieces of information.
# in Report model
def as_json(options={})
super(:id, :title).merge(templates: templates)
end
# in Template model
def as_json(options={})
super(:id, :name, :sections, :columns)
end
Thanks to eirikir, this is all I need to do:
Report model
def self.minned
includes(:templates).as_json(only: [:id, :title], include: {templates: {only: [:id, :name, :sections, :columns]}})
end
Then when using this with pagination order, limit or anything like that, just drop it at the end:
paginate pre_paginated_reports.count, max_per_page do |limit, offset|
render json: pre_paginated_reports.order(id: :desc).limit(limit).offset(offset).minned
end
Now I'm not overriding as_json and have complete control over the data I get back.
If I understand correctly, you should be able to achieve this specifying the output in the options given to as_json:
current_user.reports.includes(:templates).as_json(only: [:id, :title], include: {templates: {only: [:id, :name, :sections, :columns]}})

tire and elasticsearch_autocomplete accents and facets

I'm using elasticsearch_autocomplete gem for autocomplete feature.
I have a problem with special characters ñ and accents áéíóú.
Model:
class Car
ac_field :name, :description, :city, :skip_settings => true
def self.ac_search(params, options={})
tire.search load: true, page: params[:page], per_page: 9 do
query do
boolean do
must { string params[:query], default_operator: "AND" } if params[:query].present?
must { term :city, params[:city] } if params[:city].present?
end
end
filter :term, city: params[:city] if params[:city].present?
facet "city" do
terms :city
end
end
end
end
This version works fine with special characters.
e.g.: Query with Martin I get all results with Martín, martín, martin, Martin
With this approach this is the problem:
Now what results is individual words. e.g. A city tagged ["San Francisco", "Madrid"] will end up having three separate tags. Similarly, if I do a query to search on "san francisco" (must { term 'city', params[:city] }), that will fail, while a query on "San" or "Francisco" will succeed. The desired behaviour here is that the tag should be atomic, so only a "San Francisco" (or "Madrid") tag search should succeed.
To fix this problem I create my custom mapping:
model = self
settings ElasticsearchAutocomplete::Analyzers::AC_BASE do
mapping _source: {enabled: true, includes: %w(name description city)} do
indexes :name, model.ac_index_config(:name)
indexes :description, model.ac_index_config(:description)
indexes :city, :type => 'string', :index => :not_analyzed
end
end
With this mapping the problem with multi-words is fixed, and now facets with city field works fine:
Instead of getting the type facets San and Francisco Now I get San Francisco
Now, the problem is that with this mapping inside of the model the search doesn't find results with special characters
e.g.: Query with Martin I get only results with Martin martin
I'm using mongoid instead active record.
How can I fix this problem? I think that the problem is with asciifolding tokenfilter.
I fixed the problem with:
class User
include Mongoid::Document
field :city, :type => String
has_one: car
end
class Car
ac_field :name, :description, :user_city, :skip_settings => true
def self.ac_search(params, options={})
tire.search load: true, page: params[:page], per_page: 9 do
query do
boolean do
must { term :user_city, params[:user_city] } if params[:user_city].present?
end
end
facet "cities" do
terms :user_city
end
end
end
model = self
settings ElasticsearchAutocomplete::Analyzers::AC_BASE do
mapping _source: {enabled: true, includes: %w(car_city name description)} do
indexes :car_city, :type => 'string', :index => :not_analyzed
end
end
def to_indexed_json
to_json(methods: [:user_city])
end
def user_city
user.city
end
end

Why multi-field mapping is not working with tire gem for elasticsearch?

I'm using elastic search to enhance search capabilities in my app. Search is working perfectly, however sorting is not for fields with multiple words.
When I try to sort the search by log 'message', I was getting the error:
"Can't sort on string types with more than one value per doc, or more than one token per field"
I googled the error and find out that I can use multi-fields mapping on the :message field (one analyzed and the other one not) to sort them. So I did this:
class Log < ActiveRecord::Base
include Tire::Model::Search
include Tire::Model::Callbacks
tire.mapping do
indexes :id, index: :not_analyzed
indexes :source, type: 'string'
indexes :level, type: 'string'
indexes :created_at, :type => 'date', :include_in_all => false
indexes :updated_at, :type => 'date', :include_in_all => false
indexes :message, type: 'multi_field', fields: {
analyzed: {type: 'string', index: 'analyzed'},
message: {type: 'string', index: :not_analyzed}
}
indexes :domain, type: 'keyword'
end
end
But, for some reason is not passing this mapping to ES.
rails console
Log.index.delete #=> true
Log.index.create #=> 200 : {"ok":true,"acknowledged":true}
Log.index.import Log.all #=> 200 : {"took":243,"items":[{"index":{"_index":"logs","_type":"log","_id":"5 ... ...
# Index mapping for :message is not the multi-field
# as I created in the Log model... why?
Log.index.mapping
=> {"log"=>
{"properties"=>
{"created_at"=>{"type"=>"date", "format"=>"dateOptionalTime"},
"id"=>{"type"=>"long"},
"level"=>{"type"=>"string"},
"message"=>{"type"=>"string"},
"source"=>{"type"=>"string"},
"updated_at"=>{"type"=>"date", "format"=>"dateOptionalTime"}}}}
# However if I do a Log.mapping I can see the multi-field
# how I can fix that and pass the mapping correctly to ES?
Log.mapping
=> {:id=>{:index=>:not_analyzed, :type=>"string"},
:source=>{:type=>"string"},
:level=>{:type=>"string"},
:created_at=>{:type=>"date", :include_in_all=>false},
:updated_at=>{:type=>"date", :include_in_all=>false},
:message=>
{:type=>"multi_field",
:fields=>
{:message=>{:type=>"string", :index=>"analyzed"},
:untouched=>{:type=>"string", :index=>:not_analyzed}}},
:domain=>{:type=>"keyword"}}
So, Log.index.mapping is the current mapping in ES which doesn't contain the multi-field that I created. Am I missing something? and why the multi-field is shown in Log.mapping but not in Log.index.mapping?
I have changed the workflow from:
Log.index.delete; Log.index.create; Log.import
to
Log.index.delete; Log.create_elasticsearch_index; Log.import
The MyModel.create_elasticsearch_index creates the index with proper mapping from model definition. See Tire's issue #613.

Having trouble with mixed complex boolean with Tire (and ElasticSearch)

I've been trying to figure out how to do mixed boolean searches that use nested objects using Tire. All the simple examples I've found don't include a more complex query (when searching on other attributes).
My search involves finding a Team that 'needs' a specific type of person. When trying to build a football team, the team needs to fill the roster with certain types of players of a given weight class, with the option of excluding one term or the other.
Other parameters such as 'region' or 'kind' have to do with where the team plays, and what kind of team (casual, competitive, etc) it is.
My current setup:
mapping do
indexes :region, index: :not_analyzed
indexes :kind, index: :not_analyzed
indexes :role_requirements do
indexes :need, type: 'boolean'
indexes :weight_class_id, type: 'integer'
indexes :role_id, type: 'integer'
end
.
.
.
end
def self.search(params)
team_params = params[:team_search]
tire.search(page: params[:page], per_page: 10) do
query do
boolean do
must { string team_params[:query], default_operator: "AND" } if team_params[:query].present?
must { term :kind, team_params[:kind] } if team_params[:kind].present?
must { term :region, team_params[:region] } if team_params[:region].present?
if team_params[:weight_class_id].present? || team_params[:role_id].present?
must { term 'role_requirements.need', true }
end
must { term 'role_requirements.weight_class_id', team_params[:job_id].to_i } if team_params[:weight_class_id].present?
must { term 'role_requirements.role_id', team_params[:role_id].to_i } if team_params[:role_id].present?
.
.
.
end
end
end
end
I've tried a number of ways, but there usually seems to be a problem either with ElasticSearch not able to parse things or Tire not having the method within the scope:
With this implementation, here is the generated to_json: https://gist.github.com/8a615e701eb31ff2e250
Which are currently not giving me any results.
All the different ways I've tried: https://gist.github.com/907c9571caa0e87bad27
None are really able to give me full results.
You seem to be missing the nested type in the mapping:
mapping do
indexes :region, index: :not_analyzed
indexes :kind, index: :not_analyzed
indexes :role_requirements, type: 'nested' do
indexes :need, type: 'boolean'
indexes :weight_class_id, type: 'integer'
indexes :role_id, type: 'integer'
end
# ..more mappings..
end
Then you can build your query like this:
tire.search(page: params[:page], per_page: 10) do
query do
boolean do
must { string team_params[:query], default_operator: "AND" } if team_params[:query].present?
must { term :kind, team_params[:kind] } if team_params[:kind].present?
must { term :region, team_params[:region] } if team_params[:region].present?
must do
nested path: 'role_requirements' do
query do
boolean do
if team_params[:weight_class_id].present? || team_params[:role_id].present?
must { term 'role_requirements.need', true }
end
must { term 'role_requirements.weight_class_id', team_params[:job_id].to_i } if team_params[:weight_class_id].present?
must { term 'role_requirements.role_id', team_params[:role_id].to_i } if team_params[:role_id].present?
end
end
end
end
end
end
end
Here are some examples in Tire's integration tests.
Hope this helps :)

Resources