■The environment
MacOS
RailsServer
Ruby 2.4.1
Ruby on Rails 5.1.7
MySQL
Elasticsearch 7.10.2-SNAPSHOT
kuromoji
Gems
elasticsearch (7.4.0)
elasticsearch-api (7.4.0)
elasticsearch-model (7.1.0 80822d6)
elasticsearch-rails (7.1.0 80822d6)
elasticsearch-transport (7.4.0)
■My wish
Hi I'm japanese.
My wish is that read faster on database to use elasticsearch on rails.
Now i'm trouble with search method on rails.
I will explain the current settings.
First, I think the record on elastic search already input on following below.
curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open es_project_development Fpu_adXWT9Gtw7KZTh0aDw 1 1 6804 0 3.7mb 3.7mb
Second, the setting of index and mapping for elasticsearch are the following below.
/models/concerns/project_searchable.rb
module ProjectSearchable
extend ActiveSupport::Concern
included do
include Elasticsearch::Model
index_name "es_project_#{Rails.env}"
settings do
mappings dynamic: 'false' do
indexes :id, type: 'integer'
indexes :title, type: 'text', analyzer: 'kuromoji'
indexes :contents, type: 'text', analyzer: 'kuromoji'
indexes :industry, type: 'text'
...
def as_indexed_json(*)
attributes
.symbolize_keys
.slice(:id, :title, :contents, :industry, ...)
end
end
class_methods do
def create_index!
client = __elasticsearch__.client
client.indices.delete index: self.index_name rescue nil
client.indices.create(index: self.index_name,
body: {
settings: self.settings.to_hash,
mappings: self.mappings.to_hash
})
end
def es_search(query)
__elasticsearch__.search({
query: {
multi_match: {
fields: %w(id title contents industry ...),
type: 'cross_fields',
query: query,
operator: 'and'
}
}
})
end
end
end
Third, the view and the controller are the following below.
/views/top/index.html.slim
= form_tag search_path, {:method=>"get"}
table border="0"
tr
td
= text_field_tag 'keyword[name]', nil, class: 'write'
td
input.push type="submit" value=""
...
/controllers/projects_controller.rb
def search
...
#keyword = params.dig('keyword', 'name')
params = ''
connection = '?'
...
if #keyword.present?
params = "#{params}#{connection}keyword=#{#keyword}"
connection = '&'
end
...
Last, ProjectSearchable is included in models/project.rb.
/models/project.rb
class Project < ApplicationRecord
include ProjectSearchable
...
■The problem
On rails console, i typed command
Project.es_search('Game')
Response was following.
※Game is included in the record
#<Elasticsearch::Model::Response::Response:0x007fc1a85621a8
#klass=[PROXY] Project (call 'Project.connection' to establish a connection),
#search=
#<Elasticsearch::Model::Searching::SearchRequest:0x007fc1a8562248
#definition=
{:index=>"es_project_development",
:type=>nil,
:body=>
{:query=>
{:multi_match=>
{:fields=>
["id",
"title",
"contents",
"industry",
"required",
...
"comment"],
:type=>"cross_fields",
:query=>"Game",
:operator=>"and"}}}},
#klass=[PROXY] Project (call 'Project.connection' to establish a connection),
#options={}>>
I think elasticsearch doesn't work.
My opinion of this problem is in search method for elasticsearch.
But i don't have enough knowledge about that.
I really need your help.
Thank you.
Related
I added the following filter in ActiveAdmin.
filter :roles, as: :select, collection Model::ROLES, multiple: true
but when i choose the filter value to search the roles. it gives me following error
PG::InvalidTextRepresentation: ERROR: malformed array literal: "teacher"LINE 1: ...ted" = $1 AND roles" IN('teacher
DETAIL: Array value must start with "{" or dimension information. ^
Any idea ? How we can search/Filter ARRAY field using AA filters? I'm using Rails 4.2.4,
ruby 2.2.2p95
I came up to a solution slightly different (and inspired by) this one over here: https://stackoverflow.com/a/45728004/1170086
Mine involves some changes (and prevent breaking contains operator in other cases). So, you're going to basically create two initializer files:
This one is for Arel, in order to support #> operator (array's contain operator in PG) for a given table column.
# config/initializers/arel.rb
module Arel
class Nodes::ContainsArray < Arel::Nodes::Binary
def operator
:"#>"
end
end
class Visitors::PostgreSQL
private
def visit_Arel_Nodes_ContainsArray(o, collector)
infix_value o, collector, ' #> '
end
end
module Predications
def contains(other)
Nodes::ContainsArray.new self, Nodes.build_quoted(other, self)
end
end
end
The other file aims to create a new Ransack predicate but I also decided to support the :array type (that's not natively supported in Ransack in terms of predicates).
# config/initializers/ransack.rb
module Ransack
module Nodes
class Value < Node
alias_method :original_cast, :cast
def cast(type)
return Array(value) if type == :array
original_cast(type)
end
end
end
end
Ransack.configure do |config|
config.add_predicate 'contains_array',
arel_predicate: 'contains',
formatter: proc { |v| "{#{v.join(',')}}" },
validator: proc { |v| v.present? },
type: :array
end
And in other to use it. All you need to do is:
User.ransack(roles_contains_array: %i[admin manager])
Or as a filter in ActiveAdmin (which is my case):
ActiveAdmin.register User do
# ...
filter :roles_contains_array, as: :select, collection: User.roles_for_select
# ...
end
I hope it works for you as it worked for me. ;)
You can set up a custom ransacker method to first collect the ids you want returned using a regular postgres search, and then return the results based on those ids:
class User < ApplicationRecord
ransacker :roles,
formatter: proc { |str|
data = where("? = ANY (roles)", str).map(&:id)
data.present? ? data : nil
} do |parent|
parent.table[:id]
end
end
If your filter is a select drop-down, then this should work fine. If you have a free-form text box, then make sure to use the "in" predicate:
filter :roles_in, as: :string
leandroico solutions works well.
But if you add the predicate with this formatter
formatter: proc { |v| "{#{v.join(', ')}}" }, (note the space after the comma)
Then you could use the multiple: true keyword in the filter input and filter by more than one value:
filter :roles_contains_array, as: :select, multiple: true, collection: User.roles_for_select
I used the answer from #leandroico to come up with the below wiki-type approach to doing this.
How to Create Custom SQL Searches for ActiveAdmin (using Arel and Ransack)
In ActiveAdmin, filters are declared in app/admin/model.rb like:
ActiveAdmin.register Model do
filter 'column_name', label: 'column_name', as: :string
end
That will make a searchbox available on the front-end with options to choose between
contains
equals
starts with
ends with
You can even do something like...
filter 'column_name_contains', label: 'column_name', as: :string
...to only have a contains type search available on the front-end.
You can also (after defining some custom methods elsewhere) specify other, non-built-in search methods, like:
filter 'column_name_custom_contains', label: 'column_name', as: :string
The rest of this doc will be about how to define this custom search method, custom_contains
Within config/initializers/arel.rb, define the following:
module Arel
# this example of custom_contains will cast the SQL column as ::text and then do a wildcard-wrapped ILIKE
class Nodes::CustomContains < Arel::Nodes::Binary
def operator
'::text ILIKE'.to_sym
end
end
class Visitors::PostgreSQL
private
def visit_Arel_Nodes_CustomContains(o, collector)
infix_value o, collector, '::text ILIKE '
end
end
module Predications
def custom_contains(column_value)
column_value = self.relation.engine.column_types[self.name.to_s].type_cast_for_database(column_value)
column_value = "%#{self.relation.engine.send(:sanitize_sql_like, column_value)}%" # wrap escaped value with % wildcard
column_value = Nodes.build_quoted(column_value, self)
Nodes::CustomContains.new(self, column_value)
end
end
end
module ActiveRecord::QueryMethods
def custom_contains(predicates)
return none if predicates.length == 0
predicates.map{ |column_name, column_value|
column_value = table.engine.column_types[column_name.to_s].type_cast_for_database(column_value)
column_value = "%#{table.engine.send(:sanitize_sql_like, column_value)}%" # wrap escaped value with % wildcard
column_value = Arel::Nodes.build_quoted(column_value)
where Arel::Nodes::CustomContains.new(table[column_name], column_value)
}.inject(:merge)
end
end
module ActiveRecord::Querying
delegate :custom_contains, :to => :all
end
Within config/initializers/ransack.rb, define the following:
Ransack.configure do |config|
config.add_predicate(
'custom_contains',
arel_predicate: 'custom_contains',
formatter: proc { |v| v.to_s },
validator: proc { |v| v.present? },
type: :string
)
end
The above has accomplished a couple of things:
1) You can use the custom_contains method that was delegate'd to all ActiveRecord models:
puts Model.custom_contains(column_name: 'search for me').to_sql
2) You can use Ransack to search against the Arel predicates that were defined:
puts Model.ransack(column_name_custom_contains: 'search for me').result.to_sql
However, in order to do the below in ActiveAdmin...
filter 'column_name_custom_contains', label: 'column_name', as: :string
...we must add a scope to Model so that there is a method, column_name_custom_contains, on Model
scope_name = "#{column_name}_custom_contains".to_sym
unless Model.methods.include?(scope_name)
Model.scope(
scope_name,
->(value) {
Model.custom_contains({column_name.to_sym => value})
}
)
end
Voila!
I'm looking for some advice on the best way forward with my app which i have began to integrate elasticsearch for the first time. Im a bit of a beginner in rails but keen to dive in so forgive any glaring errors!
I followed a tutorial http://www.sitepoint.com/full-text-search-rails-elasticsearch/ and have also implemented some additional elasticsearch dsl features from reading documentation etc.. im just not convinced i'm there yet. (I certainly need to move out of the model, as currently most sits in the Product active record model.)
I am trying to implement a search on the Product model with ability to partial word search, fuzzy search (misspellings). From what I understand, I am able to set my own analyzers and filters for the elasticsearch, which I have done and currently reside in the Product model. I would like to move these to a more sensible location too, once I have established if indeed I am actually doing this correctly. I do get results when i search but im including things like deleting the index, creating a new index with mapping all in the end of the product model, if what i have below is not "the correct way", what better way is there than what i have to 1, implement elastic search using rails 2, seperate concerns more efficiently.
thanks and much appreciated
CODE:
lib/tasks/elasticsearch.rake:
require 'elasticsearch/rails/tasks/import'
View:
<%= form_tag search_index_path, class: 'search', method: :get do %>
<%= text_field_tag :query, params[:query], autocomplete: :off, placeholder: 'Search', class: 'search' %>
<% end %>
Gems i used:
gem 'elasticsearch-model', git: 'git://github.com/elasticsearch/elasticsearch-rails.git'
gem 'elasticsearch-rails', git: 'git://github.com/elasticsearch/elasticsearch-rails.git'
Search Controller:
class SearchController < ApplicationController
def index
if params[:query].nil?
#products = []
else
#products = Product.search(params[:query])
end
end
end
Product Model:
require 'elasticsearch/model'
class Product < ActiveRecord::Base
# ElasticSearch
include Elasticsearch::Model
include Elasticsearch::Model::Callbacks
settings index: {
number_of_shards: 1,
analysis: {
filter: {
trigrams_filter: {
type: 'ngram',
min_gram: 2,
max_gram: 10
},
content_filter: {
type: 'ngram',
min_gram: 4,
max_gram: 20
}
},
analyzer: {
index_trigrams_analyzer: {
type: 'custom',
tokenizer: 'standard',
filter: ['lowercase', 'trigrams_filter']
},
search_trigrams_analyzer: {
type: 'custom',
tokenizer: 'whitespace',
filter: ['lowercase']
},
english: {
tokenizer: 'standard',
filter: ['standard', 'lowercase', 'content_filter']
}
}
}
} do
mappings dynamic: 'false' do
indexes :name, index_analyzer: 'index_trigrams_analyzer', search_analyzer: 'search_trigrams_analyzer'
indexes :description, index_analyzer: 'english', search_analyzer: 'english'
indexes :manufacturer_name, index_analyzer: 'english', search_analyzer: 'english'
indexes :type_name, analyzer: 'snowball'
end
end
# Gem Plugins
acts_as_taggable
has_ancestry
has_paper_trail
## -99,6 +146,33 ## def all_sizes
product_attributes.where(key: 'Size').map(&:value).join(',').split(',')
end
def self.search(query)
__elasticsearch__.search(
{
query: {
query_string: {
query: query,
fuzziness: 2,
default_operator: "AND",
fields: ['name^10', 'description', 'manufacturer_name', 'type_name']
}
},
highlight: {
pre_tags: ['<em>'],
post_tags: ['</em>'],
fields: {
name: {},
description: {}
}
}
}
)
end
def as_indexed_json(options={})
as_json(methods: [:manufacturer_name, :type_name])
end
end
# Delete the previous products index in Elasticsearch
Product.__elasticsearch__.client.indices.delete index: Product.index_name rescue nil
# Create the new index with the new mapping
Product.__elasticsearch__.client.indices.create \
index: Product.index_name,
body: { settings: Product.settings.to_hash, mappings: Product.mappings.to_hash }
# Index all article records from the DB to Elasticsearch
Product.import(force: true)
end
If you are using elasticsearch for searching,then i will recommend gem 'chewy' with elasticsearch server.
For more information just go to the links provided below.
for chewy:
https://github.com/toptal/chewy
Integrate chewy with elasticsearch:
http://www.toptal.com/ruby-on-rails/elasticsearch-for-ruby-on-rails-an-introduction-to-chewy
Thanks
I can recommend searchkick:
https://github.com/ankane/searchkick
Several apps in production running with searchkick and it's easy to use.
Also check out the documentation of searchkick where a search for products is described in detail with facets, suggestions, etc.
I have an Exhibition model that has three named scopes:
scope :opening, -> { where(start_date: (Date.today .... }
scope :closing, -> { where(end_date: .... }
scope :reception, -> { where("reception is not null")... }
I have the following tire mapping inside the same model:
tire.mapping do
...
indexes :start_date, type: "date"
indexes :end_date, type: "date"
indexes :reception, type: "date"
...
end
And my search method:
def self.search(params)
tire.search(load: true) do
query { string params[:query] } if params[:query].present?
...
end
Inside my exhibitions_controller.rb I have the following (and the same for closing and reception)
def opening_this_week
#exhibitions = Exhibition.opening.search(params)
...
end
When I actually search for one of the indexed fields I get back the correct response of just the exhibitions that meet the query and are opening this week, however, when I want all opening exhibitions and do not pass a search query I get ALL exhibitions, not just those that are opening this week.
What am I doing wrong here? How do I transition from a named scope to a facet? Or should I be doing it as a filter?
I'm using elastic search to enhance search capabilities in my app. Search is working perfectly, however sorting is not for fields with multiple words.
When I try to sort the search by log 'message', I was getting the error:
"Can't sort on string types with more than one value per doc, or more than one token per field"
I googled the error and find out that I can use multi-fields mapping on the :message field (one analyzed and the other one not) to sort them. So I did this:
class Log < ActiveRecord::Base
include Tire::Model::Search
include Tire::Model::Callbacks
tire.mapping do
indexes :id, index: :not_analyzed
indexes :source, type: 'string'
indexes :level, type: 'string'
indexes :created_at, :type => 'date', :include_in_all => false
indexes :updated_at, :type => 'date', :include_in_all => false
indexes :message, type: 'multi_field', fields: {
analyzed: {type: 'string', index: 'analyzed'},
message: {type: 'string', index: :not_analyzed}
}
indexes :domain, type: 'keyword'
end
end
But, for some reason is not passing this mapping to ES.
rails console
Log.index.delete #=> true
Log.index.create #=> 200 : {"ok":true,"acknowledged":true}
Log.index.import Log.all #=> 200 : {"took":243,"items":[{"index":{"_index":"logs","_type":"log","_id":"5 ... ...
# Index mapping for :message is not the multi-field
# as I created in the Log model... why?
Log.index.mapping
=> {"log"=>
{"properties"=>
{"created_at"=>{"type"=>"date", "format"=>"dateOptionalTime"},
"id"=>{"type"=>"long"},
"level"=>{"type"=>"string"},
"message"=>{"type"=>"string"},
"source"=>{"type"=>"string"},
"updated_at"=>{"type"=>"date", "format"=>"dateOptionalTime"}}}}
# However if I do a Log.mapping I can see the multi-field
# how I can fix that and pass the mapping correctly to ES?
Log.mapping
=> {:id=>{:index=>:not_analyzed, :type=>"string"},
:source=>{:type=>"string"},
:level=>{:type=>"string"},
:created_at=>{:type=>"date", :include_in_all=>false},
:updated_at=>{:type=>"date", :include_in_all=>false},
:message=>
{:type=>"multi_field",
:fields=>
{:message=>{:type=>"string", :index=>"analyzed"},
:untouched=>{:type=>"string", :index=>:not_analyzed}}},
:domain=>{:type=>"keyword"}}
So, Log.index.mapping is the current mapping in ES which doesn't contain the multi-field that I created. Am I missing something? and why the multi-field is shown in Log.mapping but not in Log.index.mapping?
I have changed the workflow from:
Log.index.delete; Log.index.create; Log.import
to
Log.index.delete; Log.create_elasticsearch_index; Log.import
The MyModel.create_elasticsearch_index creates the index with proper mapping from model definition. See Tire's issue #613.
I am still doing something wrong.
Could somebody pls help me?
I want to create a custom analyzer with ascii filter in Rails + Mongoid.
I have a simple model product which has field name.
class Product
include Mongoid::Document
field :name
settings analysis: {
analyser: {
ascii: {
type: 'custom',
tokenizer: 'whitespace',
filter: ['lowercase','asciifolding']
}
}
}
mapping do
indexes :name, analyzer: 'ascii'
end
end
Product.create(name:"svíčka")
Product.search(q:"svíčka").count #1
Product.search(q:"svicka").count #0 can't find - expected 1
Product.create(name:"svicka")
Product.search(q:"svíčka").count #0 can't find - expected 1
Product.search(q:"svicka").count #1
And when I check the indexes with elasticsearch-head I expected that the index is stored without accents like this "svicka", but the index looks like this "Svíčka".
What am I doing wrong?
When I check it with API it looks OK:
curl -XGET 'localhost:9200/_analyze?tokenizer=whitespace&filters=asciifolding' -d 'svíčka'
{"tokens":[{"token":"svicka","start_offset":0,"end_offset":6,"type":"word","position":1}]}
http://localhost:9200/development_imango_products/_mapping
{"development_imango_products":{"product":{"properties":{"name":{"type":"string","analyzer":"ascii"}}}}}
curl -XGET 'localhost:9200/development_imango_products/_analyze?field=name' -d 'svíčka'
{"tokens":[{"token":"svicka","start_offset":0,"end_offset":6,"type":"word","position":1}]}
You can check how you are actually indexing your document using the analyze api.
You need also to take into account that there's a difference between what you index and what you store. What you store is returned when you query, and it is exactly what you send to elasticsearch, while what you index determines what documents you get back while querying.
Using the asciifolding is a good choice for you usecase, it should return results either query ing for svíčka or svicka. I guess there's just a typo in your settings: analyser should be analyzer. Probably that analyzer is not being used as you'd expect.
UPDATE
Given your comment you didn't solve the problem yet. Can you check what your mapping looks like (localhost:9200/index_name/_mapping)? The way you're using the analyze api is not that useful since you're manually providing the text analysis chain, but that doesn't mean that chain is applied as you'd expect to your field. Better if you provide the name of the field like this:
curl -XGET 'localhost:9200/index_name/_analyze?field=field_name' -d 'svíčka'
That way the analyze api will rely on the actual mapping for that field.
UPDATE 2
After you made sure that the mapping is correctly submitted and everything looks fine, I noticed you're not specifying the field that you want to to query. If you don't specify it you're querying the _all special field, which contains by default all the field that you're indexing, and uses by default the StandardAnalyzer. You should use the following query: name:svíčka.
elasticsearch needs settings and mapping in a single api call. I am not sure if its mentioned in tire docs, but I faced a similar problem, using both settings and mapping when setting up tire. Following should work:
class Product
include Mongoid::Document
# include tire stuff
field :name
settings(self.tire_settings) do
mapping do
indexes :name, analyzer: 'ascii'
end
end
# this method is just created for readablity,
# settings hash can also be passed directly
def self.tire_settings
{
analysis: {
analyzer: {
ascii: {
type: 'custom',
tokenizer: 'whitespace',
filter: ['lowercase','asciifolding']
}
}
}
}
end
end
Your notation for settings/mappings is incorrect, as #rubish suggests, check documentation in https://github.com/karmi/tire/blob/master/lib/tire/model/indexing.rb (no question the docs should be better)
Always, always, always check the mapping of the index to see if your desired mapping has been applied.
Use the Explain API, as #javanna suggests, to check how your analyzing chain works quickly, without having to store documents, check results, etc.
Please note that It is very important to add two lines in a model to make it searchable through Tire. Your model should look like
class Model
include Mongoid::Document
include Mongoid::Timestamps
include Tire::Model::Search
include Tire::Model::Callbacks
field :filed_name, type: String
index_name "#{Tire::Model::Search.index_prefix}model_name"
settings :analysis => {
:analyzer => {
"project_lowercase_analyzer" => {
"tokenizer" => "keyword",
"filter" => ["lowercase"],
"type" => "custom"
}
},
} do
mapping do
indexes :field_name, :boost => 10, :type => 'String', :analyzer => 'standard', :filter => ['standard', 'lowercase','keyword']
end
end
def self.search( params = {} )
query = params[:search-text_field_name_from_form]
Model.tire.search(load: true, page: params[:page], per_page: 5) do
query { string query, default_operator: "AND" } if query.present?
end
end
You can change the index_name(should be unique) and the analyzer
And Your controller would be like
def method_name
#results = Model.search( params ).results
end
You can use #results in your view. Hope this may help you.