Elasticsearch/tire Nested Queries with persistant objects - ruby-on-rails

I'm trying to use Tire to perform a nested query on a persisted model. The model (Thing) has Tags and I'm looking to find all Things tagged with a certain Tag
class Thing
include Tire::Model::Callbacks
include Tire::Model::Persistence
index_name { "#{Rails.env}-thing" }
property :title, :type => :string
property :tags, :default => [], :analyzer => 'keyword', :class => [Tag], :type => :nested
end
The nested query looks like
class Thing
def self.find_all_by_tag(tag_name, args)
self.search(args) do
query do
nested path: 'tags' do
query do
boolean do
must { match 'tags.name', tag_name }
end
end
end
end
end
end
end
When I execute the query I get a "not of nested type" error
Parse Failure [Failed to parse source [{\"query\":{\"nested\":{\"query\":{\"bool\":{\"must\":[{\"match\":{\"tags.name\":{\"query\":\"TestTag\"}}}]}},\"path\":\"tags\"}},\"size\":10,\"from\":0,\"version\":true}]]]; nested: QueryParsingException[[test-thing] [nested] nested object under path [tags] is not of nested type]; }]","status":500}
Looking at the source for Tire it seems that mappings are created from the options passed to the "property" method, so I don't think I need a separate "mapping" block in the class. Can anyone see what I am doing wrong?
UPDATE
Following Karmi's answer below, I recreated the index and verified that the mapping is correct:
thing: {
properties: {
tags: {
properties: {
name: {
type: string
}
type: nested
}
}
title: {
type: string
}
}
However, when I add new Tags to Thing
thing = Thing.new
thing.title = "Title"
thing.tags << {:name => 'Tag'}
thing.save
The mapping reverts to "dynamic" type and "nested" is lost.
thing: {
properties: {
tags: {
properties: {
name: {
type: string
}
type: "dynamic"
}
}
title: {
type: string
}
}
The query fails with the same error as before. How do I preserve the nested type when adding new Tags?

Yes, indeed, the mapping configuration in property declarations is passed on in the Persistence integration.
In a situation like this, there's always the and and only first question: how does the mapping look like for real?
So, use eg. the Thing.index.mapping method or the Elasticsearch's REST API: curl localhost:9200/things/_mapping to have a look.
Chances are, that your index was created with the dynamic mapping, based on the JSON you have used, and you have changed the mapping later. In this case, the index creation logic is skipped, and the mapping is not what you expect.
There's a Tire issue opened about displaying warning when the index mapping is different from the mapping defined in the model.

Related

How do I override the root key in a Grape API response payload?

module Entities
class StuffEntity < Grape::Entity
root 'stuffs', 'stuff'
...
How can I DRY up my code by reusing this entity while still having the flexibility to rename the root keys ('stuffs' and 'stuff') defined in the entity?
I might need to do this in a scenario where I'm exposing a subset of a collection represented by an existing entity or exposing an associated collection that can be represented by an existing entity.
Hiding the root key when you're exposing an associated object or collection
Let's say I have an object with a name attribute and some collection of scoped_stuff that I want to expose as some_stuffs. I could do that with an entity like this:
module Entities
class CoolStuffEntity < Grape::Entity
root 'cool_stuffs', 'cool_stuff'
expose :some_stuffs, documentation: {
type: Entities::StuffEntity
} do |object, _options|
Entities::StuffEntity.represent(
object.class.scoped_stuff,
root: false
)
end
expose :name, documentation: { type: 'string' }
end
end
Passing root: false to the represent method ensures that the nested association is represented without a root key. Here are what the representations look like with and without that argument:
# Without root: false
cool_stuff: {
some_stuffs: {
stuffs: [ /* collection represented by StuffEntity */ ]
},
name: 'Something'
}
# With root: false
cool_stuff: {
some_stuffs: [ /* collection represented by StuffEntity */ ],
name: 'Something'
}
In this instance, passing root: false ensures that the nested entity's root key isn't included in our representation.
Setting a root key name when presenting an entity with no defined root
Let's say we have this entity where we did not specify root:
module Entities
class StuffEntity < Grape::Entity
expose :name, documentation: { type: 'string' }
end
end
The serializable hash for an object represented with this entity will look like: { name: 'Object name' }
In our API, we can specify the response key like so:
get do
stuff_object = Stuff.find_by(user_id: current_user)
present stuff_object,
with: Entities::StuffEntity,
root: :stuff
end
So that our response will look like this: { stuff: { name: 'Object name' } }
Note that 'root' accepts string and symbol arguments here.
If you want to rename the root key in your API response
So what if I have an entity where I specified a root key and I want the key in my response to be different (e.g., exposing a subset of the collection)? Instead of using present, I can use represent again. Except this time, instead of disabling the root key by passing 'false', I can give it a key name:
get do
my_stuff = Stuff.find_by(user_id: current_user)
Entities::StuffEntity.represent(
my_stuff,
root: :my_stuff
)
end

trailblazer reform gem, how to handle this type of input validation?

We are looking at use the reform gem for validating input.
One of the issues we are facing is that we accept input in this format:
params = {
records: {
"record-id-23423424": {
name: 'Joe Smith'
}
"record-id-43234233": {
name: 'Jane Doe'
}
"record-id-345234555": {
name: 'Fox trot'
}
"record-id-34234234": {
name: 'Alex'
}
}
}
so if we were to create reform class
class RecordForm < Reform::Form
property :records
validates :records, presence: true
# ?????????
end
How do we validate the contents of the records to make sure each one has a name? The record-id-values are not known ahead of time.
Reform currently doesn't allow dynamic properties, and actually, it's not planned since Reform is supposed to be a UI-specific form object.
The solution would be to pre-parse your input into something what Laura suggests. You could then have nested properties for each field.
collection :records do
property :id # manually parsed
property :name
end

Using filters with elasticsearch and tire to mimic named scopes in rails

I have an Exhibition model that has three named scopes:
scope :opening, -> { where(start_date: (Date.today .... }
scope :closing, -> { where(end_date: .... }
scope :reception, -> { where("reception is not null")... }
I have the following tire mapping inside the same model:
tire.mapping do
...
indexes :start_date, type: "date"
indexes :end_date, type: "date"
indexes :reception, type: "date"
...
end
And my search method:
def self.search(params)
tire.search(load: true) do
query { string params[:query] } if params[:query].present?
...
end
Inside my exhibitions_controller.rb I have the following (and the same for closing and reception)
def opening_this_week
#exhibitions = Exhibition.opening.search(params)
...
end
When I actually search for one of the indexed fields I get back the correct response of just the exhibitions that meet the query and are opening this week, however, when I want all opening exhibitions and do not pass a search query I get ALL exhibitions, not just those that are opening this week.
What am I doing wrong here? How do I transition from a named scope to a facet? Or should I be doing it as a filter?

Custom analyzer in Tire Elastic not working with Mongoid

I am still doing something wrong.
Could somebody pls help me?
I want to create a custom analyzer with ascii filter in Rails + Mongoid.
I have a simple model product which has field name.
class Product
include Mongoid::Document
field :name
settings analysis: {
analyser: {
ascii: {
type: 'custom',
tokenizer: 'whitespace',
filter: ['lowercase','asciifolding']
}
}
}
mapping do
indexes :name, analyzer: 'ascii'
end
end
Product.create(name:"svíčka")
Product.search(q:"svíčka").count #1
Product.search(q:"svicka").count #0 can't find - expected 1
Product.create(name:"svicka")
Product.search(q:"svíčka").count #0 can't find - expected 1
Product.search(q:"svicka").count #1
And when I check the indexes with elasticsearch-head I expected that the index is stored without accents like this "svicka", but the index looks like this "Svíčka".
What am I doing wrong?
When I check it with API it looks OK:
curl -XGET 'localhost:9200/_analyze?tokenizer=whitespace&filters=asciifolding' -d 'svíčka'
{"tokens":[{"token":"svicka","start_offset":0,"end_offset":6,"type":"word","position":1}]}
http://localhost:9200/development_imango_products/_mapping
{"development_imango_products":{"product":{"properties":{"name":{"type":"string","analyzer":"ascii"}}}}}
curl -XGET 'localhost:9200/development_imango_products/_analyze?field=name' -d 'svíčka'
{"tokens":[{"token":"svicka","start_offset":0,"end_offset":6,"type":"word","position":1}]}
You can check how you are actually indexing your document using the analyze api.
You need also to take into account that there's a difference between what you index and what you store. What you store is returned when you query, and it is exactly what you send to elasticsearch, while what you index determines what documents you get back while querying.
Using the asciifolding is a good choice for you usecase, it should return results either query ing for svíčka or svicka. I guess there's just a typo in your settings: analyser should be analyzer. Probably that analyzer is not being used as you'd expect.
UPDATE
Given your comment you didn't solve the problem yet. Can you check what your mapping looks like (localhost:9200/index_name/_mapping)? The way you're using the analyze api is not that useful since you're manually providing the text analysis chain, but that doesn't mean that chain is applied as you'd expect to your field. Better if you provide the name of the field like this:
curl -XGET 'localhost:9200/index_name/_analyze?field=field_name' -d 'svíčka'
That way the analyze api will rely on the actual mapping for that field.
UPDATE 2
After you made sure that the mapping is correctly submitted and everything looks fine, I noticed you're not specifying the field that you want to to query. If you don't specify it you're querying the _all special field, which contains by default all the field that you're indexing, and uses by default the StandardAnalyzer. You should use the following query: name:svíčka.
elasticsearch needs settings and mapping in a single api call. I am not sure if its mentioned in tire docs, but I faced a similar problem, using both settings and mapping when setting up tire. Following should work:
class Product
include Mongoid::Document
# include tire stuff
field :name
settings(self.tire_settings) do
mapping do
indexes :name, analyzer: 'ascii'
end
end
# this method is just created for readablity,
# settings hash can also be passed directly
def self.tire_settings
{
analysis: {
analyzer: {
ascii: {
type: 'custom',
tokenizer: 'whitespace',
filter: ['lowercase','asciifolding']
}
}
}
}
end
end
Your notation for settings/mappings is incorrect, as #rubish suggests, check documentation in https://github.com/karmi/tire/blob/master/lib/tire/model/indexing.rb (no question the docs should be better)
Always, always, always check the mapping of the index to see if your desired mapping has been applied.
Use the Explain API, as #javanna suggests, to check how your analyzing chain works quickly, without having to store documents, check results, etc.
Please note that It is very important to add two lines in a model to make it searchable through Tire. Your model should look like
class Model
include Mongoid::Document
include Mongoid::Timestamps
include Tire::Model::Search
include Tire::Model::Callbacks
field :filed_name, type: String
index_name "#{Tire::Model::Search.index_prefix}model_name"
settings :analysis => {
:analyzer => {
"project_lowercase_analyzer" => {
"tokenizer" => "keyword",
"filter" => ["lowercase"],
"type" => "custom"
}
},
} do
mapping do
indexes :field_name, :boost => 10, :type => 'String', :analyzer => 'standard', :filter => ['standard', 'lowercase','keyword']
end
end
def self.search( params = {} )
query = params[:search-text_field_name_from_form]
Model.tire.search(load: true, page: params[:page], per_page: 5) do
query { string query, default_operator: "AND" } if query.present?
end
end
You can change the index_name(should be unique) and the analyzer
And Your controller would be like
def method_name
#results = Model.search( params ).results
end
You can use #results in your view. Hope this may help you.

Can MongoDB automatically generate special indexes for geolocation data?

I currently have a Mongoid model in a Ruby on Rails application as such:
class Listen
include Mongoid::Document
field :song_title, type: String
field :song_artist, type: String
field :loc, :type => Array
field :listened_at, type: Time, default: -> { Time.now }
index(
[[:loc, Mongo::GEO2D]], background: true
)
end
When I try to query the collection for example
listens = Listen.where(:loc => {"$within" => {"$centerSphere" => [location, (radius.fdiv(6371))]}})
I am returned the error (locations have been blanked out, the X's are not returned)
Mongo::OperationFailure (can't find special index: 2d for: { loc: { $within: { $centerSphere: [ [ XX.XXXXXXX, X.XXXXXXX ], 0.0001569612305760477 ] } } }):
I know I can create the indexed through a rake task such as rake db:mongoid:create_indexes but I don't want to have to do this every time a model is created. Is there any way for the model to create this automatically on insert to the collection?
Nope there is no way.
You must create indexes (not just Geo) once, to use it.

Resources