Escaping # at symbol in Ruby Elastic Search gem? - ruby-on-rails

Question

I have the following code in the custom ES 'where' wrapper method
filter: { term: params }
Then we have a sample ES document that contains:
"emails" => { "email" => "johndoe#email.com" }
It is returned when my search is:
query.where("emails.email" => "johndoe")
but I get no results when:
query.where("emails.email" => "johndoe#email.com")
It seems like I have to escape at symbol somehow when using ES gem?

Answer 1

It's probably because your field is analyzed using the default standard analyzer and is thus tokenized at the # sign.
You can see what ES has indexed by running the command below:
curl -XGET 'localhost:9200/_analyze?analyzer=standard&pretty' -d 'johndoe#email.com'
And the result is
{
"tokens" : [ {
"token" : "johndoe",
"start_offset" : 0,
"end_offset" : 7,
"type" : "<ALPHANUM>",
"position" : 1
}, {
"token" : "email.com",
"start_offset" : 8,
"end_offset" : 17,
"type" : "<ALPHANUM>",
"position" : 2
} ]
}
As you can see, your email field has been tokenized as two different tokens and that's probably why searching for johndoe works, while searching for the full email address doesn't.
There are a few ways out from here, but one way that would work is to create your own analyzer based on a pattern_capture token filter and use it as index_analyzer for your emails.email field.
{
"settings" : {
"analysis" : {
"filter" : {
"email" : {
"type" : "pattern_capture",
"preserve_original" : 1,
"patterns" : [ "([^#]+)", "(\\p{L}+)", "(\\d+)", "#(.+)" ]
}
},
"analyzer" : {
"email" : {
"tokenizer" : "uax_url_email",
"filter" : [ "email", "lowercase", "unique" ]
}
}
}
},
"mappings": {
"emails": {
"properties": {
"email": {
"type": "string",
"analyzer": "email" <-- use the analyzer here
}
}
}
}
}
At indexing time, that analyzer will produce all of the following tokens, which will allow you to search for any parts of your email address:
johndoe#email.com
johndoe
email.com
email
com