I am new to elastisearch and I just set it up and tried default search. I am using elasticsearch rails gem. I need to write custom query with priority search (some fields in table are more important then others, etc. title, updated_at in last 6 months...). I tried to find explanation or tutorial for how to do this but nothing seems understandable. Can anyone help me with this, soon better.
Never having used the ruby/elasticsearch integration, it doesn't seem too hard... The docs here show that you'd want to do something like this:
client.search index: 'my-index', body: { query: { match: { title: 'test' } } }
To do a basic search.
The ES documentation here shows how to do a field boosted query:
{
"multi_match" : {
"query" : "this is a test",
"fields" : [ "subject^3", "message" ]
}
}
Putting it all together, you'd do something like this:
client.search index: 'my-index', body: { query: { multi_match : {
query : "this is a test",
fields : [ "subject^3", "message" ]
} } }
That will allow you to search/boost on fields -- in the above case, the subject field is given 3 times the score of the message field.
There is a very good blog post about how to do advanced scoring. Part of it shows an example of adjusting the score based on a date:
...
"filter": {
"exists": {
"field": "date"
}
},
"script": "(0.08 / ((3.16*pow(10,-11)) * abs(now - doc['date'].date.getMillis()) + 0.05)) + 1.0"
...
I have done in php, Never used the gem from Ruby on rails. Here you can give the priority for the fields using the caret (^) notation.
Example:- Suppose if we have fields namely name, email, message and address in table and the priority should be given for the name and message then you can write as below
> { "multi_match" : {
> "query" : "this is a test",
> "fields" : [ "name^3", "message^2".... ] } }
Here name has 3 times higher priority than other fields and message has got 2 times higher priority than other fields.
Related
I want to dummy in the real-time database in this structure:
{
"0" : {
"0c1592ca-0fa5-43b9-88d2-c9cd77b30611" : {
"token" : "0cu9CJPb_DIUfbr-Ay8vh6:-KQXn....",
"member_id" : "123456789102",
"update_at" : "2021/06/14 08:08:08"
},
"<uid>" : {
"token" : "167 random characters",
"member_id" : "12 random numbers",
"update_at" : "YYYY/mm/DD HH:mm:ss"
}
},
"1" : {
....
},
"2" : {
....
},
.....
"9" : {
....
}
}
A record is like this:
"36 random characters" : {
"token" : "167 random characters",
"member_id" : "12 random numbers",
"update_at" : "YYYY/mm/DD HH:mm:ss"
}
I've tried to import JSON files from the firebase console for a million records per node. But I got a crash from the second node, like the image below. I can't import easily like before.
Is there any other way that I can dummy 10 million child nodes like above, faster and stable?
https://i.stack.imgur.com/nuBoM.png
The error message says that the JSON is invalid, so you might want to pass it through a JSON validator like https://jsonlint.com/.
Aside from that, I can imagine that you browser, the console, or the server runs into memory problems with this number of nodes in one write (see limits). I recommend using the API to instead read the JSON file locally, and then add it to Firebase in chunks, or use a tool like https://github.com/FirebaseExtended/firebase-import.
Also see https://www.google.com/search?q=firebase+realtime+database+upload+large+JSON
I am trying to get a unique document count in an index based on an id property via elastic search web API. The thing is that I have millions of entries. How can I scroll on an aggregation ?
this is the url:
http://my.servers.ip:9200/index_name/doc_type/_search?scroll=1m
And this is the body:
{
"_source": "false",
"aggs" : {
"Ids" : {
"terms" : {
"field" : "somePropertyIWantToGoupBy",
"size" : 100
},
"aggs": {
"unique": {
"cardinality": {
"field": "someCategoryIWantUniqueCount"
}
}
}
}
},"size":0
}
I get the scrollId , but on the next call with scroll id I'll get the next 100 aggregations, instead I get an empty result set.
Is it possible to scroll on aggregations ?
What am I doing wrong ?
There's no way to paginate terms aggregation.
You should use Composite Aggregation but it's a beta aggregation and might be removed or changed in the future...
I have elasticsearch set up for searching across a products catalog's variants. Basically where:
Product has_many variants
Variant belongs_to product
And the variant index json / mapping contains the product name.
I am trying to search variants, grouped by product id, bucket size of 1. I am able to do it and sort by min price, max price, etc.
This works:
POST /variants/_search?size=0
{
"aggs" : {
"min_price" : { "min" : { "field" : "price" } }
}
}
This is (sort of) what I need next:
POST /variants/_search?size=0
{
"aggs" : {
"product_name" : { "sort by product_name asc / desc" }
}
}
My last task is about sorting them alphabetically, but I dont seem to be able to sort by a keyword field (asc/desc) using an aggregator.
In ES 6.0, you could do this. Note that size limits how many are returned, and the more you request the more expensive the query will be to execute. So if you really need many thousands you will probably want to try a different approach. Probably something where you created a separate rolled up index for products that you could search/sort instead of trying to do it through aggregations.
GET /variants/_search
{
"size": 0,
"aggs" : {
"product_name" : {
"terms" : {
"field" : "product_name",
"size": 1000,
"order" : { "_key" : "asc" }
}
}
}
}
Reference:
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order
I have this json model:
model/data.json
{
"orders" : [
{
"header" : { "id" : "00001", "description" : "This is the first order" },
"items" : [
{ "name" : "Red Book","id" : "XXYYZZ" },
{ "name" : "Yellow Book", "id" : "AACCXX" },
{ "name" : "Black Book", "id" : "UUEEAA" },
]
},
{
// another order with header + items
},
.....
]
}
and I'm assigning it onInit to the view, like this:
var model = new sap.ui.model.json.JSONModel("model/data.json");
sap.ui.getCore().setModel(reqModel);
I'm trying to display a list of orders in the first view (showing the id), like this:
var list = new sap.m.List({
id: "mainList",
items: []
});
var items = new sap.m.ActionListItem({
text : "{id}",
press : [ //click handler, onclick load the order details page ]
});
list.bindItems("/orders", items);
.... // add list to the page etc etc
What I cannot do, is connect each order to its header->id.. I tried
text: "/header/{id}"
text: "{/header/id}"
in the items declaration, and
list.bindItems("/orders/header", items)
in the list binding, but none of them works.. The id value is not displayed, even though a "blank" list item is shown..
Any idea? What am I doing wrong?
Thank you
The solution was one of those I tried (but I don't know why it didn't work at that time)
text: "{/header/id}"
The ListItem acts as a Template for a list/array of objects. That's why you bind it against an array structure in your data:
list.bindItems("/orders", itemTemplate)
That makes bindings of the ListItem relative to /orders and therefore your item should look like this without leading '/' (absolute paths would look like this /orders/0/header/id asf.):
var itemTemplate = new sap.m.ActionListItem({
text : "{header/id}",
press : [ //click handler, onclick load the order details page ]
});
Not quite sure how you made it work the way you have shown... May be it's not as picky as I thought.
Btw: For whatever reason the ResourceModel builds an exception of that syntax. You can always omit the leading '/' when dealing with ResourceModels (probably because they do not allow nested structures).
BR
Chris
Cannot add comments yet, therefore an answer to you solved Problem, that could answer the initial problem. (And inform People using that example in any way)
In the current code listing you use the variable "reqModel" to set the model, but the variable with the model in it is named "model" in the line before. Maybe that was the first reason why both of your examles would not work?
Perhaps this error was cleared on rewriting some passages while testing.
greetings! -nx
I have a document that has an array:
{
_id: ObjectId("515e10784903724d72000003"),
association_chain: [
{
name: "Product",
id: ObjectId("4e1e2cdd9a86652647000003")
}
],
//...
}
I'm trying to search the collection for documents where the name of the first item in the association_chain array matches a given value.
How can I do this using Mongoid? Or if you only know how this can be done using MongoDB, if you post an example, then I could probably figure out how to do it with Mongoid.
Use the positional operator. You can query the first element of an array with .0 (and the second with .1, etc).
> db.items.insert({association_chain: [{name: 'foo'}, {name: 'bar'}]})
> db.items.find({"association_chain.0.name": "foo"})
{ "_id" : ObjectId("516348865862b60b7b85d962"), "association_chain" : [ { "name" : "foo" }, { "name" : "bar" } ] }
You can see that the positional operator is in effect since searching for foo in the second element doesn't return a hit...
> db.items.find({"association_chain.1.name": "foo"})
>
...but searching for bar does.
> db.items.find({"association_chain.1.name": "bar"})
{ "_id" : ObjectId("516348865862b60b7b85d962"), "association_chain" : [ { "name" : "foo" }, { "name" : "bar" } ] }
You can even index this specific field without indexing all the names of all the association chain documents:
> db.items.ensureIndex({"association_chain.0.name": 1})
> db.items.find({"association_chain.0.name": "foo"}).explain()
{
"cursor" : "BtreeCursor association_chain.0.name_1",
"nscanned" : 1,
...
}
> db.items.find({"association_chain.1.name": "foo"}).explain()
{
"cursor" : "BasicCursor",
"nscanned" : 3,
...
}
Two ways to do this:
1) if you already know that you're only interested in the first product name appearing in "association_chain", then this is better:
db.items.find("association_chain.0.name":"something")
Please note that this does not return all items, which mention the desired product, but only those which mention it in the first position of the 'association_chain' array.
If you want to do this, then you'll need an index:
db.items.ensureIndex({"association_chain.0.name":1},{background:1})
2) if you are looking for a specific product, but you are not sure in which position of the association_chain it appears, then do this:
With the MongoDB shell you can access any hash key inside a nested structure with the '.' dot operator! Please note that this is independent of how deeply that key is nested in the record (isn't that cool?)
You can do a find on an embedded array of hashes like this:
db.items.find("association_chain.name":"something")
This returns all records in the collection which contain the desired product mentioned anywhere in the association_array.
If you want to do this, you should make sure that you have an index:
db.items.ensureIndex({"association_chain.name":1},{background: 1})
See "Dot Notation" on this page: http://docs.mongodb.org/manual/core/document/
You can do this with the aggregation framework. In the mongo shell run a query that unwinds the documents so you have a document per array element (with duplicated data in the other fields), then group by id and any other field you want to include, plus the array with the operator $first. Then just include the $match operator to filter by name or mongoid.
Here's the query to match by the first product name:
db.foo.aggregate([
{ $unwind:"$association_chain"
},
{
$group : {
"_id" : {
"_id" : "$_id",
"other" : "$other"
},
"association_chain" : {
$first : "$association_chain"
}
}
},
{ $match:{ "association_chain.name":"Product"}
}
])
Here's how to query for the first product by mongoid:
db.foo.aggregate([
{ $unwind:"$association_chain"
},
{
$group : {
"_id" : {
"_id" : "$_id",
"other" : "$other"
},
"association_chain" : {
$first : "$association_chain"
}
}
},
{ $match:{ "association_chain.id":ObjectId("4e1e2cdd9a86652647000007")}
}
])