While I was looking at the documentation for query parameters here, I noticed that there were two query parameters that seemingly did the exact same thing: filter and search.
I'm just wondering what the difference is between them and when is one used over the other.
While they're similar, they operate a little differently.
$search uses Keyword Query Language (KQL) and is only supported by message and person collections (i.e. you can't use $search on most endpoints). By default, it searches multiple properties. Most importantly, $search is a "contains" search, meaning it will look for your search word/phrase anywhere within a string.
For example, /messages?$search="bacon" will search for the word "bacon" anywhere in the from, subject, or body properties.
Unlike $search, the $filter parameter only searches the specified property and does not support "contains" search. It also works with just about every endpoint. In most places, it supports the following operators: equals (eq), not equals (ne), greater than (gt), greater than or equals (ge), less than (lt), less than or equals (le), and (and), or (or), not (not), and (on some endpoints) starts with (startsWith).
For example, /messages?$filter=subject eq 'bacon' will return only messages where the subject is "bacon".
Both search and filter reduce the result set that you ultimately receive, however they operate in different ways.
Search operates on the query against the entire graph and reduces the amount of information a search query returns. This is often optimized for queries that search is good at, e.g. performing searches for items that can be indexed.
Filter operates on the much smaller result set returned by the search to provide more fine grain filtering. Separating this out allows filtering to perform tasks that would not be performant against the full collection.
This is indicated in Microsoft's documentation:
Search: Returns results based on search criteria.
Filter: Filters results (rows). (results that could be returned by search)
For performance purposes, it's good to use both if you can, search to narrow the results (e.g. using search indexes) and then do fine grain filtering on the returned results.
Related
I'm using a query to execute lucene search over a property in a Neo4j database. As I want to query by several different strings I've come with the following query
CALL db.index.fulltext.queryNodes('descs', 'abc') YIELD node
with collect(node) as matches1
CALL db.index.fulltext.queryNodes('descs', 'def') YIELD node
with matches1, collect(node) as matches2
RETURN apoc.coll.intersection(matches1,matches2) AS res
Query works fine sometimes but seems to behave weirdly when too many results are returned in any of the calls (still don't know the actual limit). More precisely, if any of the queries returns a big list, then the query returns just "[]". It seems to work perfectly if query returns a list with few nodes (or no nodes, which provides no results due to doing an intersection with an empty list). Is there any configurable (or non-configurable) limit on apoc.coll.intersection function or any of the other calls? Things are aggravated when using the '~' operand, as it usually returns more results. This means that similar queries will or will not work as expected depending on how many matches provides the queryNodes call.
Also, as there could be any number of words to search, is there a way to generalize this kind of query? queryNodes seems to not work well with spaces inside the text to search. Regex could be an option but it doesn't really work well when dealing with accents and/or searching for multiple words in any given order.
I'm using the YouTube Data API's search.list method to return a list of videos by date. I'm interested in filtering out certain content without having to specify a search term. The documentation specifies that you can use the - operator as a Boolean NOT, but this only seems to work if I precede that with a search term, meaning I can do this:
q:'food -pizza'
which will return results for the query term 'food' but not 'pizza'. Now say I want it to return any result excluding pizza you'd think this would work:
q:'-pizza'
but this returns an empty Array (no results). Am I doing this wrong? is there a way to exclude certain terms without having to specify a specific search term to include before hand?
While searching on azure using Rest API provided by Microsoft Search API
Not behaving correctly when search string contains '#'.
Example: I've 3 rows in Azure Document
CES
CES#123
CES#1234
When My search string was CES* then all 3 were the result.
When My Search string was CES#123* then only one exact matching record was in result.
When My Search string was CES#* then there was no result.
As per my requirement in case of "CES#*" search string, all 3 records should be part of result set.
I've tried " "(space) in replacement of # it works, but my data contains # for search I need to maintain this.
I'm using SearchMode:Any.
This behavior is expected.
Query terms of prefix queries are not analyzed. Therefore, in your example with "CES#*" you are searching for term CES# while the # sign was stripped from the terms in the index: CES, 123, 1234.
Here is an excerpt from the How full text search works in Azure Search article:
Exceptions to lexical analysis
Lexical analysis applies only to query types that require complete
terms – either a term query or a phrase query. It doesn’t apply to
query types with incomplete terms – prefix query, wildcard query,
regex query – or to a fuzzy query. Those query types, including the
prefix query with term air-condition* in our example, are added
directly to the query tree, bypassing the analysis stage. The only
transformation performed on query terms of those types is lowercasing.
I am working on a model using wikipedia topics' names for my experiments in full-text index.
I set up and index on 'topic' (legacy), and do a full text search for : 'united states':
start n=node:topic('name:(united states)') return n
The first results are not relevant at all:
'List of United States National Historic Landmarks in United States commonwealths and territories, associated states, and foreign states'
[...]
and the actual 'united states' is buried deep down the list.
As such, it raises the problem that, in order to find the best match (e.g. levershtein, bi-gram, and so on algorithms) on results, you first must fetch all the items matching the pattern.
That would be a serious constraint, cause just in this case I have 21K rows, ~4 seconds.
Which algorithms does neo4j use to order the results of a full-text search (START)?
Which rationale does it use to sort result and how to change it using cypher?
In the doc is written to use JAVA api to apply sort() - it would be very useful to have a tutorial for appointing to which files to modify and also to know which ranking rationale is used before any tweak.
EDITED based on comments below - pagination of results is possible as:
n=node:topic('name:(united states)') return n skip 10 limit 50;
(skip before limit) but I need to ensure first results are meaningful before pagination.
I don't know which order algorithms does lucene use to order the results.
However, about the pagination, if you change the order of limit and skip like follows, should be ok.
start n=node:topic('name:(united states)') return n skip 10 limit 50 ;
I would also add that if you are performing full-text search maybe a solution like solr is more appropriate.
For just a lucene index lookup with scoring you might be better off with this:
http://neo4j.com/docs/stable/rest-api-indexes.html#rest-api-find-node-by-query
I'm having an issue with Lucene's Term [Boosting][1] query syntax, specifically in Ruby on Rails via the sunspot_rails gem. This is whereby you can specify the weight of a specific term during a query, and is not related to the weighting of a particular field.
The HTML query generated by sunspot uses the qf parameter to specify the fields to be searched as configured, and the q parameter for the query itself. When the caret is added to a search term to specify a boost (i.e. q=searchterm^5) it returns no results, even though results would be returned without the boost term.
If, on the other hand, I create an HTTP query manually and manually specify the field to search (q=title_texts:searchterm^5), results are returned and scores seem affected by the boost.
In short, it appears as though query term boosting doesn't work in conjunction with fields specified with qf.
My application calls for search across several fields, using the respective boosts associated to those fields, conditionally in turn with boosting on individual terms of a query.
Any insight?
[1]: http://lucene.apache.org/java/2_9_1/queryparsersyntax.html#Boosting a Term
Sunspot uses the dismax parser for fulltext search, which eschews the usual Lucene query syntax in favor of a limited (but user-input-friendly) query syntax combined with a set of additional parameters (such as qf) that can be constructed by the client application to tune how search works. Sunspot provides support for per-field boost using the boost_fields method in the fulltext DSL:
http://outoftime.github.com/sunspot/docs/classes/Sunspot/DSL/Fulltext.html#M000129
The solution I have found is to use DisMax, but adding the bq parameter with a boolean string with the boosted terms therein.