im using spring-data-elasticsearch for developing api with es as backend. and im using startScroll(long scrollTimeInMillis, SearchQuery searchQuery, Class<T> clazz) method for getting results from elasticsearch. but sort is not working.
i set sorting in the searchQuery as follows.
NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder().withIndices(<indices>).withTypes(<types>).withSort(<sort>)
and in i added the following for <sort>
new FieldSortBuilder("created_at").unmappedType("date").order(SortOrder.valueOf("ASC"))
i also tried putting sort in pageable like below.
NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder().withIndices(<indices>).withTypes(<types>).withPageable(<pageable>)
and in i added the following for <pageable>
Sort sortRequest = Sort.by(Sort.Direction.valueOf('ASC'), "created_at")
PageRequest.of(<pageNumber>, <pageSize>, sortRequest)
both isn't working.
and i start to think maybe scroll does not support sorting.
expected is for the result to show in order of created_at ASC.
but now its just randomly retrieved.
I encounter the same problem using 3.2, and I notice that 4.0 has resolved it
Related
I am using the Microsoft.Graph.Beta/5.14.0-preview SDK to query calendar events and I am migration from an 4.22 version of the SDK. Some of my queries uses $expand and $filter on expanded properties, but for some reason I can't figure out how to re-write these queryes in the new SDK where the query builder is moved to a configuration on the get part.
My old setup looks like this:
client
.Users[userId]
.CalendarView
.Request(queryOptions)
.Filter(filterQuery)
.Expand(expandOptions)
.Select(SelectQuery)
.GetAsync()
And the re-written looks like this:
client
.Users[userId]
.CalendarView
.GetAsync(cfg =>
{
cfg.QueryParameters.StartDateTime = from;
cfg.QueryParameters.EndDateTime = to;
cfg.QueryParameters.Filter = filterQuery;
cfg.QueryParameters.Select = SelectQuery;
})
But I can't find any option to expand anymore in the new SDK - I also tried just til filter on the nested objects and then look up the event ids, but I can't filter on nested objects without expanding them.
My query that don't work look like this:
https://graph.microsoft.com/beta/me/calendarView?startDateTime=2022-12-01T00:00:00&endDateTime=2022-12-31T00:00:00&$select=id&$filter=SingleValueExtendedProperties/any(c: c/Id eq 'String {543ec2fa-e468-4b54-be8e-787c70d4a79f} Name ParentEventId' and c/Value eq 'AAMkADJiZWQxZDVmLTViNDAtNGVjOS1hZjdkLWRlMjVlNTQwYzkzOABGAAAAAADJmx9CKBiKQoBAarsrrol5BwAZTSoj4SE7Rbx0wxVE_m1BAAAAAAENAAAZTSoj4SE7Rbx0wxVE_m1BAAADk9fzAAA=')
While the one that works, and does the expand and filter correctly looks like this:
https://graph.microsoft.com/beta/me/calendarView?startDateTime=2022-12-01T00:00:00&endDateTime=2022-12-31T00:00:00&$select=id&$filter=SingleValueExtendedProperties/any(c: c/Id eq 'String {543ec2fa-e468-4b54-be8e-787c70d4a79f} Name ParentEventId' and c/Value eq 'AAMkADJiZWQxZDVmLTViNDAtNGVjOS1hZjdkLWRlMjVlNTQwYzkzOABGAAAAAADJmx9CKBiKQoBAarsrrol5BwAZTSoj4SE7Rbx0wxVE_m1BAAAAAAENAAAZTSoj4SE7Rbx0wxVE_m1BAAADk9fzAAA=')&$expand=singleValueExtendedProperties($filter=id eq 'String {543ec2fa-e468-4b54-be8e-787c70d4a79f} Name ParentEventId')
I am not sure if this is the best way of querying SingleValueExtendedProperties but the old setup works - and i can't figure out how to migrate it in the new SDK.
We have been using spring-data-elasticsearch for 4.1.13 until recently for querying from elastic search. For grouping something we used aggregations
Consider a index of books. For each book there can be one or multipe authors
To get count of books by author we used TermsAggregationbuilder to get this grouping as shown below
SearchSourceBuilder builder = this.getQuery(filter, false);
String aggregationName = "group_by_author_id";
TermsAggregationBuilder aggregationBuilders =
AggregationBuilders.terms(aggregationName).field("authors");
var query =
new NativeSearchQueryBuilder()
.withQuery(builder.query())
.addAggregation(aggregationBuilders)
.build();
var result = elasticsearchOperations.search(query, EsBook.class, ALIAS_COORDS);
if (!result.hasAggregations()) {
throw new IllegalStateException("No aggregations found after query with aggregations!");
}
Terms groupById = result.getAggregations().get(aggregationName);
var buckets = groupById.getBuckets();
Map<Long, Integer> booksCount = new HashMap<>();
buckets.forEach(
bucket ->
booksCount.put(
bucket.getKeyAsNumber().longValue(), Math.toIntExact(bucket.getDocCount())));
return booksCount ;
We recently upgraded to spring-data-elasticsearch 4.4.2 and saw that there are some breaking changes.
First .addAggregations were replaced by withAggregations
Second unlike before I cant seem to directly get Terms and buckets after querying as
result.getAggregations().get(aggregationName); is no more possible and only other option I see is result.getAggregations().aggregations(). So am wondering if anyone has done the same. The documentation itself is so poor in Elastic search.
First .addAggregations were replaced by withAggregations
addAggregation(AbstractAggregationBuilder<?>) has been deprecated and should be replaced by withAggregations. This is not a breaking change.
The change in the returned value for SearchHits.getAggregations() is documented in the migration guides from 4.2 to 4.3 "Removal of org.elasticsearch classes from the API."
So from 4.3 on result.getAggregations().aggregations() returns the same that previously result.getAggregations() returned.
I'm using Legacy indexing (now called Manual indexing). After migration from Neo4j 2 to version 3 I have some problems with numeric sorting.
Example of correct statement in Neo4j 2:
queryContext.sort(new Sort(new SortField(AGE, SortField.INT, false)));
This stament should be changed for Neo4j 3 (Lucene 5):
queryContext.sort(new Sort(new SortField(AGE, SortField.Type.INT, false)));
But if you use this sort statement you will get an exception:
java.lang.IllegalStateException: unexpected docvalues type SORTED_SET for field 'firstName' (expected=SORTED). Use UninvertingReader or index with docvalues.
at org.apache.lucene.index.DocValues.checkField(DocValues.java:208)
at org.apache.lucene.index.DocValues.getSorted(DocValues.java:264)
at org.apache.lucene.search.FieldComparator$TermOrdValComparator.getSortedDocValues(FieldComparator.java:762)
at org.apache.lucene.search.FieldComparator$TermOrdValComparator.getLeafComparator(FieldComparator.java:767)
at org.apache.lucene.search.FieldValueHitQueue.getComparators(FieldValueHitQueue.java:183)
at org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector.getLeafCollector(TopFieldCollector.java:164)
at org.neo4j.kernel.api.impl.index.collector.DocValuesCollector.replayTo(DocValuesCollector.java:297)
at org.neo4j.kernel.api.impl.index.collector.DocValuesCollector.getTopDocs(DocValuesCollector.java:275)
at org.neo4j.kernel.api.impl.index.collector.DocValuesCollector.getIndexHits(DocValuesCollector.java:150)
at org.neo4j.index.impl.lucene.legacy.LuceneLegacyIndex.search(LuceneLegacyIndex.java:346)
at org.neo4j.index.impl.lucene.legacy.LuceneLegacyIndex.query(LuceneLegacyIndex.java:261)
at org.neo4j.index.impl.lucene.legacy.LuceneLegacyIndex.query(LuceneLegacyIndex.java:205)
at org.neo4j.index.impl.lucene.legacy.LuceneLegacyIndex.query(LuceneLegacyIndex.java:217)
at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.nodeLegacyIndexQuery(StateHandlingStatementOperations.java:1440)
at org.neo4j.kernel.impl.api.OperationsFacade.nodeLegacyIndexQuery(OperationsFacade.java:1162)
at org.neo4j.kernel.impl.coreapi.LegacyIndexProxy$Type$1.query(LegacyIndexProxy.java:83)
at org.neo4j.kernel.impl.coreapi.LegacyIndexProxy.query(LegacyIndexProxy.java:365)
I think this is caused by new added statement in Neo4j indexer class (Neo4j is indexing field for sorting automatically now?). See in:
org.neo4j.index.impl.lucene.legacy.IndexType CustomType addToDocument( Document document, String key, Object value )
new line:
document.add( instantiateSortField( key, value ) );
and method instantiateSortField is creating SortedSetDocValuesField
So I changed my code to:
queryContext.sort(new Sort(new SortedSetSortField(AGE, false)));
This runs OK but sorting is not working because numbers are sorted as string. I see that "value" parameter is String every time in method "addToDocument". I think the root cause is explained it this old comment:
see comment in class org.neo4j.index.impl.lucene.legacy.IndexType CustomType
// TODO We should honor ValueContext instead of doing value.toString() here.
// if changing it, also change #get to honor ValueContext.
Am I missing some new way how to index, search and sort data in Neo4j 3 or this is really problem that values are indexed as string in Neo4j?
Simple unit test for Neo4j 2 and Neo4j 3 can be downloaded
Solution added by MishaDemianenko at GH issue
Hi I'm new to rails and developing an application to pull results from database in preparation for charting. I have the following code in my controller:
#statistic = OutstandingWorkIndex.find_by_sql ["SELECT Result_Set.Set_Code, Request.Specimen_Number ,
DATEDIFF('hh',Result_Set.Date_Time_Booked_In,current_timestamp) as HrsIn FROM iLabTP.Outstanding_Work_Index, iLabTP.Result_Set Result_Set, iLabTP.Request
WHERE Outstanding_Work_Index.Request_Row_ID = Result_Set.Request_Row_ID and Outstanding_Work_Index.Request_Row_ID = Request.Request_Row_ID and Result_Set.Set_code=?
order by Result_Set.Date_Time_Booked_In DESC", params[:set_code].upcase]
What I'd like to do is count the number of records returned in addition to the object from above which I then use to create and XML stream of paired values or use the google charts java script api in the view.
Do I need to issue commands like:
#statistic = OutstandingWorkIndex.find_by_sql ["SELECT Result_Set.Set_Code, Request.Specimen_Number ,
DATEDIFF('hh',Result_Set.Date_Time_Booked_In,current_timestamp) as HrsIn
FROM iLabTP.Outstanding_Work_Index, iLabTP.Result_Set Result_Set, iLabTP.Request
WHERE Outstanding_Work_Index.Request_Row_ID = Result_Set.Request_Row_ID and Outstanding_Work_Index.Request_Row_ID = Request.Request_Row_ID and Result_Set.Set_code=?
order by Result_Set.Date_Time_Booked_In DESC", params[:set_code].upcase].**count**
And if so does this result in the query being reissued?
Thanks
You should do:
#size = #statistic.size
It's well explained here.
How do you paginate a query that has a join in symfony? I am trying this code but it times out:
$query=Doctrine_Query::create()
->select('c.name, d.phone')
->from('company c, companyDetails d')
->where('c.companyId=d.companyId');
$pager = new sfDoctrinePager('company',10);
$pager->setQuery($query);
$pager->setPage(1);
$pager->init();
$this->companys=$pager->getResults();
Your paginator code seems fine. I thk the problem is with your query.
Try running it without the pager and see the result.
If you continue facing issues with the query,Give this a try.
$query=Doctrine_Query::create()
->select('c.name, d.phone')
->from('company c')
->innerJoin('c.companyDetails d');
I havent tried it myself but it should work if your schema relations are defined the way i think.
One more thing, probably u already know that.
$pager->setPage(1);
should be something like
$pager->setPage($request->getParameter('page', 1));
I needed to change the count query in the pagination. To do this:
$pager = new Doctrine_Pager($query,$request->getParameter('page',1),10);
$pager->setCountQuery('SELECT COUNT(id) FROM items WHERE city_id='.$city.' AND category_id='.$category);
$this->pager=$pager;
The only thing you have to do is in your pagination helper, use:
$pagerRange = $pager->getRange('Sliding',array('chunk' => 5));
$pages = $pagerRange->rangeAroundPage();