I have the following code
PagedResultList res = myService.getPage(paginateParams, ...)
println res.size() // returns 2
println res.getTotalCount() // returns 1
getPage looks like:
def criteria = MyDomain.createCriteria()
criteria.list(max: paginateParams.max, offset: paginateParams.offset) { // max is 10, offset is 0, sortBy is updatedAt and sortOrder is desc
eq('org', org)
order(paginateParams.sortBy, paginateParams.sortOrder)
}
why do the two method return different values? The documentation doesn't explain the difference, but does mention that getTotalCount is for number of records
currently on grails 2.4.5
edits:
println on res prints out:
res: [
com.<hidden>.MyDomain: 41679f98-a7c5-4193-bba8-601725007c1a,
com.<hidden>.MyDomain: 41679f98-a7c5-4193-bba8-601725007c1a]
Yes, res has a SINGLE object twice - that's the bug I'm trying to fix. How do I know that? I have an primary key on MyDomain's ID, and when I inspect the database, it's also showing one record for this particular org (see my criteria)
edit 2: I found this comment (http://docs.grails.org/2.4.5/ref/Domain%20Classes/createCriteria.html)
listDistinct If subqueries or associations are used, one may end up
with the same row multiple times in the result set. In Hibernate one
would do a "CriteriaSpecification.DISTINCT_ROOT_ENTITY". In Grails one
can do it by just using this method.
Which, if I understand correctly, is their way of saying "list" method doesn't work in this scenario, use listDistinct instead but then they go on to warn:
The listDistinct() method does not work well with the pagination
options maxResult and firstResult. If you need distinct results with
pagination, we currently recommend that you use HQL. You can find out
more information from this blog post.
However, the blog post is a dead link.
Related: GORM createCriteria and list do not return the same results : what can I do?
Not related to actual problem after question edited but this quote seems useful
Generally PagedResultList .size() perform size() on resultList property (in-memory object represent database record), while .getTotalCount() do count query against database. If this two value didn't match your list may contain duplicate.
After viewing related issues (GORM createCriteria and list do not return the same results : what can I do?) I determined that there were several approaches:
Use grails projection groupBy('id') - doesn't work b/c i need the entire object
USe HSQL - Domain.executeQuery - actually this didn't work for my scenario very well because this returns a list, whereas criteria.list returns a PagedResultList from which I previously got totalCount. This solution had me learning HSQL and also made me break up my existing logic into two components - one that returned PagedResultList and one that didn't
Simply keep a set of IDs as I process my PagedResultList and make sure that I didn't have any duplicates.
I ended up going with option 3 because it was quick, didn't require me to learn a new language (HSQL) and I felt that I could easily write the code to do it and I'm not limited by the CPU to do such a unique ID check.
Related
Problem:
We're querying our database with a lot of entries. The overall application performance is okay. But I think it could be better if our bottleneck, querying a special table, could be done parallel.
We are using for that dynamic finders like:
Object.findAllByObjectCollectionAndParentIsNullAndDeletedByUserIsNull(objectCollection, [sort: 'attr1.number', fetch: [objectType: 'eager']])
I think about this solution: Dividing the query into 2 steps.
The first step loads only the Ids in a sorted way.
The second step (could be done in parallel threads) loads the objects itself by the id and inserts it into the resultset.
I already googled for some hints about that, but find nothing.
Is it possible, that it makes no sense?
Or is there already a adequate solution in grails standard/extensions?
If it makes sense and there is no solution: Can someone give me a hint implementing it? But, we need the ids in that sorted manner, explained in the example.
We're using grails 2.3.11, hibernate:3.6.10.13 with jdk 1.7 under it.
I found a solution!
ArrayList<Long> objectIds = Attribute.executeQuery("select o.id from Object o, ObjectType ot where o.objectCollection = :oc and o.parent is null and o.deletedByUser is null and oc.typeUsage = ot order by ot.nr asc", [oc: objectCollection])
Object[] tmp = new Object[objectIds.size()]
withPool(10, {
objectIds.eachWithIndexParallel {
Long objectId, i ->
Object o = Object.findById(objectId, [cache: true])
tmp[i] = a
}
})
for (Object a : tmp)
a.merge()
Its round about 3 times slower!
I think the reason is, that
we have to create the threads and databaseconnections too.
After loading the objects, they need to be merged into our local thread.
But in the end - we cant use it, beause the lazy loaded fields couldn't be loaded on demand later. So this could be only a solution for read-only Objects
I found a very strange behavior in our grails application today that i want to share with you.
We are using grails 2.3.11 on mysql 5.1.48.
We had a DomainObject.findById( id ) in one of your Controller actions.
We failed to check the id for a null value so DomainObject.findById( null )
would be called when no id is passed as an argument.
Normally DomainObject.findById( null )
will return null but there is a special condition that will yield other results!
If the controller action called before that inserted a new record in the database (lets call it Object B), regardless of the domain object stored, the DomainObject.findById( null ) will find the DomainObject with the same Id the Object B got on insert.
So when the controller action called before saved anything the findById(null) will return a row. And that row will have the same id the last inserted element got.
I am totally aware that using findById(null) is not the desired way to do it but I was quite shocked about the results it yielded. But returning any seemingly "random" result seems very strange to me.
I also want to note that DomainObject.get(null) will not suffer from this problem.
Anybody else witnessed this?
There is an active Jira pointing in this direction: https://jira.grails.org/browse/GRAILS-9628 but its not really describing this issue.
We don't really support passing null as an argument to a dynamic finder like that. Dynamic finders have explicit support for querying by null. Instead of DomainClass.findByName(null) you would call DomainClass.findByNameIsNull(). If you have a reference that may or may not be null, instead of passing that as an argument to a dynamic finder, the code can almost always be made cleaner by writing a criteria query or a "where" query that has a conditional in it.
I hope that helps.
Thx for your information scot.
I have further details. This behaviour is also altered by the underlying database.
While mysql suffers from this, maria-db (a mysql clone) does not!
So what happens is bound to the underlying database system.
That should not happen to an abstraction layer ....
I have been using parameters to query node indexes as such (using the rest api in java)-
final QueryResult<Map<String,Object>> result = engine.query("start nd=node:name_index(name={src}) return nd.age as age", MapUtil.map("src", "Susan");
However I haven't been able to get this to work for a collection of nodes/names. I have been trying something along the lines of-
final QueryResult<Map<String,Object>> result = engine.query("start nd=node:name_index(name={src}) return nd.age as age", MapUtil.map("src", Arrays.asList("Susan","Brian", "Ian"));
But it refuses to compile. I as wondering if there is something wrong in my syntax or that parameters are not designed to work in this context.
The name= syntax in the start is meant to do an index lookup on a property. It won't do an IN lookup. The way you can do this sort of lookup is like this (note it depends on Apache's StringUtils):
List<String> names = Arrays.asList("Susan","Brian", "Ian");
String luceneQuery = "name:("+StringUtils.join(names, ",")+")";
engine.query("start nd=node:name_index({luceneQuery}) return nd.age as age", MapUtil.map("luceneQuery", luceneQuery));
Just a note, this is the "legacy" index way of doing things. In 2.0 they've introduced label-based indexes, which work entirely differently.
Thanks a lot; though it would still only return a non empty answer when I added a space after the comma in line 2. I used-
String luceneQuery = "name:("+StringUtils.join(names, ", ")+")";
and it returned the age of one person. When I tried this:
String luceneQuery = "fs:(fs:"+ StringUtils.join(names, " OR fs:")+")";
it gave me all three ages. However, I am still unsure about whether this query will be able to leverage the usual advantages of parameters , i.e. will the engine be able to reuse the query and execution path the next time around (this time we may want to query for 4 names instead of 3)
I am new to learning and understanding how Hydration works, just wanted to point that out first. I'm currently able to Hydrate Select and Insert queries without any problems.
I am currently stuck on trying to Hydrate Update queries now. In my entity I have setup the get/set options for each type of column in my database. I've found that the ObjectProperty() Hydrator works best for my situation too.
However whenever I try to update only a set number of columns and extract via the hydrator I am getting errors because all the other options are not set and are returning null values. I do not need to update everything for a particular row, just a few columns.
For example in my DB Table I may have:
name
phone_number
email_address
But I only need to update the phone_number.
$entity_passport = $this->getEntityPassport();
$entity_passport->setPrimaryPhone('5551239876');
$this->getTablePassport()->update($this->getHydrator()->extract($entity_passport), array(
'employeeid' => '1'
));
This returns an error because setName() and setEmailAddress() are not included in this update and the query returns that the values cannot be null. But clearly when you look at the DB Table, there is data already there. The data that is there does not need to be changed either, only in this example does the PrimaryPhone() number.
I've been looking and reading documentation all over the place but I cannot find anything that would explain what I am doing wrong. I should note that I am only using Zend\Db (Not Doctrine).
I'm assuming I've missed something someplace due to my lack of knowledge with this new feature I'm trying to understand.
Perhaps you don't Hydrate Update queries... I'm sort of lost / confused. Any help would be appreciated. Thank you!
I think you're having a fundamental misconception of hydration. A hydrator simply populates an entity object from data (hydrate) and extracts data from an entity object (extract). So there are no separate hydrators for different types of queries.
In your update example you should first retrieve the complete entity object ($entity_passport) and then pass it to the TableGateway's update method. You would retrieve the entity by employeeid, since that's the condition you're using to update. So something like this:
$entity_passport = $passportMapper->findByEmployeeId(1);
$entity_passport->setPrimaryPhone('5551239876');
$this->getTablePassport()->update($this->getHydrator()->extract($entity_passport), array(
'employeeid' => $entity_passport->getId()
));
This is assuming you have some sort of mapper layer. Otherwise you could use your passport TableGateway (I assume that's what getTablePassport() returns, no?).
Otherwise, if you think retrieving the object is too much overhead and you just want to run the query you could use just a \Zend\Db\Sql\Sql object, ie:
$sql = new \Zend\Db\Sql\Sql($dbAdapter);
$update = $sql->update('passport')
->set(array('primary_phone' => $entity_passport->getPrimaryPhone()))
->where(array('employeeid' => $employeeId));
Edit:
Maybe it was a mistake to bring up the mapper, because it may cause more confusion. You could simply use your TableGateway to retrieve the entity object and then hydrate the returned row:
$rows = $this->getTablePassport()->select(array('employeeid' => 1));
$entity_passport = $this->getHydrator($rows->current());
[...]
Edit 2:
I checked your gist and I noticed a few things, so here we go:
I see that your getTablePassport indeed does return an object which is a subclass of TableGateway. You have already set up this class for it to use a HydratingResultset. This means you don't need to do any manual hydrating when retrieving objects using the gateway.
You also already implemented a Search method in that same class, so why not just use that? However I would change that method, because right now you're using LIKE for every single column. Not only is it very inefficient, but it will also give you wrong results, for example on the id column.
If you were to fix that method then you can simply call it in the Service object:
$this->getTablePassport->Search(array('employeeid' => 1));
Otherwise you could just implement a separate method in that tablegateway class, such as
public function findByEmployeeId($employeeId)
{
return $tableGateway->select(array('employeeid' => $employeeId));
}
This should already return an array of entities (or one in this specific case). P.S. make sure to debug and check what is actually being returned when you retrieve the entity. So print_r the entity you get back from the PassportTable before trying the update. You first have to make sure the retrieval code works well.
Is there any way to get a PagedResultList in grails without using criteria? I would like to avoid criteria as they are slightly more complex and make unit testing rather annoying. Code Below
def pagedResultList = MyDomainClass.createCriteria().list(max:10, offset:0)
{ order("id", "asc") }
//Below does not return pagedResultList
def aList = MyDomainClass.list(sort:"id", order:"asc", max:10, offset: 0)
PagedResultList is just used to wrap the results of Criteria-based queries (you can see its use in the source here). If you really want to use it, you could always just invoke the constructor directly, since it will handle any list. Of course the totalCount property (which is probably what you are interested in) will then be unset.
If the idea is to get both the paged results list and the total number of results, I'm not aware of any magic that can get both in one query (even the use of PagedResultList in the source linked above issues two queries).
This seems to break in Grails 2.x because there is no costructor that just takes a list. Compare http://grails.org/doc/2.1.0/api/grails/orm/PagedResultList.html and http://docs.huihoo.com/grails/1.3.7/api/grails/orm/PagedResultList.html