I have an application in production use, when a user goes to a proposal index page it takes a very long time and sometimes times out. I've narrowed down the issue to be a SQL statement that is selecting all the Proposal objects. The problem is the Proposal object has many images (byte[]) stored in memory that aren't being used in the index page. These images are huge thus causing the problem.
What are the different ways I can optimize this query in Grails to remove the attributes I don't need on that page or only add the attributes I have in my GSP?
Here is the controller code (scaffolded):
def index(Integer max) {
params.max = Math.min(max ?: 10, 100)
respond Proposal.list(params), model:[proposalInstanceCount: Proposal.count()]
}
Thanks!
I wrote a plugin for this scenario, see http://grails.org/plugin/lazylob
Another option is to refactor the domain class into two. Put the image data in the new domain class:
class ProposalImage {
byte[] image
}
and reference it from the Proposal class:
class Proposal {
ProposalImage proposalImage
// other properties
}
Since references are lazy by default, GORM will only load the image data from the new domain class if you specifically refer to it.
EDIT (updated with subselect approaches):
You can also use custom queries to select a subset of the properties. Probably the most convenient would be using "select new map" in an HQL query:
def results = Proposal.executeQuery(
'select new map(prop1 as prop1, prop2 as prop2) from Proposal',
[max:params.max as int, params.offset as int])
This is convenient because each element in the results list is a map keyed with the property names, so it will look the same as a real Proposal instance in the GSP.
Another option if you prefer criteria queries is to use projections to limit which properties are returned:
def results = Proposal.withCriteria {
projections {
property 'prop1'
property 'prop2'
}
maxResults(params.max as int)
firstResult(params.offset as int)
}
Each item in the results is an Object[] array and each element in the array is the actual type of the property. You would need to manually build a list of maps, e.g.
results = results.collect { result -> [prop1: result[0], prop2: result[1]] }
Additionally, you can automate this by finding all of the names of the persistent properties and excluding the one (or ones) you want to avoid: def propNames = grailsApplication.getDomainClass(Proposal.name).persistentProperties*.name
Related
In my application, the query is being built by appending the first part(where clause) with the second part(order by) using a separate script like QueryBuilder.groovy and hence the order by part is prone to HQL injection which can't be sanitized by using Named Parameters. Therefore, I want to use findAll to retrieve a set of records by passing it a query and sorting and paging parameters separately. I saw an implementation like this:
domainClass.findAll(query,[namedParams],[max: 10, offset: 5])
When i passed sortColumn and sortDirection as named parameters, sortColumn worked fine but sortDirection didn't work. i need a way to either make sortDirection work as a named parameter or any other way which will combine 'sorting by direction' with the findAll result. Many people have suggested on various forums to just use the parameters directly as part of the query but it is unacceptable for my application as it will expose the query to HQL Injection.
Thanks in advance.
here is an example:
queryString = "FROM BookCatalog b WHERE b.bookNumber = :bookNumber"
this is passed to the QueryBuilder.groovy where something like this happens:
sort = "$params.sortColumn $params.sortDirection"
queryString.order(sort)
public void sort(String query){
this.query = this.query+" order by "+query
}
finally findAll retrieves the list of records:
def list = findAll(queryString,namedParams,queryParams)
so as the logic just appends the sorting parameters to the query string a potential hacker can do something like this:
bookCatalogView?offset=2&max=5&sortColumn=1,2,3 **or 1=1**
or
bookCatalogView?offset=2&max=5&sortColumn=1,2,3;**select * from whatever**
Don't concat strings, it's bad practice.
If you want to create complex queries, consider using createCriteria()
SomeDomainClass.createCriteria().list {
order("propName", "desc")
}
or, if you need more control, in the sessionFactory way:
query = sessionFactory.getCurrentSession().createCriteria(DomainClass.class)
query.addOrder(Order.asc("someField"))
query.addOrder(Order.desc("someotherField"))
Say we have something like the standard Book domain object and bookCategory object. In my controller I want to return a subset of list of books to the view. That subset is not achievable using a find query. When I try to filer the return object, it deletes relationships from the database!
I tried this:
class BookCategory{
String name
static hasMany = [books:Book]
}
class Book{
String title
}
def myController() {
def categories
categories = BookCategory.list()
def user = getCurrentUser()
categories.each { category ->
category.books.removeAll { book ->
!isBookBannedForThisUser(book.title, user)
}
[bookCategories: categories]
}
}
The problem is that it permanently removes these books from the categories for all users from the database!!!
I tried putting the method in a service and using a readonly transaction, but this did not help.
I assume that even if I copy all the categories and books into new list, they will still update the DB as they will still have the book IDs (which I need)
Saving to the database when you dont say save() is very dangerous. is there a way to disable this feature completely?
There is a fundamental flaw in your approach. Do not modify your domain instances if you don't intend to have the changes persisted. Doing so is going to cause you headaches.
Your domain model is suppose to be your system of record. Any changes to it are suppose to be persisted.
If you need to gather up data and manipulate it without having it reflected in your domain model then use a DTO (data transfer object) or similar pattern.
Simply calling .discard() will discard the changes you have made from being persisted when the session automatically flushes.
Instead of working against the framework, and disabling behavior, change your approach to be correct.
I have an object from which I must filter certain attributes, some of which could also be "null". I have a Filter object and a Product object.
In the Filter object I have certain attributes reflecting the Product object which can be filled out or be left blank. Here a shortened view on the classes.
Product: String name, Boolean isEmpty, ...., belongsTo [Producer, Distributor]...
Filter: Boolean isEmpty, ... belongsTo [Producer, Distributor]...
With this filter I can search for all Products having certain attributes (empty, Producer, Distributor).
I have an export functionality where I can select the filter and it outputs the information based on that selection for the Products.
As all of these attributes can be null, but also contain a value, I first of thought to construct an own search query (combining strings etc) to construct an SQL-string and then using Product.findAll(string_query, string_params). But as this is quite tedious, I changed it now to someting like this:
if(filter.producer)
prods = Product.findAllWhere(producer:filter.producer)
if(filter.distributor)
prods = prods.findAll {it.distributor == filter.distributor}
if(filter.isEmpty!=null) //as it could be NULL but also false/true
prods = prods.findAll {it.isEmpty == filter.isEmpty}
But this becomes quite a larger task if I have 10-15 attributes to be filtered. I'm not very experienced with Grails or Groovy but I guess this can be solved easier, right?
I believe you'll find Grails Criteria queries to be a very nice way to accomplish tasks like this. See:
http://grails.org/doc/latest/guide/single.html#criteria
http://viaboxxsystems.de/the-grails-hibernatecriteriabuilder
Your sample might look something like this when expressed as a criteria query:
def prods = Product.createCriteria().list {
if(filter.producer) eq("producer", filter.producer)
if(filter.distributor) eq("distributor", filter.distributor)
if(filter.isEmpty != null) eq("isEmpty", filter.isEmpty)
}
LoadProperty or Include retrieves all relationated rows from a principal entity. How can I filter the rows retrieved from a LoadProperty call? I would want not to do a post-processing of data recover from DB.
My case is something like this
public Expression<Func<TipoReforma, bool>> predicadoFiltroIdioma(String filtro)
{
return x => x.DetalleTipoReforma.Any(y=>filtro.Contains(y.Idioma.idioma));
}
IEnumerable<T> resultado = objectSet.Where<T>(predicadoFiltroIdioma("en");
Contexto.LoadProperty(resultado.ToList()[0], "DetalleTipoReforma");
I want only "TipoReforma" but related information should be only those which idioma is "x".
Thanks in advance,
Neither LoadProperty or Include support filtering = they always load all related entities. You must use different approach. You can try to use CreateSourceQuery. Something like:
var data = ((EntityCollection<TipoReforma>)resultado.ToList[0].DetalleTipReforma)
.CreateSourceQuery().OrderBy(predicadoFiltroIdioma).ToList();
It should also fill your navigation property in the principal entity. Also make sure that lazy loading is turned off before you execute this code.
I have a question for how to implement multiple criteria for a repository pattern in ASP.net MVC. Imagine a POCO class in EF4
public class people
{ String Name {get;set;}
float Height {get;set;}
float Weight {get;set;}
int age {get;set;}
....
}
If I build up a repository as IPeopleRepository, what kind of methods should I implement for a multiple criteria search (e.g Age > 30, Height >80). Those criteria would be related to the properties in the class and some of the input could be null. Of course I can write a method like
People SearchPeople (int age, float height.....)
but I have to judge if every variable would be null and append onto the search queries..
So do you have any good ideas on how to implement this function in EF?
It sounds like your looking for something like the Specification pattern.
There is a great article involving EF4 / POCO / Repository / Specification pattern here.
Although i like the pattern, i find it a bit overkill in simple scenarios.
I ended up using the "pipes and filters" technique - basically IQueryable<T> extension methods on your objects to make your repository code fluent.
For a search criteria however, i would be tempted to allow the consuming code to supply the predicate, then you don't have to worry about the parameters.
So the definition would be like this:
public People SearchPeople(Expression<Func<People,bool>> predicate)
{
return _context.People.SingleOrDefault(predicate);
}
Then the code simply supplies the predicate.
var person = _repository.SearchPeople(p => p.Age > 30 && p.Height > 80);
Some people don't like this technique, as it gives too much "power" to the consumer, because they might supply a predicate like p.Id > 0 and return all the rows in the database.
To counteract that, provide an optional parameter for maxRows. If it's not supplied, default to 100 rows.
First, you need to think if you really need repository search method.
You might want to do direct queries instead of wrapping them to repository.
However, if you think you need the search method than you will likely use something like this:
private People SearchPeople(int? age, float? height)
{
var baseQuery = db.People;
if (age != null)
baseQuery = baseQuery.Where(arg => arg.Age > age);
if (height != null)
baseQuery = baseQuery.Where(arg => arg.Height > height);
return baseQuery.ToList();
}
Although you didn't want to do this, I can't think of better solution.
Basically I think there are three options:
Use Specification pattern and create as many single specification as you need then you can more complex specification by combining them via And/Or/Not operators. You can look at here for an example http://code.google.com/p/linq-specifications/
Create a search method that accept an input predicate, it's most simple one since it leaves all criteria filtering works to consumers.
Create a search method with different criteria, then build dynamic Linq expression. There is a PredicateBuilder here: http://www.linqpad.net (look for LinqKit project).