I am using telerik mvc grid in my mvc project , my table have around 1 Million records. My grid taking too much time to load.
This is my Query
//
var bib = (from a in db.Bibs
join inf in db.InfoTypes
on a.InfoTypeId equals inf.Id
where a.Status == "A"
select new BibViewModel
{
Id = a.Id,
Type = inf.Type,
InfoType = inf.Description,
Title = (from asd in db.BibContents where asd.BibId == a.Id && asd.TagNo == "245" && asd.Sfld == "a" select asd.Value).FirstOrDefault(),
Author = (from asd in db.BibContents where asd.BibId == a.Id && asd.TagNo == "100" && asd.Sfld == "a" select asd.Value).FirstOrDefault(),
CatalogueDate = a.CatalogDate,
Contents = "",
CreatedOn = a.CreatedOn,
ItemRelation = db.Items.Any(item => item.BibId == a.Id),
IssueRelation = db.Issues.Any(item => item.BibId == a.Id),
});
return View(new GridModel(bib.OrderByDescending(x => x.CreatedOn).Tolist()));
ToList() actually invokes the query, so if calling ToList() is taking too long, that means the issue is with the query.
In LINQ, you can use paging like in the following post; the idea is to use Skip and Take to skip X records, and only take Y records, as in:
var results = (from .. select ..).Skip(X).Take(Y)
With 1M records, I would highly suggest replacing it with a stored procedure, which will be much, much faster for what you are trying to do. Consider a custom pagination approach, which works very well for me with large result sets:
http://www.neiland.net/blog/article/dynamic-paging-in-sql-server-stored-procedures/
http://www.beansoftware.com/ASP.NET-Tutorials/Paging-Stored-Procedures.aspx
http://www.sqlpointers.com/2009/10/custom-sorting-and-paging-in-sql-server.html
T-SQL stored procedure with sorting and paging enabled not working properly
If you can't use stored procedures, reading this will help understand what needs to be accomplished with pagination. With LINQ, you'll want to examine the SQL being generated to see where you also can fine-tune the query, either using SQL profiler, or LINQPad.
Related
Getting the last record is trivial in SQL, e.g. (for MySQL)
class ExchangeRate {
Currency from
Currency to
BigDecimal rate // e.g. 10
Date dateCreated
}
class Currency {
String iso
etc..
}
SQL to get the latest is trivial:
Select max(id), rate
from exchange_rate
where from_id = 1
and to_id = 3
or
select rate
from exchange_rate
where from_id = 2
order by id desc
limit 1
The question is, how does one do this efficiently in Grails? I only want a single result.
This obviously wont work:
def query = ExchangeRate.where {
from == from && to == to && id == max(id)
}
ExchangeRate exchangeRate = query.find()
There have been several posts on this, but not with an actual answer which I could figure out how to apply (I am a SQL guy, and don't know hibernate language and would prefer a solution which did not involve it if there was one)
If there was an easy way to run SQL directly without having to hand manage result sets that would work (as we will never use another DB other than MySQL)
I am sure it could be done with sort and limit, but a) haven't found an example I could copy, and b) would assume this be inefficient, because it appears that the sorting and limiting is done in code, not SQL?
This example is in the documentation:
Book.findAll("from Book as b where b.author=:author",
[author: 'Dan Brown'], [max: 10, offset: 5])
could lead to this:
def exchangeRates = ExchangeRate.findAll("from ExchangeRate as e where e.from = :from order by id desc", [from: from], [max: 1])
if (exchangeRates.size() == 1) {
return exchangeRates.first().rate
}
return null
is there a better way (e.g. one which doesnt use hibernate low level language, or one which uses SQL instead, or one which is more efficient?)
Try using a subquery according to the documentation.
def query = ExchangeRate.where {
id = max(id).of { from == fromValue } && to == toValue
}
query.find()
so I have a form with several fields which are criteria for searching in a database.
I want to formulate a query using LINQ like so:
var Coll = (from obj in table where value1 = criteria1 && value2 = criteria2...)
and so on.
My problem is, I don't want to write it using If statements to check if every field has been filled in, nor do I want to make separate methods for the various search cases (criteria 1 and criteria 5 input; criteria 2 and criteria 3 input ... etc.)
So my question is: How can I achieve this without writing an excessive amount of code? If I just write in the query with comparison, will it screw up the return values if the user inputs only SOME values?
Thanks for your help.
Yes, it will screw up.
I would go with the ifs, I don't see what's wrong with them:
var query = table;
if(criteria1 != null)
query = query.Where(x => x.Value1 == criteria1);
if(criteria2 != null)
query = query.Where(x => x.Value2 == criteria2);
If you have a lot of criteria you could use expressions, a dictionary and a loop to cut down on the repetitive code.
In an ASP.NET MVC app, chances are your user input is coming from a form which is being POSTed to your server. In that case, you can make use of strongly-typed views, using a viewmodel with [Required] on the criteria that MUST be provided. Then you wrap your method in if (ModelState.IsValid) { ... } and you've excluded all the cases where the user hasn't given you something they need.
Beyond that, if you can collect your criteria into a list, you can filter it. So, you could do something like this:
filterBy = userValues.Where(v => v != null);
var Coll = (from obj in table where filterBy.Contains(value1) select obj);
You can make this more complex by having a Dictionary (or Lookup for non-unique keys) that contains a user-entered value along with some label (an enum, perhaps) that tells you which field they're filtering by, and then you can group them by that label to separate out the filters for each field, and then filter as above. You could even have a custom SearchFilter object that contains other info, so you can have filters with AND, NOT and OR conditions...
Failing that, you can remember that until you trigger evaluation of an IQueryable, it doesn't hit the database, so you can just do this:
var Coll = (from obj in table where value1 == requiredCriteria select obj);
if(criteria1 != null)
{
query = query.Where(x => x.Value1 == criteria1);
}
//etc...
if(criteria5 != null)
{
query = query.Where(x => x.Value5 == criteria5);
}
return query.ToList();
That first line applies any criteria that MUST be there; if there aren't any mandatory ones then it could just be var Coll = table;.
That will add any criteria that are provided will be applied, any that aren't will be ignored, you catch all the possible combinations, and only one query is made at the end when you .ToList() it.
As I understand of your question you want to centralize multiple if for the sake of readability; if I were right the following would be one of some possible solutions
Func<object, object, bool> CheckValueWithAnd = (x, y) => x == null ? true : x==y;
var query = from obj in table
where CheckValue(obj.value1, criteria1) &&
CheckValue(obj.value2, criteria2) &&
...
select obj;
It ls flexible because in different situations or scenarios you can change the function in the way that fulfill your expectation and you do not need to have multiple if.
If you want to use OR operand in your expression you need to have second function
Func<object, object, bool> CheckValueWithOr = (x, y) => x == null ? false : x==y;
So I have a linq query, where s1.code comes from an object before this linq query.
var q1 = from cf in db.Control_Franchises
join t1 in db.Territories
on SqlFunctions.StringConvert((double)cf.FranchiseID).Trim()
equals t1.FranchiseID.Trim()
join cu in db.Control_Users on t1.FK_CompanyID equals cu.PrimaryCompanyID
join u in db.Users on cu.UserID.ToLower() equals u.Username.ToLower()
where cf.Status == "ACTIVE"
&& cf.FranchiseID > 1000
&& cu.UserType == "Franchisee"
&& cu.Status == "ACTIVE"
&& t1.Province == s1.Code
orderby cu.LastName ascending, cf.FranchiseID ascending
select new
{
FranchiseId = cf.FranchiseID,
Province = cf.StateCode,
DisplayName = cu.LastName + ", " + cu.FirstName,
UserId = u.PK_UserID
};
I have the same block of code but this time in my where clause, I changed the filter from using t1.Province == s1.Code to cf.StateCode == s1.Code
var q1 = from cf in db.Control_Franchises
join t1 in db.Territories
on SqlFunctions.StringConvert((double)cf.FranchiseID).Trim()
equals t1.FranchiseID.Trim()
join cu in db.Control_Users on t1.FK_CompanyID equals cu.PrimaryCompanyID
join u in db.Users on cu.UserID.ToLower() equals u.Username.ToLower()
where cf.Status == "ACTIVE"
&& cf.FranchiseID > 1000
&& cu.UserType == "Franchisee"
&& cu.Status == "ACTIVE"
&& cf.StateCode == s1.Code // DIFFERENT FROM ABOVE
orderby cu.LastName ascending, cf.FranchiseID ascending
select new
{
FranchiseId = cf.FranchiseID,
Province = cf.StateCode,
DisplayName = cu.LastName + ", " + cu.FirstName,
UserId = u.PK_UserID
};
Now the first query runs 10 times as fast as the second one.
How I am measuring the speed though, is the load time of my Edit Page, which in turn is loading a mvc tree view of the nodes. This query as well portions of a different query.
Now I'm trying to understand why my first query loads much faster and the only reasoning I can think of is because I am doing a condition on a table that will be joined, so rather then joining the entire "Territories" table with "Control_Franchises" I am only joining a portion of the "Territories" table.
Any thoughts?
Most likely the two different columns t1.Province and cf.StateCode are indexed differently in underlying data source (or one isn't at all)
Using the technique found on the MSDN article "How to: Perform Left Outer Joins (C# Programming Guide)", I attempted to create a left outer join in my Linq code. The article mentions using the DefaultIfEmpty method in order to create a left outer join from a group join. Basically, it instructs the program to include the results of the left (first) collection even if there are no results in the right collection.
The way this program executes, however, it does so as if the outer join has not been specified.
In our database, AgentProductTraining is a collection of courses our agents have taken. Normally you cannot enter a Course onto it's appropriate table without entering a corresponding value into the CourseMaterials table. However, occasionally this may happen, so we want to make sure we return results even when a Course is listed in AgentProductTraining without any corresponding information in CourseMaterials.
var training = from a in db.AgentProductTraining
join m in db.CourseMaterials on a.CourseCode equals m.CourseCode into apm
where
a.SymNumber == id
from m in apm.DefaultIfEmpty()
where m.EffectiveDate <= a.DateTaken
&& ((m.TerminationDate > a.DateTaken) | (m.TerminationDate == null))
select new
{
a.AgentProdTrainId,
a.CourseCode,
a.Course.CourseDescription,
a.Course.Partner,
a.DateTaken,
a.DateExpired,
a.LastChangeOperator,
a.LastChangeDate,
a.ProductCode,
a.Product.ProductDescription,
m.MaterialId,
m.Description,
a.Method
};
The MSDN example uses a new variable subpet:
var query = from person in people
join pet in pets on person equals pet.Owner into gj
from subpet in gj.DefaultIfEmpty()
select new { person.FirstName, PetName = (subpet == null ? String.Empty : subpet.Name) };
So you must use your own "subpet", I rewrote your code using the submat variable:
var training = from a in db.AgentProductTraining
join m in db.CourseMaterials on a.CourseCode equals m.CourseCode into apm
where
a.SymNumber == id
from submat in apm.DefaultIfEmpty()
where
(submat.EffectiveDate <= a.DateTaken || submat.EffectiveDate == null) &&
(submat.TerminationDate > a.DateTaken || submat.TerminationDate == null)
select new
{
a.AgentProdTrainId,
a.CourseCode,
a.Course.CourseDescription,
a.Course.Partner,
a.DateTaken,
a.DateExpired,
a.LastChangeOperator,
a.LastChangeDate,
a.ProductCode,
a.Product.ProductDescription,
MaterialId = (submat==null?-1:submat.MaterialId),
Description = (submat==null?String.Empty:submat.Description),
a.Method
};
I can't find the correct way to build an EF (4.1) query that will return the same result as this SQL containing a LEFT JOIN:
SELECT
s.id_service,
s.description,
x.id_service as isDisponible
FROM
role.service_disponible s
LEFT JOIN
role.service_disponible_x_ue x
ON s.id_service = x.id_service AND x.id_ue = 1 and flg_actif = '1'
In fact I'm just trying to obtain the complete list of services disponible (ServiceDisponible) adding a field that tell me if service is disponible for a specific entity (filtered with the id_ue) which information come from a many to many related table (ServiceDisponibleXUe).
My model is:
Ideally, I would like this query to return this viewModel object what is basically my serviceDisponible domain with one more field indicating the disponibility of the service.
public ServiceDisponibleViewModel(ServiceDisponible ServiceDisponible, bool isDisponible)
{
this.serviceDisponible = serviceDisponible;
this.isDisponible = isDisponible;
}
What I have so far is this query but the syntax is invalid:
services = context.ServiceDisponible
.Select(a => new ServiceDisponibleViewModel
{
c => new ServiceDisponible
{
id_service = a.id_service,
description = a.description
},
isDisponible = a.ServiceDisponibleXUe
.Any(b => b.flg_actif && b.id_ue == idUe)
}).ToList();
Try this:
ServiceDisponibleViewModel services =
from sd in context.ServiceDisponible
from sdx in context.ServiceDisponibleXUe
.Where(x => x.id_ue == 1 && flg_actif == '1' && x.id_service == sd.id_service)
.DefaultIfEmpty()
select new ServiceDisponibleViewModel(
new ServiceDisponible
{
id_service = sd.id_service,
description = sd.description
},
sdx.id_service
);
Having SQL as example often makes one jump to a join in linq. But using navigation properties produces much more succinct syntax:
from sd in context.ServiceDisponible
from sdx in sd.ServiceDisponibleXUes.Where(x => x.id_ue == 1
&& x.flg_actif == "1")
.DefaultIfEmpty()
select new
{ sd.id_service,
sd.description,
isDisponible = sdx.id_service
};
(I couldn't help using the plural form of ServiceDisponibleXUe which imo is more clear).