I have this problem where I need to perform a Linq query on a big table (1m records) joining other smaller tables. Running the query is taking too long and sometimes ending in a Execution Timeout Expired. I am not sure how to improve the query to cut execution time.
var result = from p in db.population
join e in db.engineers on p.id equals e.personId into ps
from e in ps.DefaultIfEmpty()
where (e == null || e.activity == "student")
select p.name;
var final result = result.Take(100).ToList();
Basically I want to get the first 100 persons from db.population who are not an engineer, engineer-students are excluded.
I am not sure if the query can be written in a better way to improve performance ?
Try This
Do it Before join
var result = from p in db.population
join e in db.engineers.WHERE( c=> c.activity == "student")
on p.id equals e.personId into ps
from e in ps.DefaultIfEmpty()
where (e == null)
select p.name;
var final result = result.Take(100).ToList();
or
join e in db.engineers.WHERE( c=> c.activity == "student").Take(100)
Related
So I have a linq query, where s1.code comes from an object before this linq query.
var q1 = from cf in db.Control_Franchises
join t1 in db.Territories
on SqlFunctions.StringConvert((double)cf.FranchiseID).Trim()
equals t1.FranchiseID.Trim()
join cu in db.Control_Users on t1.FK_CompanyID equals cu.PrimaryCompanyID
join u in db.Users on cu.UserID.ToLower() equals u.Username.ToLower()
where cf.Status == "ACTIVE"
&& cf.FranchiseID > 1000
&& cu.UserType == "Franchisee"
&& cu.Status == "ACTIVE"
&& t1.Province == s1.Code
orderby cu.LastName ascending, cf.FranchiseID ascending
select new
{
FranchiseId = cf.FranchiseID,
Province = cf.StateCode,
DisplayName = cu.LastName + ", " + cu.FirstName,
UserId = u.PK_UserID
};
I have the same block of code but this time in my where clause, I changed the filter from using t1.Province == s1.Code to cf.StateCode == s1.Code
var q1 = from cf in db.Control_Franchises
join t1 in db.Territories
on SqlFunctions.StringConvert((double)cf.FranchiseID).Trim()
equals t1.FranchiseID.Trim()
join cu in db.Control_Users on t1.FK_CompanyID equals cu.PrimaryCompanyID
join u in db.Users on cu.UserID.ToLower() equals u.Username.ToLower()
where cf.Status == "ACTIVE"
&& cf.FranchiseID > 1000
&& cu.UserType == "Franchisee"
&& cu.Status == "ACTIVE"
&& cf.StateCode == s1.Code // DIFFERENT FROM ABOVE
orderby cu.LastName ascending, cf.FranchiseID ascending
select new
{
FranchiseId = cf.FranchiseID,
Province = cf.StateCode,
DisplayName = cu.LastName + ", " + cu.FirstName,
UserId = u.PK_UserID
};
Now the first query runs 10 times as fast as the second one.
How I am measuring the speed though, is the load time of my Edit Page, which in turn is loading a mvc tree view of the nodes. This query as well portions of a different query.
Now I'm trying to understand why my first query loads much faster and the only reasoning I can think of is because I am doing a condition on a table that will be joined, so rather then joining the entire "Territories" table with "Control_Franchises" I am only joining a portion of the "Territories" table.
Any thoughts?
Most likely the two different columns t1.Province and cf.StateCode are indexed differently in underlying data source (or one isn't at all)
Using the technique found on the MSDN article "How to: Perform Left Outer Joins (C# Programming Guide)", I attempted to create a left outer join in my Linq code. The article mentions using the DefaultIfEmpty method in order to create a left outer join from a group join. Basically, it instructs the program to include the results of the left (first) collection even if there are no results in the right collection.
The way this program executes, however, it does so as if the outer join has not been specified.
In our database, AgentProductTraining is a collection of courses our agents have taken. Normally you cannot enter a Course onto it's appropriate table without entering a corresponding value into the CourseMaterials table. However, occasionally this may happen, so we want to make sure we return results even when a Course is listed in AgentProductTraining without any corresponding information in CourseMaterials.
var training = from a in db.AgentProductTraining
join m in db.CourseMaterials on a.CourseCode equals m.CourseCode into apm
where
a.SymNumber == id
from m in apm.DefaultIfEmpty()
where m.EffectiveDate <= a.DateTaken
&& ((m.TerminationDate > a.DateTaken) | (m.TerminationDate == null))
select new
{
a.AgentProdTrainId,
a.CourseCode,
a.Course.CourseDescription,
a.Course.Partner,
a.DateTaken,
a.DateExpired,
a.LastChangeOperator,
a.LastChangeDate,
a.ProductCode,
a.Product.ProductDescription,
m.MaterialId,
m.Description,
a.Method
};
The MSDN example uses a new variable subpet:
var query = from person in people
join pet in pets on person equals pet.Owner into gj
from subpet in gj.DefaultIfEmpty()
select new { person.FirstName, PetName = (subpet == null ? String.Empty : subpet.Name) };
So you must use your own "subpet", I rewrote your code using the submat variable:
var training = from a in db.AgentProductTraining
join m in db.CourseMaterials on a.CourseCode equals m.CourseCode into apm
where
a.SymNumber == id
from submat in apm.DefaultIfEmpty()
where
(submat.EffectiveDate <= a.DateTaken || submat.EffectiveDate == null) &&
(submat.TerminationDate > a.DateTaken || submat.TerminationDate == null)
select new
{
a.AgentProdTrainId,
a.CourseCode,
a.Course.CourseDescription,
a.Course.Partner,
a.DateTaken,
a.DateExpired,
a.LastChangeOperator,
a.LastChangeDate,
a.ProductCode,
a.Product.ProductDescription,
MaterialId = (submat==null?-1:submat.MaterialId),
Description = (submat==null?String.Empty:submat.Description),
a.Method
};
I am using telerik mvc grid in my mvc project , my table have around 1 Million records. My grid taking too much time to load.
This is my Query
//
var bib = (from a in db.Bibs
join inf in db.InfoTypes
on a.InfoTypeId equals inf.Id
where a.Status == "A"
select new BibViewModel
{
Id = a.Id,
Type = inf.Type,
InfoType = inf.Description,
Title = (from asd in db.BibContents where asd.BibId == a.Id && asd.TagNo == "245" && asd.Sfld == "a" select asd.Value).FirstOrDefault(),
Author = (from asd in db.BibContents where asd.BibId == a.Id && asd.TagNo == "100" && asd.Sfld == "a" select asd.Value).FirstOrDefault(),
CatalogueDate = a.CatalogDate,
Contents = "",
CreatedOn = a.CreatedOn,
ItemRelation = db.Items.Any(item => item.BibId == a.Id),
IssueRelation = db.Issues.Any(item => item.BibId == a.Id),
});
return View(new GridModel(bib.OrderByDescending(x => x.CreatedOn).Tolist()));
ToList() actually invokes the query, so if calling ToList() is taking too long, that means the issue is with the query.
In LINQ, you can use paging like in the following post; the idea is to use Skip and Take to skip X records, and only take Y records, as in:
var results = (from .. select ..).Skip(X).Take(Y)
With 1M records, I would highly suggest replacing it with a stored procedure, which will be much, much faster for what you are trying to do. Consider a custom pagination approach, which works very well for me with large result sets:
http://www.neiland.net/blog/article/dynamic-paging-in-sql-server-stored-procedures/
http://www.beansoftware.com/ASP.NET-Tutorials/Paging-Stored-Procedures.aspx
http://www.sqlpointers.com/2009/10/custom-sorting-and-paging-in-sql-server.html
T-SQL stored procedure with sorting and paging enabled not working properly
If you can't use stored procedures, reading this will help understand what needs to be accomplished with pagination. With LINQ, you'll want to examine the SQL being generated to see where you also can fine-tune the query, either using SQL profiler, or LINQPad.
I am developing a ASP.NET MVC 3 Application with EF4. I have about 50,000 entities and I'm querying them through LINQ to find what matches best with the given search criteria. There are multiple search criterion (up to about 12) and these are matcheon a step by step basis.
Ex: 50,000 students
Get the students within the age range -> A
From A, get the students who are male -> B
From B, get students who are enrolled to course CS101 -> C
What's the best way to achieve this?
step by step doesn't mean a lot for SQL and your linq query will be transformed to sql to query the db...
so
//I'm a IQueryable
var queryableStudents =
students.Where(m =>
m.Age > 10 &&
m.Gender == 'm' &&
m.CourseList.Any(x => x.Name == 'CS101');
//I'm no more an IQueryable
var result = queryableStudents.ToList();//the query will be sent to db and result returned.
But if search criteria are optional, you can do
//I'm a IQueryable
var queryableStudents = students;
if (searchCriteria.Age > 0)
//I'm still a IQueryable
queryableStudents = queryableStudents.Where(m => m.Age => searchCriteria.Age);
if (!String.IsNullOrEmpty(searchCriteria.Gender))
//I'm still a IQueryable
queryableStudents = queryableStudents.Where(m => m.Gender == searchCriteria.Gender);
//Now I'm no more an IQueryable
var result = queryableStudents.ToList()//the query will be sent to db and result returned.
If you want a "REAL" step by step, (showing results for each step), you can do
//I'm not an IQueryable
var a= students.Where(m => m.Age > 10).ToList();//you will get all students from your db who respect your first criterion, and then work on an IEnumerable, not an IQueryable.
//I'm not IQueryable
var b= a.Where(m => m.Gender == 'm');
//I'm not an IQueryable
var c= b.Where(m => m.CourseList.Any(x => x.Name == "CS101");
var A = from s in students
where ((s.age < max) && (s.age > min))
select s;
var B = from a in A
where (a.gender.Equals("Male"))
select a;
var C = from b in B
where (b.EnrolledCourses().Contains("CS101"))
select b;
Answering my own question - after some thought I figured out the most effiecient way of doing this is to use something like ElasticSearch to index the entries I want.
The given use case is not a very good one for LINQ/C#.
I have three tables. I have to retrieve the data using Linq statement. My three table names are A,B,C. I connected join for connecting two tables A and B based on the id's like:
select ol, fN, LN, ci, co
from member
join details
on member_id = details_id
where details_id in
(select contacts_id from contacts where
contacts_id1 = 1 and contacts_usr_id = 1)
I am able to write the query up to the where condition, how can I write the query for the inner while condition?
you can try this:
var idlist = (from tbl in table3
where tbl.usr_id == 1 && tbl.contacts_id == 1
select tbl.contacts_id ).ToList();
var x = from A in table1
from B in table2 where A.user_id == B.user_id
&& idlist.Contains(A.user_id)
select new { a = A.a, b = A.b, c = A.c, d = B.d, e = B.e };
check and let me know if that work.
Try flipping the query upside down. How about the following:
var query =
from t3 in table3
where t3.user_id = 1 && t3.contacts_id = 1
join t2 in table2 on t3.contacts_id equals t2.usr_id
join t1 in table1 on t2.usr_id equals t1.userid
select new {t2.a, t2.b, t2.c, t1.d, t1.e};