The required column 'xxxID' was not present in the results of a 'FromSql' operation - .Net Core EF - entity-framework-core-3.1

I am trying to find better and performance efficient approach for bulk delete in .NET Core EF (3.1.9). (Approx 500K to 1 Mil records to be deleted in one shot.)
Model:
public class Employee
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int EmpID { get; set; }
public string EmpName { get; set; }
}
And database table as:
CREATE TABLE [dbo].[Employee]
(
[EmpID] [int] IDENTITY(1,1) NOT NULL,
[EmpName] [nchar] (20) NULL,
CONSTRAINT [PK_dbo].[Employee]
PRIMARY KEY CLUSTERED ([EmpID] ASC)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
Trying to delete records with generic method-1.
public int Delete<TEntity>(Func<TEntity, bool> predicate) where TEntity: class
{
return DbContext.Set<TEntity>
.FromSqlRaw($"Delete from dbo.Employee")
.Where(predicate).Count();
}
And calling this method as
Func<Employee, bool> myPredicate = x => x.EmpID > 10;
int deletedCount = myclass.Delete(myPredicate);
Exception thrown:
InvalidOperationException: The required column 'EmpID' was not present in the results of a 'FromSql' operation.
ASP.NET Core EF generates query:
DELETE FROM [dbo].[Employee]
fail: Microsoft.EntityFramework.Query[10100] .... stack trace with above error.
Already looked into this:
.NET Core Entity Framework InvalidOperationException
The required column 'CustomerId' was not present in the results of a 'FromSql' operation
The required column 'id' was not present in the results of a `FromSql` operation in EFcore
So not sure why it is throwing above error. As database has the correct PK and model also has it. Tried with some other entities too, it always throws the same error with column name 'xxxEntityColID'.
However, if I use following code then it works:
public int Delete<TEntity>(string whereCondition, params object[] parameters) where TEntity: class
{
return DbContext.Database.ExecuteSqlRaw($"Delete from dbo.Employee WHERE {whereCondition}", parameters);
}
// and calling like
string myCondition = "EmpID > 10";
int deletedCount = myclass.Delete<Employee>(myCondition, new object[0]);
.NET Core EF generates following SQL and work without any error.
DELETE FROM [dbo].[Employee] WHERE EmpID > 10;
Questions
Why does this 'xxxEntityColID' error occur?
What is the better approach to do bulk deletes in .NET Core EF?

Looks like FromSqlRaw() is worked with "SELECT.." queries.
For Bulk delete I evaluated following approach and it is better than dbContext.RemoveRange(entities); Because RemoveRange() generated separate query for each entity. Then every single query takes some time, so not suitable for Bulk delete. In contrast ExecuteSqlRaw(with where condition), just generates single query which hardly takes 250ms for 200K records. On high end servers it will take lesser for sure.
public int Delete<TEntity>(string whereCondition, params object[] parameters) where TEntity: class
{
return DbContext.Database.ExecuteSqlRaw($"Delete from dbo.Employee WHERE {whereCondition}", parameters);
}

Related

.NET mvc adding more then 1 entity to db

I'm trying to add several entities to the db but I get this error:
"Saving or accepting changes failed because more than one entity of type 'Schema.Domain.DataModels.ActivitySummery' have the same primary key value. Ensure that explicitly set primary key values are unique. Ensure that database-generated primary keys are configured correctly in the database and in the Entity Framework model. Use the Entity Designer for Database First/Model First configuration. Use the 'HasDatabaseGeneratedOption" fluent API or 'DatabaseGeneratedAttribute' for Code First configuration."
public void CreateNewGeneratedSchema(List<WeekDayViewModel> weekDays, int userId)
{
List<ActivitySummery> savedActivities = _schemaRepository.GetAllActivitySummeries(userId).ToList();
foreach(ActivitySummery activitySummery in savedActivities)
{
_schemaRepository.DeleteActivitySummery(activitySummery.ActivitySummeryId);
}
foreach (WeekDayViewModel weekDay in weekDays)
{
foreach (ActivitySummeryViewModel activitySummeryViewModel in weekDay.ActivitiySummeries)
{
try
{
ActivitySummery activitySummery = new ActivitySummery()
{
ActivityId = activitySummeryViewModel.ActivityId,
WeekDayId = activitySummeryViewModel.WeekDayId,
//Change userId
UserId = 1,
StartTime = activitySummeryViewModel.StartTime,
EndTime = activitySummeryViewModel.EndTime,
ActivityDescription = activitySummeryViewModel.Description,
Activity = _schemaRepository.GetSpecificActivity(activitySummeryViewModel.ActivityId),
WeekDay = _schemaRepository.GetSpecificWeekDay(activitySummeryViewModel.WeekDayId)
};
_schemaRepository.CreateActivitySummery(activitySummery);
}
catch (Exception)
{
throw new Exception("Something went wrong when trying to save the activity summery");
}
}
}
_schemaRepository.Save();
_schemaRepository.Dispose();
}
I know one solution that might work and it's to save and dispose the _schemaRepository after every time i added one model to the db and create a new instace of _schemaRepository. But im not sure if that is the right way. I know that every model that im trying to save has the pk of 0 and i think that might be the problem. However it still works and the db accepts the new entities but I still get the exception.
If your model (which you didn't show) has the primary key as ActivityId (which you also didn't indicate) and it is setup to auto-generate the primary key:
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int ActivityId { get; set; }
Then you must not include the primary key in the CreateActivitySummery method (also which you haven't provided).
// Remove this line...
// ActivityId = activitySummary.ActivityId;
Do note that a (reasonable) alternative to having the database generate the primary key automatically is to use Guid/UniqueIdentifier without using DatabaseGenerated. Then your application code can use Guid.NewGuid to generate new (presumably unique) primary keys that it can insert with no problem and track related entities easily.

Execute raw SQL query in ASP.NET MVC, database first mode

The model of my project is database first, and uses remote access to database on another server.
I need to use raw SQL query because my query is very complex and I feel more comfortable in SQl not LINQ.
This is how I do:
string query = "select * from Inquiry_TBL where ...";
using (educationEntities db = new educationEntities())
{
var list = db.Database.SqlQuery<Inquiry_TBL>(query);
ViewData["total"] = list.Count();
}
The problem is sometimes I get the query result within a second, sometimes it just keep loading for a long time and gives me an error that 'Calling 'Read' when the data reader is closed is not a valid operation.'
Why is that? Is there something wrong with my code, or because I'm using remote access to another server? Will switching to local server solve the problem?
The Entity Framework Code First API includes methods that enable you to pass SQL commands directly to the database. You have the following options:
• Use the DbSet.SqlQuery method for queries that return entity types. The returned objects must be of the type expected by the DbSet object, and they are automatically tracked by the database context unless you turn tracking off. (See the following section about the AsNoTracking method.)
• Use the Database.SqlQuery method for queries that return types that aren't entities. The returned data isn't tracked by the database context, even if you use this method to retrieve entity types.
• Use the Database.ExecuteSqlCommand for non-query commands.
Calling a Query that Returns Entities:
public async Task<ActionResult> Details(int? id)
{
if (id == null)
{
return new HttpStatusCodeResult(HttpStatusCode.BadRequest);
}
// Commenting out original code to show how to use a raw SQL query.
//Department department = await db.Departments.FindAsync(id);
// Create and execute raw SQL query.
string query = "SELECT * FROM Department WHERE DepartmentID = #p0";
Department department = await db.Departments.SqlQuery(query, id).SingleOrDefaultAsync();
if (department == null)
{
return HttpNotFound();
}
return View(department);
}
Calling a Query that Returns Other Types of Objects:
public ActionResult About()
{
//Commenting out LINQ to show how to do the same thing in SQL.
//IQueryable<EnrollmentDateGroup> = from student in db.Students
// group student by student.EnrollmentDate into dateGroup
// select new EnrollmentDateGroup()
// {
// EnrollmentDate = dateGroup.Key,
// StudentCount = dateGroup.Count()
// };
// SQL version of the above LINQ code.
string query = "SELECT EnrollmentDate, COUNT(*) AS StudentCount "
+ "FROM Person "
+ "WHERE Discriminator = 'Student' "
+ "GROUP BY EnrollmentDate";
IEnumerable<EnrollmentDateGroup> data = db.Database.SqlQuery<EnrollmentDateGroup>(query);
return View(data.ToList());
}
Calling an Update Query:
[HttpPost]
public ActionResult UpdateCourseCredits(int? credit)
{
if (credit != null)
{
ViewBag.RowsAffected = db.Database.ExecuteSqlCommand(
"UPDATE Course SET Credits = Credits * {0}", credit);
}
return View();
}
For more information have a look at Advanced Entity Framework 6 Scenarios for an MVC 5 Web Application (12 of 12).

Insert to multiple foreign key tables in MVC with super-type/sub-type database-first model

I have two sub-types of a super-type "Entity", namely "Household" and "Involved Body".
I've modeled them as shown below in my database and they were auto-generated to the EF Model (again shown below).
database
edmx model
Using the default scaffolding for MVC I am able to add a new Household without any problems. However, when I try to add a new Involved Body I hit an error when it tries to add the Entity Type.
There only relevant (as far as I can tell) difference between the two sub-types is that the EntityType for a Household is hard-coded as "Household" whereas the EntityType for an Involved Body can be any EntityType except "Household" - this is selected from a list by the user.
The Create Action on the HTTP POST for the Involved Body throws an error relating to the foreign key between tEntity and tEntityType with the tEntityType being null. Code as follows:
[HttpPost]
public ActionResult Create([Bind(Exclude = "entityID")]tEntity tentity
, tInvolvedBody tinvolvedbody
, tAddress taddress
, tAddressEntity taddressentity
//, tEntityType tentitytype
, int entityTypeID
)
{
#region entity type
//find entity type from id
var tentitytype = db.tEntityTypes.Find(entityTypeID);
#endregion
#region address
//assume start date of involved body not needed for reporting
taddressentity.startDate = DateTime.Now.Date;
#endregion
if (ModelState.IsValid)
{
db.tEntities.Add(tentity);
db.tInvolvedBodies.Add(tinvolvedbody);
db.tAddresses.Add(taddress);
db.tAddressEntities.Add(taddressentity);
db.SaveChanges();
return RedirectToAction("Index");
}
//recreate viewbag for entityType dropdown
var q = (
from e in db.tEntityTypes
where e.entityType != "Household"
select e
);
ViewBag.entityTypeID = new SelectList(q, "entityTypeID", "entityType");
return View(tinvolvedbody);
}
I've tried adding the tEntityType to the parameters list for the create but this results in the ModelState.IsValid returning false due to the entityType being null on all the objects.
I've also tried actively linking the entity type to each of the other objects using:
tentity.tEntityType = tentitytype;
tinvolvedbody.tEntity.tEntityType = tentitytype;
taddressentity.tEntity.tEntityType = tentitytype;
The above ends up working but it creates a new Entity for each of the other objects i.e. I get three new rows in my tEntity table, one is the Entity, one links to tInvolvedBody and one links to tAddressEntities. This makes no sense...
How can I insert a new InvolvedBody that creates an Entity, picks up the Entity Type and then links to the AddressEntity junction table?
Finally worked through this. Not sure if the answer is 'perfect' from a developer perspective but it works.
After intense debugging I realised that the navigation properties for the involved body and address entity were both looking for an entitytypeID which I had assumed would be provided by the entity object.
If I passed these in directly with the code shown:
tinvolvedbody.tEntity.tEntityType = tentitytype;
taddressentity.tEntity.tEntityType = tentitytype;
...I ended up with three new entitites and no relational data existing between all of {entity, involved body, address}
The code that works removes the explicit addition of a new entity and relies on EF to create an entity from the Involved Body. I then used the newly created entityID to map the address via addressentity as follows:
[HttpPost]
public ActionResult Create([Bind(Exclude="entityID")]tEntity tentity
,tInvolvedBody tinvolvedbody
,tAddress taddress
,tAddressEntity taddressentity
,int entityTypeID
)
{
#region entity type
var t =
(
from e in db.tEntityTypes
where (e.entityTypeID == entityTypeID)
select e
);
tinvolvedbody.tEntity.tEntityType = t.First();
#endregion
#region address
//assume start date of involved body not needed for reporting
taddressentity.startDate = DateTime.Now.Date;
#endregion
if (ModelState.IsValid)
{
db.tInvolvedBodies.Add(tinvolvedbody);
db.tAddresses.Add(taddress);
taddressentity.tEntity = db.tEntities.Find(tinvolvedbody.bodyID);
db.tAddressEntities.Add(taddressentity);
db.SaveChanges();
return RedirectToAction("Index");
}
Have you tried setting the typeID specifically? Also, from what I gathered from your model, the taddress is a child of taddressentity? As such, should it not be inserted first in order for the foreign key?
if (ModelState.IsValid)
{
tentity.entityTypeID = entityTypeID;
db.tEntities.Add(tentity);
tinvolvebody.bodyID= tentity.entityID
db.tInvolvedBodies.Add(tinvolvedbody);
taddressentity.entityID = tentity.entityID;
db.tAddressEntities.Add(taddressentity);
taddress.UPRN = taddressentity.UPRN;
db.tAddresses.Add(taddress);
db.SaveChanges();
return RedirectToAction("Index");
}

How to update eagerly loaded entity with relations

I am using EF4 with ASP.Net MVC, and I wonder how am I to update entity which was loaded eagerly?
When I load the entity as follows:
static readonly Func<Entities, Guid, IQueryable<Call>> CallByGuid =
CompiledQuery.Compile<Entities, Guid, IQueryable<Call>>(
(db, callGuid) =>
db.Calls
.Where(c => !c.Removed && c.CallGUID.Equals(callGuid)));
public Call GetCall(Guid guid)
{
return CallByGuid.Invoke(_db, guid).Single();
}
and pass the entity to the View, edit it there and return back to Controller and save it as follows:
public Call UpdateCall(Call call)
{
_db.ObjectStateManager.ChangeObjectState(call, EntityState.Modified);
_db.Call_VoiceCall.ApplyCurrentValues(call);
_db.SaveChanges();
Call dbCall = FindCall(call.CallGUID);
return dbCall;
}
everything works fine, though quite slow. When I load the entity as follows:
static readonly Func<Entities, Guid, IQueryable<Call>> CallByGuid =
CompiledQuery.Compile<Entities, Guid, IQueryable<Call>>(
(db, callGuid) =>
db.Calls
.Include("Call_VoiceCall")
.Include("Call_Rating")
.Include("Memos")
.Include("Company")
.Include("PhoneService")
.Where(c => !c.Removed && c.CallGUID.Equals(callGuid)));
public Call GetCall(Guid guid)
{
return CallByGuid.Invoke(_db, guid).Single();
}
the list of entities loads very fast, but I cannot update an entity: what I get is
"Violation of PRIMARY KEY constraint 'PK_Call_5599FCED3AE4474F'.
Cannot insert duplicate key in object 'dbo.Call'. The duplicate key
value is (6d078f37-29e6-4e56-9853-d793994ce163). The statement has
been terminated."
or something like this.
Now, since it is almost impossible to google questions on the similar topic, I am doing something seriously wrong. What is it?

compiled query only scalar parameters are allowed !

I am using POCO objects in EF 4 without any T4 template generation.
I have a DataContext class that encapsulates all ObjectSets, something like this
public sealed class DataContext :IDisposable
{
public IObjectSet GetObjectSet() where T : MyBase
{
object objectSet = null;
this.objectSets.TryGetValue(typeof(T), out objectSet);
if (objectSet == null)
{
objectSet = this.context.CreateObjectSet();
this.objectSets.Add(typeof(T), objectSet);
}
return (IObjectSet)objectSet;
}
public ObjectContext ObjectContext
{
get
{
return this.context;
}
}
}
When i write the following compiled query and try to pass in this class as one of the parameters, it gives me a runtime error saying only scalar parameters are allowed
static readonly Func<ObjectContext , DataContext, string, int?> getOperationByOrchestrationName
= CompiledQuery.Compile(
(ObjectContext ctx, DataContext container, string name) =>
(from or in container.GetObjectSet<MyOrClass>()
join op in container.GetObjectSet<MyOpClass>()
on or.Id equals op.Id
where op.Name == name
select op.Id).FirstOrDefault()
);
If i modify the query like this it works, but i deeply suspect its being compiled every time, since i am not seeing the performance boost i would see from a compiled query, can someone point out whats going on ?
static readonly Func, IObjectSet, string, IQueryable>
getOperationByOrchestrationName
= CompiledQuery.Compile(
(ObjectContext ctx, IObjectSet ors, IObjectSet ops,string operationName) =>
from or in ors
join op in ops
on or.Id equals op.Id
where op.Name == name
select op.Id
);
for anyone interested, if you return IQueryable from compiled query and call any of the methods that can change the query ( singleordefault, or firstordefault etc), you would not get benefit of a compiled query.

Resources