"Invalid column name" with Entity Framework migrations - entity-framework-6

I'm having an issue with updating a database with a migration that adds a column to a table and assign values to it. I have reduced the issue to a very simple case.
Here's the model:
public class Model
{
public int Id { get; set; }
public int Col2 { get; set; }
}
This is the context:
public class Context : DbContext
{
public DbSet<Model> Models { get; set; }
}
First, I enabled migrations, created an initial migration and create a database.
PM> Enable-Migrations
PM> Add-Migration -Name Initial
PM> Update-Database
Then, I extended my model:
public int Col3 { get; set; }
and created a new migration:
PM> Add-Migration -Name AddedCol3
I modified that migration to update the values in Col3, see the call to Sql():
public override void Up()
{
AddColumn("dbo.Models", "Col3", c => c.Int(nullable: false));
Sql("update dbo.Models set Col3 = Col2");
}
public override void Down()
{
DropColumn("dbo.Models", "Col3");
}
When I update the database with this migration, I get:
Msg 207, Level 16, State 1, Line 2 Invalid column name 'Col3'.
The generated script is:
ALTER TABLE [dbo].[Models] ADD [Col3] [int] NOT NULL DEFAULT 0
update dbo.Models set Col3 = Col2
-- Removed update to migration history.
Apparently, SQL Server cannot handle the alter table and update in one batch.
I tried to add SQL("GO"); in between, but this results in the error:
The argument 'sql' cannot be null, empty or contain only white space.
when trying to update.
How can I use migrations to achieve this. I want my Up() and Down() methods to consistently update the database.

The work-around provided by #scsimon translates to the SQL-statement
Sql("EXEC('update dbo.Models set Col3 = Col2')");
and script
ALTER TABLE [dbo].[Models] ADD [Col3] [int] NOT NULL DEFAULT 0
EXEC('update dbo.Models set Col3 = Col2')
and works fine.
I prefer this method, as I expect it to be less fulnerable to EF modifications.

I found a work-around by adding a dummy SQL-statement:
Sql("select 1\nGO");
The resulting script is:
ALTER TABLE [dbo].[Models] ADD [Col3] [int] NOT NULL DEFAULT 0
GO
select 1
update dbo.Models set Col3 = Col2
Note that the GO statement has automagically been moved to another location!
Anyway, the script is now accepted by SQL Server and the changes are applied the way they are itended.

Related

The required column 'xxxID' was not present in the results of a 'FromSql' operation - .Net Core EF

I am trying to find better and performance efficient approach for bulk delete in .NET Core EF (3.1.9). (Approx 500K to 1 Mil records to be deleted in one shot.)
Model:
public class Employee
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int EmpID { get; set; }
public string EmpName { get; set; }
}
And database table as:
CREATE TABLE [dbo].[Employee]
(
[EmpID] [int] IDENTITY(1,1) NOT NULL,
[EmpName] [nchar] (20) NULL,
CONSTRAINT [PK_dbo].[Employee]
PRIMARY KEY CLUSTERED ([EmpID] ASC)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
Trying to delete records with generic method-1.
public int Delete<TEntity>(Func<TEntity, bool> predicate) where TEntity: class
{
return DbContext.Set<TEntity>
.FromSqlRaw($"Delete from dbo.Employee")
.Where(predicate).Count();
}
And calling this method as
Func<Employee, bool> myPredicate = x => x.EmpID > 10;
int deletedCount = myclass.Delete(myPredicate);
Exception thrown:
InvalidOperationException: The required column 'EmpID' was not present in the results of a 'FromSql' operation.
ASP.NET Core EF generates query:
DELETE FROM [dbo].[Employee]
fail: Microsoft.EntityFramework.Query[10100] .... stack trace with above error.
Already looked into this:
.NET Core Entity Framework InvalidOperationException
The required column 'CustomerId' was not present in the results of a 'FromSql' operation
The required column 'id' was not present in the results of a `FromSql` operation in EFcore
So not sure why it is throwing above error. As database has the correct PK and model also has it. Tried with some other entities too, it always throws the same error with column name 'xxxEntityColID'.
However, if I use following code then it works:
public int Delete<TEntity>(string whereCondition, params object[] parameters) where TEntity: class
{
return DbContext.Database.ExecuteSqlRaw($"Delete from dbo.Employee WHERE {whereCondition}", parameters);
}
// and calling like
string myCondition = "EmpID > 10";
int deletedCount = myclass.Delete<Employee>(myCondition, new object[0]);
.NET Core EF generates following SQL and work without any error.
DELETE FROM [dbo].[Employee] WHERE EmpID > 10;
Questions
Why does this 'xxxEntityColID' error occur?
What is the better approach to do bulk deletes in .NET Core EF?
Looks like FromSqlRaw() is worked with "SELECT.." queries.
For Bulk delete I evaluated following approach and it is better than dbContext.RemoveRange(entities); Because RemoveRange() generated separate query for each entity. Then every single query takes some time, so not suitable for Bulk delete. In contrast ExecuteSqlRaw(with where condition), just generates single query which hardly takes 250ms for 200K records. On high end servers it will take lesser for sure.
public int Delete<TEntity>(string whereCondition, params object[] parameters) where TEntity: class
{
return DbContext.Database.ExecuteSqlRaw($"Delete from dbo.Employee WHERE {whereCondition}", parameters);
}

How to use COLLATE Latin1_General_bin in Entity framework?

I have to use COLLATE in entity framework query. How to write SQL query equivalent in Entity Framework as show below code?
SQL query:
select * from AspNetUsers order by Email COLLATE Latin1_General_bin
Entity Framework:
using (var db = new testEntities())
{
var appUsers = await db.Users.OrderBy(x => x.Email).ToListAsync();
}
It's possible to use Entity Framework's interception hooks.
The first step it to define an interface:
interface ISortInterceptable
{
IEnumerable<string> AdaptableSortFieldNames { get; set; }
}
Then make your context implement it:
class TestEntities : DbContext, ISortInterceptable
{
...
public IEnumerable<string> AdaptableSortFieldNames { get; set; }
...
}
Next, create a command interceptor:
class SortCommandInterceptor : DbCommandInterceptor
{
public override void ReaderExecuting(DbCommand command,
DbCommandInterceptionContext<DbDataReader> interceptionContext)
{
if (interceptionContext.DbContexts.First() is ISortInterceptable interceptable
&& interceptable.AdaptableSortFieldNames != null)
{
var query = command.CommandText;
foreach (var fieldName in interceptable.AdaptableSortFieldNames)
{
var pattern = $#"(.*\s*ORDER BY\s*.*\.)(\[{fieldName}\])(.*)";
query = Regex.Replace(query, pattern, "$1$2 COLLATE Latin1_General_bin $3");
}
command.CommandText = query;
}
base.ReaderExecuting(command, interceptionContext);
}
}
This is where all the magic happens.
The interceptor first checks if it has to do with a ISortInterceptable (maybe this check can be refined by getting all ISortInterceptables from interceptionContext.DbContexts).
The command text in the command to be executed is analyzed on any occurence of strings like ORDER BY [Alias].[fieldName] where fieldName is a variable. This search pattern is in keeping with the pattern EF always follows to generates queries.
The field name part of the ORDER BY clause, which is in the third group ($2) of the regex match, is extended by the collation phrase.
The replacement is repeated for all field names.
Finally, an example of how to use this interceptor:
DbInterception.Add(new SortCommandInterceptor());
using (var db = new TestEntities())
{
db.AdaptableSortFieldNames = new[] { "LastName", "Email" };
var users = db.AspNetUsers
.OrderBy(u => u.LastName)
.ThenBy(u => U.Email)
.ToList();
}
As always with string manipulation, one caveat: this works in a couple of my own tests, but I can't guarantee it to be rock solid. For one, the sorting fields should be text fields, of course.
One last note. EF core 3 also offers interception hooks that can be used in a similar way.

Map SQL columns in SqlMapper.Query<T1,T2,TReturn> to class properties

First I do not have much experience with dapper.net and maybe this problem has simple solution.
I would like use multi mapping features.
I simplified my problem.
I have 2 tables.
First table
CARS
Columns:
CARD_ID (PK)
CAR_LOCATION
CAR_STATUS
Second table
BRANDS
Columns:
ID (PK)
CARD_ID (FK)
BRAND_NAME
BRAND_LOGO
I need execute this query:
SQL_CMD:
SELECT * FROM CARS
LEFT JOIN BRANDS
ON CARS.CARD_ID = BRANDS.CARD.ID;
In .NET application I map these tables on 2 POCO classes.
public class Car
{
public int CarId {get;set}
public string CarLocation {get;set;}
public string CarStatus {get;set;}
public Brand Brand {get;set;}
}
public class Brand
{
public int Id {get;set}
public int CardId {get;set;}
public string BrandName {get;set;}
public string BrandLogo {get;set;}
}
When I query sql cmd above:
var data = connection.Query<Car, Brand, Car>(SQL_CMD, (car, brand) =>
{
car.Brand = brand;
return car;
}, commandTimeout: 50000, splitOn:"ID")
I get empty result.
IMHO problem is in mapping SQL columns on class properties because if I changed prop Card.CarId to Car.CAR_ID this properties is filled.
Is there any way how to map "SQL columns" to class properties in SqlMapper.Query<T1,T2,TReturn> to class properties?
I know that exist ClassMapper but I didn’t find way how to use in this scenario.
Thanks
EDITED:
I tried add mapping classes, but not work
public class CarMapper : ClassMapper<Car>
{
Table("CARS");
Map(c=>c.CarId).Column("CAR_ID").KeyType(KeyType.Assigned);
Map(c=>c.CarLocation).Column("CAR_LOCATION");
Map(c=>c.CarStatus).Column("CAR_STATUS");
}
public class BrandMapper : ClassMapper<Brand>
{
Table("BRANDS");
Map(c=>c.Id).Column("ID").KeyType(KeyType.Assigned);
Map(c=>c.CarId).Column("CAR_ID");
Map(c=>c.BrandName).Column("BRAND_NAME");
Map(c=>c.BrandLogo).Column("BRAND_LOGO");
}
add mapping assemblies
DapperExtensions.DapperExtensions.SetMappingAssemblies(new List<Assembly>
{
Assembly.GetAssembly(typeof(CarMapper)),
Assembly.GetAssembly(typeof(BrandMapper)),
});
and after this set up execute query
var data = connection.Query<Car, Brand, Car>(SQL_CMD, (car, brand) =>
{
car.Brand = brand;
return car;
}, commandTimeout: 50000, splitOn:"ID")
As I told not work not work properties of objects are empty but if I tried simple insert, update or delete it works.
I am confuse now where can be root of problem.
Dapper Extensions provides 4 extensions: Get, Insert, Update and Delete, so your second example wont work. Query extension is a part of core Dapper. As far as I can tell, Dapper Extensions does not support multi-mapping.
The only option I can think of is to modify your select statement and rename the attributes, for example:
SELECT CAR_ID as CarId, CAR_LOCATION as CarLocation, CAR_STATUS as CarStatus, etc.
FROM CARS
LEFT JOIN BRANDS
ON CARS.CARD_ID = BRANDS.CARD.ID
Then multi-mapping should work.
Also, you don't need to specify splitOn if you are splitting by ID. Dapper splits on Id fields automatically.
Dapper assumes your Id columns are named "Id" or "id", if your primary key is different or you would like to split the wide row at point other than "Id", use the optional 'splitOn' parameter.

Possible to default DateTime field to GETDATE() with Entity Framework Migrations?

I added EntityFramework.Migrations (Beta 1) to an existing Code-First app that is going through some changes (for both migration capabilities and more fine-tuning of the tables I am generating from my code-first API) and ran into the GETDATE() scenario.
I was already using a custom initializer class in my DbContext to run SQL scripts to set some fields and create indexes in my database. A handful of my AlterTable scripts are primary just to setup fields with default values(such as certain DateTime fields being set to GETDATE()). I was really hoping EntityFramework.Migrations would have an answer for this since you can easily specify defaultValue, but so far I'm not seeing one.
Any ideas? I was really hoping that doing the following would magically work. (It is 'magic unicorn', after all)
DateCreated = c.DateTime(nullable: false, defaultValue: DateTime.Now)
Unfortunately, and logically, it set my default value to the time when the Update-Database command was executed.
You can use
DateCreated = c.DateTime(nullable: false, defaultValueSql: "GETDATE()")
Usage:
public partial class MyMigration : DbMigration
{
public override void Up()
{
CreateTable("dbo.Users",
c => new
{
Created = c.DateTime(nullable: false, defaultValueSql: "GETDATE()"),
})
.PrimaryKey(t => t.ID);
...
Update 2012-10-10:
As requested by Thiago in his comment, I add a little extra context.
The code above is a migration-file generated by EF Migrations by running Add-Migration MyMigration as a command in the package manager console. The generated code is based on the models in the DbContext associated with migrations. The answer suggests that you modify the generated script so that a default value is added when the database is created.
You can read more about Entity Framework Code First Migrations here.
I recently ran in to this issue in EF6 (since they still haven't fixed it). The easiest way I found to do it without having to manually modify the Migration class is to override the CodeGenerator in your Configuration class.
By creating a class that implements MigrationCodeGenerator and then overriding the Generate method you can iterate through all of the operations and apply what ever modifications you want.
Once your modifications have been made, you can then initialize your the CSharpMigrationCodeGenerator and return its default value.
public class ExtendedMigrationCodeGenerator : MigrationCodeGenerator
{
public override ScaffoldedMigration Generate(string migrationId, IEnumerable<MigrationOperation> operations, string sourceModel, string targetModel, string #namespace, string className)
{
foreach (MigrationOperation operation in operations)
{
if (operation is CreateTableOperation)
{
foreach (var column in ((CreateTableOperation)operation).Columns)
if (column.ClrType == typeof(DateTime) && column.IsNullable.HasValue && !column.IsNullable.Value && string.IsNullOrEmpty(column.DefaultValueSql))
column.DefaultValueSql = "GETDATE()";
}
else if (operation is AddColumnOperation)
{
ColumnModel column = ((AddColumnOperation)operation).Column;
if (column.ClrType == typeof(DateTime) && column.IsNullable.HasValue && !column.IsNullable.Value && string.IsNullOrEmpty(column.DefaultValueSql))
column.DefaultValueSql = "GETDATE()";
}
}
CSharpMigrationCodeGenerator generator = new CSharpMigrationCodeGenerator();
return generator.Generate(migrationId, operations, sourceModel, targetModel, #namespace, className);
}
}
internal sealed class Configuration : DbMigrationsConfiguration<Project.Models.Context.DatabaseContext>
{
public Configuration()
{
AutomaticMigrationsEnabled = false;
MigrationsDirectory = #"Migrations";
this.CodeGenerator = new ExtendedMigrationCodeGenerator();
}
}
I hope this helps
You must use custom SQL script in Up method for setting default value:
Sql("ALTER TABLE TableName ADD CONSTRAINT ConstraintName DEFAULT GETDATE() FOR ColumnName");
Setting default value in code allows only static values - no database level functions.
Anyway setting it in POCO constructor is correct way if you are going to use code first. Also if you want to set the value in the application for some special cases you cannot use a default value in the database because the default value in the database requires either DatabaseGeneratedOption.Identity or DatabaseGeneratedOption.Computed. Both these options allow setting the property only in the database.
Edit:
Since the product is still in development my answer is no longer valid. Check #gius answer for actual way to achieve this requirement by using defaultValueSql (it wasn't available in EF Migrations Beta 1 but was added in EF 4.3 Beta 1 which already includes migrations).
Create a migration:
public partial class Table_Alter : DbMigration
{
public override void Up()
{
AddColumn("dbo.tableName", "columnName",
c => c.DateTime(nullable: false, defaultValueSql: "GETDATE()"));
}
public override void Down()
{
DropColumn("dbo.tableName", "columnName");
}
}
For existing records it will set the datetime when you will run Update-Database command, for new records it will be set the datetime of creation
Alternatively if your entities inherit from a common interface you can override the SaveChanges method on the DbContext and set or update properties at that point (great for Created Date and Last Changed Date)
This is the most simple way.
First Add DatabaseGeneratedOption.Computed DataAnnotion to your property
and now you can modify de SqlServerMigrationSqlGenarator, override Genarate method and set the DefaultValueSql = "GETDATE()" or "GETUTCDATE()";
Using Entity Framework with .net 6 I was able to make the migration add the default time by changing the defaultValue to defaultValueSql:
public partial class ReportQueueAddCreatedAt : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<DateTime>(
name: "CreatedAt",
table: "ReportQueue",
type: "datetime2",
nullable: false,
defaultValueSql: "GETDATE()");
}
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "CreatedAt",
table: "ReportQueue");
}
}
An improvement: check if the constraint exists:
Sql(#"
if not exists (
select *
from sys.all_columns c
join sys.tables t on t.object_id = c.object_id
join sys.schemas s on s.schema_id = t.schema_id
join sys.default_constraints d on c.default_object_id = d.object_id
where
d.name = 'DF_ThubOutputEmail_Created'
)
begin
ALTER TABLE dbo.ThubOutputEmails ADD CONSTRAINT DF_ThubOutputEmail_Created default getdate() for Created;
end");

Creating history table using Entity Framework 4.1

I am working on asp.net MVC 3 application and I am using codeFirst approach. I am trying to create history table or user table, Where I want to keep track of what columns were modified by user. How can I do this using EF Code First.
Do I need to do it after DataContext.savechanges ?
Please suggest.
Thanks.
The DbContext has a method called Entry<T>:
var entity = context.Items.Find(id);
entity.Name = "foobar";
var entry = context.Entry<Item>(entity);
entry will be of type DbEntityEntry<T> and has the properties OriginalValues and CurrentValues.
You could probably write something that will generically inspect these properties to see what has changed and then automatically insert a new record into your history table.
Either that, or use database triggers.
I'm not sure if this is really the "appropiate" way to do it, but this is how its usually done in sql:
Create an extra property version of type int or something.
Because you probably do not want to loop every time, add another property IsLatestVersion of type bool
When an entity is saved, check if the entity already exists. If so, set the entity on IsLatestVersion = false.
Increment the version, and save the changes as new entity.
Sounds to me like you want an a filter that inherits from ActionFilterAttribute. In my case, this is the simplest example that I have. This is my model, notice that the attributes dictate the mapping to the database.
[Table("UserProfile")]
public class UserProfile
{
[Key, DatabaseGeneratedAttribute(DatabaseGeneratedOption.Identity)]
public int UserId { get; set; }
public string UserName { get; set; }
}
In my case, it was as simple as the following, although it was not historical:
public sealed class UsersContext : DbContext
{
public UsersContext() : base("DefaultConnection")
{
}
public DbSet<UserProfile> UserProfiles { get; set; }
}
public override void OnActionExecuting(ActionExecutingContext filterContext)
{
LazyInitializer.EnsureInitialized(ref _initializer, ref isInitialized, ref initializerLock);
}
public void CheckDatabase()
{
Database.SetInitializer<YourDBContextType>(null);
using (var context = new YourDBContextType())
{
if (!context.Database.Exists())
{
((IObjectContextAdapter)context).ObjectContext.CreateDatabase();
}
}
// Uses your connection string to build the following table.
WebSecurity.InitializeDatabaseConnection("DefaultConnection", "UserProfile", "UserId", "UserName", autoCreateTables: true);
}
The end result is not only EF being code first, but also allows for your models for your views to use primitives derived from your complex entities. Now, if you have another, lets say historical, DBContext then I would recommend modifying either the text transformation file or creating a base class for your entities. Both ways allow for an easy generation of code that could insert into your table, then follow up with that entity, clone it into a historical model and save. All that being said, I am a fan of database first approaches with concentration on constraints, triggers, etc. instead of a framework.

Resources