Can you perform a Sql Bulk Insert of a Geometry data type with entity framework 2.2 with NetTopologySuite? - spatial

I'm using EF net core 2.2 and NetTopologySuite and trying to merge some attribute data with the associated geometry data. The attribute data is in text file format, and I can read that in fine. The geometry data is in its own table, and I can read that from the MS-Sql server just fine. The linq join works fine and this process completes in ~11 seconds.
The relevant code lines are:
The goal is to write the newly created list back into its own table in MS-Sql, and if I foreach over the loop the individual items will write to the table (45 min later). Bulk insert (using a dataadapter method or EFCore.Bulkinsert methods both fail -- they don't seem to want to put the geometry type into the table.
// This starts the update process.
// Truncate the table (its faster).
Console.WriteLine("Beginning data update");
context.Database.ExecuteSqlCommand("Truncate table Tax_Parcels");
// context.BulkInsert(result); This is generating an error on insert from nettopologysuite
/*
* Also generates error on insert. I'm thinking bulk insert will have to wait to net core 3 releases ...
*
var objBulk = new BulkUploadToSql<Parcel>()
{
InternalStore = parcels,
TableName = "BOA_Staging",
CommitBatchSize = 10000,
ConnectionString = "Data Source=localhost;Initial catalog=<deleted>; UseTrustedConnection=True;"
};
objBulk.Commit();
*/
// standard add works, but takes 45 min to process records ><
int counter = 0;
int loop = 0;
foreach(var item in result)
{
context.Add(item);
counter++;
if (counter > 1000)
{
context.SaveChanges();
counter = 0;
loop++;
Console.WriteLine("I've added " + loop.ToString() + "000 Records so far");
}
}
Console.WriteLine("Done. I've added " + loop.ToString() + counter.ToString("{0,3}") + " Records.");
context.SaveChanges();
Console.WriteLine("Data update complete");

Workaround:
Make a staging table for the attributes and load the text file data into a SQL table. Do any conversions needed during the import process.
Create a view to join the table with the geometry field to the attribute table.
Create a combined view table (if you don't have one already)
Create a stored procedure to insert the data from the view into the combined view table.
In the update program, use context.Database.ExecuteSql to truncate the staging table and combined view table, run the update to the staging table to load the new data, then trigger the stored procedure to update the combined view table.
Stats:
Using EF.Add() - 45 min.
Using workaround above 45 sec.

Related

Entity Framework returns zero/null values for columns after 1st column

I am using EF6.1.2, database-first.
In one situation, this linq to sql query:
var data = from dl in ctx.ObjectContext.DocumentLines.Where(drow => drow.DocumentId == documentId)
join di in ctx.ObjectContext.DocumentsInfoes on dl.DocumentId equals di.Id
join ba in ctx.ObjectContext.Addresses on dl.DeliveryAddressId equals ba.Id into tba
from xba in tba.DefaultIfEmpty()
join sc in ctx.ObjectContext.SalesCategories on dl.SalesCategoryId equals sc.Id into tsc
from xsc in tsc.DefaultIfEmpty()
select new { Line = dl, DocumentInfo = di, DeliveryAddress = xba, SalesCategory = xsc };
var abc = data.ToList();
is not working.
The first documentline is correctly bound, but subsequent rows have some NULL or zero properties. If I run the generated SQL query, all columns are present and correct.
Is this a bug in Entity Framework? I have downloaded the EF source and I want to step into it to attempt to understand what's happening but don't know where to start looking.
The documentline table has a timestamp Version column for concurrency purposes. I have read a few similar issues where problems with a timestamp column, or other columns, (see Entity Framework does not pull through data from some columns) could cause subsequent columns to fail.
EDIT
On further investigation, it seems that, in the one call to this code that exhibits the problem, I can step over var abc = data.ToList(); without hitting any of the breakpoints in the setters of the properties of the DocumentLine entity, in the data model. Bizarre. All other calls to the code hit the breakpoints. Why isn't it loading the entities?

How to use inner join inside the update query in SQLite in Objective C?

I want to update whole table from on query. Following is my update functionality is happened:
Database (Database A) stored in the iDevice
Temperory Database (Batabase B) downloads to the device and store in the temp folder inside the device. (Both DB has same database structure)
First I attach temp db to device database. Attached db name is SECOND1
Then I insert new records from temp db to device folder from following Insert code. It is working fine.
INSERT INTO main.fList SELECT * FROM SECOND1.fList WHERE not exists (SELECT 1 FROM main.fList WHERE main.fList.GUID = SECOND1.fList.GUID)
But when I use following code to update it is not working fine. It update same top value for all device db table's rows.
UPDATE fList SET Notes = (SELECT SECOND1.fList.Notes FROM SECOND1.fList WHERE SECOND1.fList.GUID = fDefectList.GUID) WHERE EXISTS (SELECT * FROM SECOND1.fList WHERE SECOND1.fList.GUID = fList.GUID
I found SQL query for bulk update. Following is the code,
UPDATE fTempRank7
SET
fTempRank7.int_key = fRank7.int_key,
fTempRank7.int_rank6 = fRank7.int_rank6,
fTempRank7.title = fRank7.title,
fTempRank7.sequence = fRank7.sequence,
fTempRank7.lastupdated = fRank7.lastupdated
FROM
fTempRank7 INNER JOIN fRank7 ON
fTempRank7.int_key = fRank7.int_key
But in sqlite this code does not work.
Anyone knows bulk update in sqlite?
SQLite does not support joins in an UPDATE statement.
When you use a table name without a database name, it refers to the first matching table in the innermost query. In other words, if you use SECOND1.fList in a subquery, any other occurrence of fList refers to the table in SECOND1.
To ensure that you always refer to the correct table, use the database name in all table references.
The main database is always named main, so all table references should be either main.fList or SECOND1.fList.
Anyway, if you are updating all columns, you can simplify the update by deleting the rows that would be updated, so that all new data can just be inserted:
BEGIN;
DELETE FROM fList WHERE GUID IN (SELECT GUID FROM SECOND1.fList);
INSERT INTO fList SELECT * FROM SECOND1.fList;
COMMIT;
When you have a UNIQUE constraint on the GUID column, this can be simplified into a single statement:
INSERT OR REPLACE INTO fList SELECT * FROM SECOND1.fList;
And here I don't use main. because I know what I'm doing. ☺

Firebird: simulating create table as?

I'm searching a way to simulate "create table as select" in Firebird from SP.
We are using this statement frequently in another product, because it is very easy for make lesser, indexable sets, and provide very fast results in server side.
create temp table a select * from xxx where ...
create indexes on a ...
create temp table b select * from xxx where ...
create indexes on b ...
select * from a
union
select * from b
Or to avoid the three or more levels in subqueries.
select *
from a where id in (select id
from b
where ... and id in (select id from c where))
The "create table as select" is very good cos it's provide correct field types and names so I don't need to predefine them.
I can simulate "create table as" in Firebird with Delphi as:
Make select with no rows, get the table field types, convert them to create table SQL, run it, and make "insert into temp table " + selectsql with rows (without order by).
It's ok.
But can I create same thing in a common stored procedure which gets a select sql, and creates a new temp table with the result?
So: can I get query result's field types to I can create field creator SQL from them?
I'm just asking if is there a way or not (then I MUST specify the columns).
Executing DDL inside stored procedure is not supported by Firebird. You could do it using EXECUTE STATEMENT but it is not recommended (see the warning in the end of "No data returned" topic).
One way to do have your "temporary sets" would be to use (transaction-level) Global Temporary Table. Create the GTT as part of the database, with correct datatypes but without constraints (those would probably get into way when you fill only some columns, not all) - then each transaction only sees it's own version of the table and data...

Left Outer Join with IEnumerable

using Entity Framework 4.1 is it possible to perform a left outer join with an IEnumerable?
For example, lets say i want to count the number of orders for each month.
var range = Enumerable.Range(1,6).AsQueryable();
var result = range
.GroupJoin(context.Orders, i => i, o => o.Month, new
{
Month = i,
Count = m.Count()
});
Which would produce the following
Month, Count
1, 0
2, 0
3, 10
4, 20
5, 0
6, 0
Only if you call context.Orders.ToList() and load all orders to your application. IEnumerable is in your application memory and context.Orders represent data in your database. You cannot join data in database with data in memory of your application. Converting IEnumerable by AsQueryable will not put it to database.
To do your requirement directly with ADO.NET and SQL you must either create and fill temporary table and perform a database join or create stored procedure with table valued parameter to perform database join. Neither technique is supported by EF.

How can I set a many-to-many EntityCollection in Entity Framework efficiently?

When Entity Framework generates an ObjectContext for a two database tables (let's say Table1 and Table2) connected with a many-to-many relationship table, it doesn't create an object for the xref table, opting instead for collection properties on either end of the relationship. So on Table1 you have EntityCollection<Table2> Table2s and on Table2 you have EntityCollection<Table2> Table1s. In most cases that's actually pretty great...
However, in this scenario, I have a list of integers that represent the database IDs of the Table2 rows that should be in the Table1.Table2s collection.
I can't see any way to just set that collection using the entity keys, so I'm stuck selecting these into the ObjectContext, which is already a ton of work to do for no reason. I let myself hope that LINQ-to-Entities will intelligently defer the execution and perform it all on the SQL server like I would like (though my Where uses Contains, which may or may not be correctly translated to IN() in SQL). So I can go as far as:
table1instance.Table2s.Clear();
var table2sToInclude = context.Table2s.Where(
t =>
listOfTable2DatabaseIds.Contains(t.Id));
But there's no EntityCollection<T>.AddRange(IEnumerable<T>) or anything, nor is there an IEnumerable<T>.ToEntityCollection<T>() extension method of course, so I don't know what to do with these results at this point. All I can do is
foreach (var table2 in table2sToInclude)
{
table1instance.Table2s.Add(table2);
}
which seems ridiculous and I know will force a lot of unnecessary evaluation.
Is there a "correct", or, perhaps, "less lame" way to do this?
No EF will not defer any query execution. There is nothing like insert from select. Linq-to-entities is just query language and responsibility of query is to execute. It is strictly separated from persistence functionality offered by EF itself.
If you want to create relations between existing item from table1 and exiting items from table2 you can use code like this:
using (var ctx = new YourContext())
{
var table1 = new Table1 { Id = 123 };
ctx.Table1s.Attach(table1);
foreach (var table2 in table2sToInclude.Select(id => new Table2 { Id = id }))
{
ctx.Table2s.Attach(table2);
order.Table2s.Add(table2);
}
ctx.SaveChanges();
}
This code creates relation between Table1's item with id 123 and all Table2's items from table2sToInclude without loading any single record from the database.
What makes adding records one by one "lame"? Do you understand what is benefit of AddRange? AddRange in typical collection extends capacity of internal array and just copy items to extended array. EntityCollection is not typical array and it must process each added entity. So even if there will be some AddRange it will internally iterate items and process them on by one.

Resources