I am trying to create a left join with an insert. There are two tables table.a table.b, the id value (which is auto increment) in table.b is used as a foreign key in table.a. Because of a broken process, I need to insert into both tables in one pass (I am creating an work around to process using a batch query). But I need the auto incremented value to be passed as well. Is this even possible. I have seen several answers that come close.
Related
How to change this stored procedure without drop tblPurchase and tblProductStock.
When I run my program with this stored procedure after adding new record table and data will be drop. I want to keep all table and data protected. please help me to resolve that.
https://stackoverflow.com/a/65774309/13338320
Indexed View
An entirely new solution based on Indexed Views is possible.
An Indexed View is a view which has a clustered index on it, and the data is actually stored on disk.
As I understand it, you are trying to keep a sum of purchases per product item stored in tblProduct. I have assumed that ItemCode is the PK of tblProduct and that ItemName is also defined there (We cannot use MAX in an indexed view). So we can define a view like this:
CREATE VIEW dbo.vwTotalPurchases
WITH SCHEMABINDING -- must be schema bound, we cannot change underlying columns after creation
AS
SELECT
ItemCode,
SUM(Quantity) QuantityPurchased,
COUNT_BIG(*) CountPurchases -- if we group, must have count also, so that rows can be maintained
FROM dbo.tblPurchase -- must use two-part names
GROUP BY itemCode;
GO
We can then create a clustered index on it to persist it on disk. SQL Server will maintain the index whenever an update to the base table happens. If there are no more rows in the grouping (identified by count being 0), then the row is deleted:
CREATE UNIQUE CLUSTERED INDEX PK_vwTotalPurchases ON dbo.vwTotalPurchases (ItemCode);
GO
Now if we want to query it, we can left join this view onto tblProducts (left join because there may be no purchases):
SELECT
p.ItemCode,
p.ItemName,
ISNULL(tp.QuantityPurchased, 0) QuantityPurchased,
ISNULL(tp.CountPurchases, 0) CountPurchases
FROM tblProducts p
LEFT JOIN vwTotalPurchases tp WITH (NOEXPAND) ON tp.ItemCode = p.ItemCode;
We can define this as a view also (not an indexed one, but a standard view) so that the definition is usable anywhere.
Note on NOEXPAND:
If you are not on SQL Server Enterprise or Developer Edition, you must use the hint WITH (NOEXPAND) to force it to use the index, otherwise it will query the base tblPurchase instead. And even in those editions, it is best to use NOEXPAND.
See this article by Paul White on this.
Where I work we initialize ("INIT" => truncate & load) a Fact\ Dimension table on the rare occasions when the need arise.
Such "INIT" requires all object referencing the initialized object to be initialized subsequently in order to preserve RI (Referential Integrity).
Example – we found a bug in an attribute rich (~25) SCD Dim_Employees which will change the effective dates of each record. This requires all referencing objects to recalculate their Foreign Keys.
Do you have the same situation and if so how do you manage it ?
We have redimensioned our data multiple times as we change from SCD2/SCD1 to SCD6 or found things wrong with one of the data streams, like you mention.
It's not too hard to remap your data, you just need to make before you truncate you data in your INIT step to clone it, (or clone within your time travel window) then you join your fact table to the old dimension on the old dimension key, and then join to you new dimension via the old dimension foreign key and time, now you know the old key to new key mapping. Now that becomes the source of your update, and if you do it one operate with all other ETL operations paused, you have maintained your data integrity.
If you cannot pause the live ETL processes, you will a multi step update where you have your new divisional table and old one live, and do your normal mapping to both, and fix your reads to coalesce the results, and then once your new facts are correctly mapping to new dimension, turn around and back-fill all the old facts with ND keys, and then you have no ND gaps, then you can stop the OD/ND coalesce, and then stop the OD mapping and drop the OD column...
I am developing an application which is very similar to todo list in its nature, except order of todos matters and can be changed by user.
What's a good way to save this order in db without having to re-save whole todo list upon change of order?
I am developing in Rails, Postgres and React, newest versions.
I am thinking to save it as an array in User Todos (there can be multiple users of the application), but I am thinking it could complicate things a little as every time I create a todo I would have to save the List also.
You can look into acts_as_list gem and for this you'll have to add an additional column position in your table. But this will do mass update on the records. But this gem is frequently updated.
If you want an optimised solution and minimise the number of updates on changing the list then you should check ranked_model gem but this one is not frequently updated. There is a brief on how it works :-
This library is written using ARel from the ground-up. This leaves the code much cleaner than many implementations. ranked-model is also optimized to write to the database as little as possible: ranks are stored as a number between -2147483648 and 2147483647 (the INT range in MySQL). When an item is given a new position, it assigns itself a rank number between two neighbors. This allows several movements of items before no digits are available between two neighbors. When this occurs, ranked-model will try to shift other records out of the way. If items can't be easily shifted anymore, it will rebalance the distribution of rank numbers across all members of the ranked group.
You can refer this gem and make your own implementation as it only supports rails 3 & 4.
This was a bit of a head scratcher but here is what I figured:
create table orderedtable (
pk SERIAL PRIMARY KEY,
ord INTEGER NOT NULL,
UNIQUE(ord) DEFERRABLE INITIALLY DEFERRED
)
DEFERRABLE INITIALLY DEFERRED is important so that intermediate states don't cause constraint violations during reordering.
INSERT INTO orderedtable (ord) VALUES (1),(2),(3),(4),(5),(10),(11)
Note that when inserting in this table it would be more efficient to leave gaps between ord values so as to minimize the amount of order values that need to be shifted when inserting or moving rows later. The consecutive values are for demonstration purposes.
Here's the trick: You can find a consecutive sequence of values starting at a particular value using a recursive query.
So for example, let's say you wanted to insert or move a row just above position 3. One way would be to move rows currently at position 4 and 5 up by one to open up position 4.
WITH RECURSIVE consecutives(ord) AS (
SELECT ord FROM orderedtable WHERE ord = 3+1 --start position
UNION ALL
SELECT orderedtable.ord FROM orderedtable JOIN consecutives ON orderedtable.ord=consecutives.ord+1 --recursively select rows one above, until there is a hole in the sequence
)
UPDATE orderedtable
SET ord=orderedtable.ord+1
FROM consecutives
WHERE orderedtable.ord=consecutives.ord;
The above renumbers the ord from 1,2,3,4,5,10,11 to 1,2,3,5,6,10,11 leaving a hole at 4.
If there was already a hole at ord=4 , the above query wouldn't have done anything.
Then just insert or move another row by giving it the now free ord value of 4.
You could push rows down instead of up by changing the +1s to -1s.
I have an SSIS routine that reads from a very dynamic table and inserts whichever rows it finds into a table in a different database, before truncating the original source table.
Due to the dynamic nature of the source table this truncation not surprisingly leads to rows not making it to the second database.
What is the best way of deleting only those rows that have been migrated?
There is an identity column on the source table but it is not migrated across.
I can't change either table schema.
A option, that might sound stupid but it works, is to delete first and use the OUTPUT clause.
I created a simple control flow that populates a table for me.
IF EXISTS
(
SELECT 1 FROM sys.tables AS T WHERE T.name = 'DeleteFirst'
)
BEGIN
DROP TABLE dbo.DeleteFirst;
END
CREATE TABLE dbo.DeleteFirst
(
[name] sysname
);
INSERT INTO
dbo.DeleteFirst
SELECT
V.name
FROM
master.dbo.spt_values V
WHERE
V.name IS NOT NULL;
In my OLE DB Source, instead of using a SELECT, DELETE the data you want to go down the pipeline and OUTPUT the DELETED virtual table. Somethinng like
DELETE
DF
OUTPUT
DELETED.*
FROM
dbo.DeleteFirst AS DF;
It works, it works!
One option would be to create a table to log the identity of your processed records into, and then a separate package (or dataflow) to delete those records. If you're already logging processed records somewhere then you could just add the identity there - otherwise, create a new table to store the data.
A second option: If you're trying to avoid creating additional tables, then separate the record selection and record processing into two stages. Broadly, you'd select all your records in the control flow, then process them on-by-one in the dataflow.
Specifically:
Create a variable of type Object to store your record list, and another variable matching your identity type (int presumably) to store the 'current record identity'.
In the control flow, add an Execute SQL task which uses a query to build a list of identity values to process, then stores them into the recordlist variable.
Add a Foreach Loop Container to process that list; the foreach task would load the current record identifier into the second variable you defined above.
In the foreach task, add a dataflow to copy that single record, then delete it from the source.
There's quite a few examples of this online; e.g. this one from the venerable Jamie Thomson, or this one which includes a bit more detail.
Note that you didn't talk about the scale of the data; if you have very large numbers of records the first suggestion is likely a better choice. Note that in both cases you lose the advantage of the table truncation (because you're using a standard delete call).
I have around 3 Million rows in a Table in Informix DB.
We have to delete it, before loading new data.
It has a primary key on one of its columns.
For deleting the same, I thought of going with rowid usage. But when I tried
select rowid from table
it responded with -857 error [Rowid does not exist].
So, I am not sure, how to go with the deletion. I prefer not going with primary key, as deletion with primary key is costly compared with rowid deletion.
Any suggestion on the above would be helpful.
If you get error -857, the chances are that the table is fragmented, and was created without the WITH ROWIDS option.
Which version of Informix are you using, and on which platform?
The chances are high that you have the TRUNCATE TABLE statement, which is designed to drop all the rows from a table very quickly indeed.
Failing that, you can use a straight-forward:
DELETE FROM TableName;
as long as you have sufficient logical log space available. If that won't work, then you'll need to do repeated DELETE statements based on ranges of the primary key (or any other convenient column).
Or you could consider dropping the table and then creating it afresh, possible with the WITH ROWIDS clause (though I would not particularly recommend using the WITH ROWIDS clause - it becomes a physical column with index instead of being a virtual column as it is in a non-fragmented table). One of the downsides of dropping and rebuilding a table is that the referential constraints have to be reinstated, and any views built on the table are automatically dropped when the table is dropped, so they have to be reinstated too.
I'm assuming this is IDS?.. How many new rows will be loaded and how often is this process repeated?.. Despite having to re-establish referential constraints and views, in my opinion, it is much better to drop the table, create it from scratch, load the data and then create the indexes because if you just delete all the rows, the deleted rows still remain physically in the table with a NULL \0 flag at the end of the row, thus the table size will be even larger when loading in the new rows and performance will suffer!.. It's also a good opportunity to create fresh indexes, and if possible, pre-sort the load data so that its in the most desirable order (like when creating a CLUSTERED INDEX). If you're going to fragment your tables on expressions or other type, then ROWID's go out the window, but use WITH ROWIDS if you're sure the table will never be fragmented. If your table has a serial column, are there any other tables using the serial columns as a foreign key?