I want to copy records of a few employees from one company into another larger one. If a primary key duplicate conflict occurs, the record has to be replaced. For Delphi DataSet there are commands "Insert", "Append", "Edit", and "Delete", but is there an easy way to "Replace" the record between the same tables, without knowing the full table structure or primary keys? There are like 30+ fields and they may be changed in the future.
In MySQL it would be REPLACE INTO table2 (SELECT * FROM table1) but I wanted to change a few fields in the target table, like employee's ID and department codes.
I'm afraid there is no way to replace/overwrite dataset records in Delphi. But using MySQL I can select the source data into a temp table, modify the temp table data, and place it into target table.
Related
I've read that dimension tables hold the primary key and and fact tables contain the foreign key which references the primary key of Dimension tables.
Now the confusion I am having is this - suppose I have an ETL pipeline which populates the dimension table (let's say customer) from a source (say another DB). Let's assume this is a frequently changing table and has over 200 columns. How do I incorporate these changes in the dimension tables? I want to have only the latest record for each customer (type 1 SCD) in the DWH.
One thing what I could do is delete the row in the dimension table and re-insert the new updated row. But this approach won't work because of the primary key - foreign key constraint (which will not allow me to delete the record).
Should I write an update statement with all 200 columns in the ETL script? Or is there any other approach?
Strictly speaking you just need to update the fields that changed. But the cost of updating all in a single record is probably similar (assuming it’s row based storage), and it’s probably easier to write.
You can’t delete and re-insert, as the new row will have a new PK and old facts will no longer be linked.
I have a DirectQuery table (Weather) which is sourced from an Azure SQL server. I would like to join this with an Imported table (Buckles) from an Excel sheet sourced from SharePoint Online.
Both tables have a UID field that is made up of a concatenation between a SiteID and timestamp. The UID field is named differently for each table.
I have created a One-To-Many relationship between the two tables.
I have tried to create a new DAX table using a NATURALINNERJOIN on Weather and Buckles but I get this error:
"No common join columns detected. The join function 'NATURALINNERJOIN' requires at-least one common join column."
I am confident it is not a problem with the underlying data because I've created a new imported Excel table (Test) with a selection of the data from Weather and I'm able to successfully create the join on Test and Buckles.
Is the joining of DirectQuery and Imported tables supported? I feel like this may be a type casting issue, but as far as I can see, both UID fields are set as Text.
The UID field is named differently for each table.
I suspect this may be the issue. NATURALINNERJOIN looks for matching column names
and if the two tables have no common column names, an error is returned.
Note that if you create a calculated DAX table using a DirectQuery source, I don't think that table will still act like DirectQuery. If I understand correctly, it will materialize the calculated table into your model and DAX that references that calculated table no longer points back to the SQL server (and consequently will only update when the calculated table gets rebuilt).
I'm working with Delphi 7 and Firebird database. I'm using TIBDatabase, TIBTransaction, TIBQuery, TIBDataSet and DBGrid to establish connection and provide user interface for working with table. In my database I have two tables:
Ships
fields
Id integer
Name varchar(20)
Type_Id(Fk) integer
Longth integer
Ship_types
fields
Id(Pk) integer
Ship_type varchar(10)
So resulting dataset which I get through "join" query has such fields
Name
Type
Longth
Type is Ship_type field from Ship_types table joined via query by Type_Id foreign key to this table from Ships table.
Data is displaying properly.
Then I need to edit my data directly through DBGrid. For this purpose I use TIBUpdateSQL component. For displaying Type(lookup) field I chose DBGrid.Columns.PickList property.
So my question is how can I make TIBUpdateSQL work with such type of field? Cause I know that if it would be single table without foreign keys I have to write update statement into ModifySQL property of update component. But what have I do with fk fields? Can I write update join statement in UpdateSQL component or, if not, what else way I can do it?
I don't need to update two tables, I just need to update only Ships table but there is varchar(word representation) field in displaying dataset and in updating dataset it must be integer(corresponding id) to suit to table structure.
Editor in TIBUpdateSQL isn't solution for me cause I'm assigning query to TIBQuery on runtime.
You can't update tables using select with JOIN, only with subselects.
Sub-select example:
SELECT TABLE_NAME.*
, (SELECT TABLE_NAME2.NAME FROM TABLE_NAME2 WHERE TABLE_NAME2.ID = TABLE_NAME.ID)
FROM TABLE_NAME
We haave Accounts, Deals, Contacts, Tasks and some other objects in the database. When a new organisation we want to set up some of these objects as "Demo Data" which they can view/edit and delete as they wish.
We also want to give the user the option to delete all demo data so we need to be able to quickly identify it.
Here are two possible ways of doing this:
Have a "IsDemoData" field on all the above objects : This would mean that the field would need to be added if new types of demo data become required. Also, it would increase database size as IsDemoData would be redundant for any record that is not demo data.
Have a DemoDataLookup table with TableName and ID. The ID here would not be a strong foreign key but a theoretical foreign key to a record in the table stated by table name.
Which of these is better and is there a better normalised solution.
As a DBA, I think I'd rather see demo data isolated in a schema named "demo".
This is simple with some SQL database management systems, not so simple with others. In PostgreSQL, for example, you can write all your SQL with unqualified names, and put the "demo" schema first in the schema search path. When your clients no longer want the demo data, just drop the demo schema.
I have a Join table in Rails which is just a 2 column table with ids.
In order to mass insert into this table, I use
ActiveRecord::Base.connection.execute("INSERT INTO myjointable (first_id,second_id) VALUES #{values})
Unfortunately this gives me errors when there are duplicates. I don't need to update any values, simply move on to the next insert if a duplicate exists.
How would I do this?
As an fyi I have searched stackoverflow and most the answers are a bit advanced for me to understand. I've also checked the postgresql documents and played around in the rails console but still to no avail. I can't figure this one out so i'm hoping someone else can help tell me what I'm doing wrong.
The closest statement I've tried is:
INSERT INTO myjointable (first_id,second_id) SELECT 1,2
WHERE NOT EXISTS (
SELECT first_id FROM myjointable
WHERE first_id = 1 AND second_id IN (...))
Part of the problem with this statement is that I am only inserting 1 value at a time whereas I want a statement that mass inserts. Also the second_id IN (...) section of the statement can include up to 100 different values so I'm not sure how slow that will be.
Note that for the most part there should not be many duplicates so I am not sure if mass inserting to a temporary table and finding distinct values is a good idea.
Edit to add context:
The reason I need a mass insert is because I have a many to many relationship between 2 models where 1 of the models is never populated by a form. I have stocks, and stock price histories. The stock price histories are never created in a form, but rather mass inserted themselves by pulling the data from YahooFinance with their yahoo finance API. I use the activerecord-import gem to mass insert for stock price histories (i.e. Model.import columns,values) but I can't type jointable.import columns,values because I get the jointable is an undefined local variable
I ended up using the WITH clause to select my values and give it a name. Then I inserted those values and used WHERE NOT EXISTS to effectively skip any items that are already in my database.
So far it looks like it is working...
WITH withqueryname(first_id,second_id) AS (VALUES(1,2),(3,4),(5,6)...etc)
INSERT INTO jointablename (first_id,second_id)
SELECT * FROM withqueryname
WHERE NOT EXISTS(
SELECT first_id FROM jointablename WHERE
first_id = 1 AND
second_id IN (1,2,3,4,5,6..etc))
You can interchange the Values with a variable. Mine was VALUES#{values}
You can also interchange the second_id IN with a variable. Mine was second_id IN #{variable}.
Here's how I'd tackle it: Create a temp table and populate it with your new values. Then lock the old join values table to prevent concurrent modification (important) and insert all value pairs that appear in the new table but not the old one.
One way to do this is by doing a left outer join of the old values onto the new ones and filtering for rows where the old join table values are null. Another approach is to use an EXISTS subquery. The two are highly likely to result in the same query plan once the query optimiser is done with them anyway.
Example, untested (since you didn't provide an SQLFiddle or sample data) but should work:
BEGIN;
CREATE TEMPORARY TABLE newjoinvalues(
first_id integer,
second_id integer,
primary key(first_id,second_id)
);
-- Now populate `newjoinvalues` with multi-valued inserts or COPY
COPY newjoinvalues(first_id, second_id) FROM stdin;
LOCK TABLE myjoinvalues IN EXCLUSIVE MODE;
INSERT INTO myjoinvalues
SELECT n.first_id, n.second_id
FROM newjoinvalues n
LEFT OUTER JOIN myjoinvalues m ON (n.first_id = m.first_id AND n.second_id = m.second_id)
WHERE m.first_id IS NULL AND m.second_id IS NULL;
COMMIT;
This won't update existing values, but you can do that fairly easily too by using with a second query that does an UPDATE ... FROM while still holding the write table lock.
Note that the lock mode specified above will not block SELECTs, only writes like INSERT, UPDATE and DELETE, so queries can continue to be made to the table while the process is ongoing, you just can't update it.
If you can't accept that an alternative is to run the update in SERIALIZABLE isolation (only works properly for this purpose in Pg 9.1 and above). This will result in the query failing whenever a concurrent write occurs so you have to be prepared to retry it over and over and over again. For that reason it's likely to be better to just live with locking the table for a while.