I have a TSimpleDataSet based on dinamically created SQL query. I need to know which field is a primary key?
SimpleDataSet1.DataSet.SetSchemaInfo(stIndexes, 'myTable' ,'');
This code tells me that i have a primary key with name 'someName', but how can i know which field (column) works with this index?
A Primary Key/Index can belong to several columns (not just one).
The schema stIndexes dataset will return the PK name INDEX_NAME and the columns that construct that PK/Index (COLUMN_NAME). INDEX_TYPE will tell you which index types you have (eSQLNonUnique/eSQLUnique/eSQLPrimaryKey).
I have never worked with TSimpleDataSet but check if the indexes information is stored in
IndexDefs[TIndexDef].Name/Fields/Options - if ixPrimary in Options then this is your PK. and Fields belongs to that index.
Take a look at the source at SqlExpr.pas: TCustomSQLDataSet.AddIndexDefs.
Note how TCustomSQLDataSet returns the TableName (and then the indexs information) from the command text:
...
if FCommandType = ctTable then
TableName := FCommandText
else
TableName := GetTableNameFromSQL(CommandText);
DataSet := FSQLConnection.OpenSchemaTable(stIndexes, TableName, '', '', '');
...
I think the simple data set does not provide that information.
However, i am sure there are components for that. Check, for Oracle database, Devart's ODAC.
Basically, it involves only one query to the database.
However, it is not something that components will offer by default as, because it involves a different query, it leads to slow response times.
For Oracle database, query on user_indexes.
Related
I have an old application I am supporting that uses a Microsoft Access database. The original table design did not add primary keys to every table. I am working on a migration program that among other things is adding and filling in a new primary key field (GUID) when needed.
This is happening in three steps:
Add a new guid field with no constraints
Fill the field with new unique guids
Add the primary key constraints
My problem is setting the unique guids when the table has duplicate rows. Here is my code to set the guids.
Query.SQL.Add('SELECT * FROM ' + TableName);
Query.Open;
while Query.Eof = false do
begin
Query.Edit;
Query.FieldByName(NewPrimaryKeyFieldName).AsGuid := TGuid.NewGuid;
Query.Post;
Query.Next;
end;
FireDac generates an update statement that contains a where clause with all the original fields/values in the row (since there is no unique field for it to use). However, because the rows are complete duplicates the statement still updates two rows.
FireDac correctly errors with this message
Update command updated [2] instead of [1] record.
I can open up the database in Access and delete the duplicate records or assign them a unique guid by editing the table. I would like my conversion tool to automatically do this.
Is there some way to work with these duplicate rows in FireDac? Either to update just one at a time, or to delete just one of them?
In my opinion there is no way to do it with just one SQL Statement.
I would do this:
1. Copy the whole table without duplicates by using a new temp table
SELECT DISTINCT * FROM <TABLENAME>
Add the Keys
Delete old table content and copy new content from new table
Notes:
The DB Should be unavailable for everyone else for that Operation
2. Make BACKUP before
I'm trying to come up with a PostgreSQL schema for host data that's currently in an LDAP store. Part of that data is the list of hostnames a machine can have, and that attribute is generally the key that most people use to find the host records.
One thing I'd like to get out of moving this data to an RDBMS is the ability to set a uniqueness constraint on the hostname column so that duplicate hostnames can't be assigned. This would be easy if hosts could only have one name, but since they can have more than one it's more complicated.
I realize that the fully-normalized way to do this would be to have a hostnames table with a foreign key pointing back to the hosts table, but I'd like to avoid having everybody need to do joins for even the simplest query:
select hostnames.name,hosts.*
from hostnames,hosts
where hostnames.name = 'foobar'
and hostnames.host_id = hosts.id;
I figured using PostgreSQL arrays could work for this, and they certainly make the simple queries simple:
select * from hosts where names #> '{foobar}';
When I set a uniqueness constraint on the hostnames attribute, though, it of course treats the entire list of names as the unique value instead of each name. Is there a way to make each name unique across every row instead?
If not, does anyone know of another data-modeling approach that would make more sense?
The righteous path
You might want to reconsider normalizing your schema. It is not necessary for everyone to "join for even the simplest query". Create a VIEW for that.
Table could look like this:
CREATE TABLE hostname (
hostname_id serial PRIMARY KEY
, host_id int REFERENCES host(host_id) ON UPDATE CASCADE ON DELETE CASCADE
, hostname text UNIQUE
);
The surrogate primary key hostname_id is optional. I prefer to have one. In your case hostname could be the primary key. But many operations are faster with a simple, small integer key. Create a foreign key constraint to link to the table host.
Create a view like this:
CREATE VIEW v_host AS
SELECT h.*
, array_agg(hn.hostname) AS hostnames
-- , string_agg(hn.hostname, ', ') AS hostnames -- text instead of array
FROM host h
JOIN hostname hn USING (host_id)
GROUP BY h.host_id; -- works in v9.1+
Starting with pg 9.1, the primary key in the GROUP BY covers all columns of that table in the SELECT list. The release notes for version 9.1:
Allow non-GROUP BY columns in the query target list when the primary
key is specified in the GROUP BY clause
Queries can use the view like a table. Searching for a hostname will be much faster this way:
SELECT *
FROM host h
JOIN hostname hn USING (host_id)
WHERE hn.hostname = 'foobar';
Provided you have an index on host(host_id), which should be the case as it should be the primary key. Plus, the UNIQUE constraint on hostname(hostname) implements the other needed index automatically.
In Postgres 9.2+ a multicolumn index would be even better if you can get an index-only scan out of it:
CREATE INDEX hn_multi_idx ON hostname (hostname, host_id);
Starting with Postgres 9.3, you could use a MATERIALIZED VIEW, circumstances permitting. Especially if you read much more often than you write to the table.
The dark side (what you actually asked)
If I can't convince you of the righteous path, here is some assistance for the dark side:
Here is a demo how to enforce uniqueness of hostnames. I use a table hostname to collect hostnames and a trigger on the table host to keep it up to date. Unique violations raise an exception and abort the operation.
CREATE TABLE host(hostnames text[]);
CREATE TABLE hostname(hostname text PRIMARY KEY); -- pk enforces uniqueness
Trigger function:
CREATE OR REPLACE FUNCTION trg_host_insupdelbef()
RETURNS trigger
LANGUAGE plpgsql AS
$func$
BEGIN
-- split UPDATE into DELETE & INSERT
IF TG_OP = 'UPDATE' THEN
IF OLD.hostnames IS DISTINCT FROM NEW.hostnames THEN -- keep going
ELSE
RETURN NEW; -- exit, nothing to do
END IF;
END IF;
IF TG_OP IN ('DELETE', 'UPDATE') THEN
DELETE FROM hostname h
USING unnest(OLD.hostnames) d(x)
WHERE h.hostname = d.x;
IF TG_OP = 'DELETE' THEN RETURN OLD; -- exit, we are done
END IF;
END IF;
-- control only reaches here for INSERT or UPDATE (with actual changes)
INSERT INTO hostname(hostname)
SELECT h
FROM unnest(NEW.hostnames) h;
RETURN NEW;
END
$func$;
Trigger:
CREATE TRIGGER host_insupdelbef
BEFORE INSERT OR DELETE OR UPDATE OF hostnames ON host
FOR EACH ROW EXECUTE FUNCTION trg_host_insupdelbef();
SQL Fiddle with test run.
Use a GIN index on the array column host.hostnames and array operators to work with it:
Why isn't my PostgreSQL array index getting used (Rails 4)?
Check if any of a given array of values are present in a Postgres array
In case anyone still needs what was in the original question:
CREATE TABLE testtable(
id serial PRIMARY KEY,
refs integer[],
EXCLUDE USING gist( refs WITH && )
);
INSERT INTO testtable( refs ) VALUES( ARRAY[100,200] );
INSERT INTO testtable( refs ) VALUES( ARRAY[200,300] );
and this would give you:
ERROR: conflicting key value violates exclusion constraint "testtable_refs_excl"
DETAIL: Key (refs)=({200,300}) conflicts with existing key (refs)=({100,200}).
Checked in Postgres 9.5 on Windows.
Note that this would create an index using the operator &&. So when you are working with testtable, it would be times faster to check ARRAY[x] && refs than x = ANY( refs ).
P.S. Generally I agree with the above answer. In 99% cases you'd prefer a normalized schema. Please try to avoid "hacky" stuff in production.
I have a database I would like to convert to use UUID's as the primary key in postgresql.
I have roughly 30 tables with deep multi-level associations. Is there an 'easy' way to convert all current ID's to UUID?
From this: https://coderwall.com/p/n_0awq, I can see that I could alter the table in migration. I was thinking something like this:
for client in Client.all
# Retrieve children
underwritings = client.underwritings
# Change primary key
execute 'ALTER TABLE clients ALTER COLUMN id TYPE uuid;'
execute 'ALTER TABLE clients ALTER COLUMN id SET DEFAULT uuid_generate_v1();'
# Get new id - is this already generated?
client_id = client.id
for underwriting in underwritings
locations = underwriting.locations
other_record = underwriting.other_records...
execute 'ALTER TABLE underwritings ALTER COLUMN id TYPE uuid;'
execute 'ALTER TABLE underwritings ALTER COLUMN id SET DEFAULT uuid_generate_v1();'
underwriting.client_id = client_id
underwriting.saved
underwriting_id = underwriting.id
for location in locations
buildings = location.buildings
execute 'ALTER TABLE locations ALTER COLUMN id TYPE uuid;'
execute 'ALTER TABLE locations ALTER COLUMN id SET DEFAULT uuid_generate_v1();'
location.undewriting_id = underwriting_id
location.save
location_id = location.id
for building in buildings
...
end
end
for other_record in other_records
...
end
...
...
end
end
Questions:
Will this work?
Is there an easier way to do this?
Will child records be retrieved properly as long as they are retrieved before the primary key is changed?
Will the new primary key be already generated as soon as the alter table is called?
Thanks very much for any help or tips in doing this.
I found these to be quite tedious. It is possible to use direct queries to PostgreSQL to convert table with existing data.
For primary key:
ALTER TABLE students
ALTER COLUMN id DROP DEFAULT,
ALTER COLUMN id SET DATA TYPE UUID USING (uuid(lpad(replace(text(id),'-',''), 32, '0'))),
ALTER COLUMN id SET DEFAULT uuid_generate_v4()
For other references:
ALTER TABLE students
ALTER COLUMN city_id SET DATA TYPE UUID USING (uuid(lpad(replace(text(city_id),'-',''), 32, '0')))
The above left pads the integer value with zeros and converts to a UUID. This approach does not require id mapping and if needed old id could be retrieved.
As there is no data copying, this approach works quite fast.
To handle these and more complicated case of polymorphic associations please use https://github.com/kreatio-sw/webdack-uuid_migration. This gem adds additional helpers to ActiveRecord::Migration to ease these migrations.
I think trying to do something like this through Rails would just complicate matters. I'd ignore the Rails side of things completely and just do it in SQL.
Your first step is grab a complete backup of your database. Then restore that backup into another database to:
Make sure that your backup works.
Give you a realistic playpen where you can make mistakes without consequence.
First you'd want to clean up your data by adding real foreign keys to match all your Rails associations. There's a good chance that some of your FKs will fail, if they do you'll have to clean up your broken references.
Now that you have clean data, rename all your tables to make room for the new UUID versions. For a table t, we'll refer to the renamed table as t_tmp. For each t_tmp, create another table to hold the mapping from the old integer ids to the new UUID ids, something like this:
create table t_id_map (
old_id integer not null,
new_id uuid not null default uuid_generate_v1()
)
and then populate it:
insert into t_id_map (old_id)
select id from t_tmp
And you'll probably want to index t_id_map.old_id while you're here.
This gives us the old tables with integer ids and a lookup table for each t_tmp that maps the old id to the new one.
Now create the new tables with UUIDs replacing all the old integer and serial columns that held ids; I'd add real foreign keys at this point as well; you should be paranoid about your data: broken code is temporary, broken data is usually forever.
Populating the new tables is pretty easy at this point: simply use insert into ... select ... from constructs and JOIN to the appropriate t_id_map tables to map the old ids to the new ones. Once the data has been mapped and copied, you'll want to do some sanity checking to make sure everything still makes sense. Then you can drop your t_tmp and t_id_map tables and get on with your life.
Practice that process on a copy of your database, script it up, and away you go.
You would of course want to shut down any applications that access your database while you're doing this work.
Didn't want to add foreign keys, and wanted to to use a rails migration. Anyways, here is what I did if others are looking to do this (example for 2 tables, I did 32 total):
def change
execute 'CREATE EXTENSION "uuid-ossp";'
execute <<-SQL
ALTER TABLE buildings ADD COLUMN guid uuid DEFAULT uuid_generate_v1() NOT NULL;
ALTER TABLE buildings ALTER COLUMN guid SET DEFAULT uuid_generate_v1();
ALTER TABLE buildings ADD COLUMN location_guid uuid;
ALTER TABLE clients ADD COLUMN guid uuid DEFAULT uuid_generate_v1() NOT NULL;
ALTER TABLE clients ALTER COLUMN guid SET DEFAULT uuid_generate_v1();
ALTER TABLE clients ADD COLUMN agency_guid uuid;
ALTER TABLE clients ADD COLUMN account_executive_guid uuid;
ALTER TABLE clients ADD COLUMN account_representative_guid uuid;
SQL
for record in Building.all
location = record.location
record.location_guid = location.guid
record.save
end
for record in Client.all
agency = record.agency
record.agency_guid = agency.guid
account_executive = record.account_executive
record.account_executive_guid = account_executive.guid unless account_executive.blank?
account_representative = record.account_representative
record.account_representative_guid = account_representative.guid unless account_representative.blank?
record.save
end
execute <<-SQL
ALTER TABLE buildings DROP CONSTRAINT buildings_pkey;
ALTER TABLE buildings DROP COLUMN id;
ALTER TABLE buildings RENAME COLUMN guid TO id;
ALTER TABLE buildings ADD PRIMARY KEY (id);
ALTER TABLE buildings DROP COLUMN location_id;
ALTER TABLE buildings RENAME COLUMN location_guid TO location_id;
ALTER TABLE clients DROP CONSTRAINT clients_pkey;
ALTER TABLE clients DROP COLUMN id;
ALTER TABLE clients RENAME COLUMN guid TO id;
ALTER TABLE clients ADD PRIMARY KEY (id);
ALTER TABLE clients DROP COLUMN agency_id;
ALTER TABLE clients RENAME COLUMN agency_guid TO agency_id;
ALTER TABLE clients DROP COLUMN account_executive_id;
ALTER TABLE clients RENAME COLUMN account_executive_guid TO account_executive_id;
ALTER TABLE clients DROP COLUMN account_representative_id;
ALTER TABLE clients RENAME COLUMN account_representative_guid TO account_representative_id;
SQL
end
I am trying to teach myself SQL. I have a web matrix project I am working on to edit and display posts backed by a SQL server Datatabase. A work colleague suggested I use a Stored Procedure to commit the post rather than writing the sql inline.
So far the procedure looks ok but I would like to check if the url slug already exists, and if so return something to say so (The url slug should be unique). I'm struggling with how I am supposed to check before the insert. I have also read that it is bad practice to return from a stored procedure, but I thought it would be a good idea to return something to let the caller know the insert did not go ahead.
Any help would be very much appreciated.
-- =============================================
-- Author: Dean McDonnell
-- Create date: 05/12/2011
-- Description: Commits an article to the database.
-- =============================================
CREATE PROCEDURE CommitPost
#UrlSlug VARCHAR(100),
#Heading VARCHAR(100),
#SubHeading VARCHAR(300),
#Body VARCHAR(MAX)
AS
INSERT INTO Posts(UrlSlug, Heading, SubHeading, Body, Timestamp)
VALUES(#UrlSlug, #Heading, #SubHeading, #Body, GETDATE())
This is what I have so far.
CREATE PROCEDURE CommitPost
#UrlSlug VARCHAR(100),
#Heading VARCHAR(100),
#SubHeading VARCHAR(300),
#Body VARCHAR(MAX)
AS
IF NOT EXISTS (SELECT * FROM Posts WHERE UrlSlug = #UrlSlug)
INSERT INTO Posts(UrlSlug, Heading, SubHeading, Body, Timestamp)
VALUES(#UrlSlug, #Heading, #SubHeading, #Body, GETDATE())
SELECT ##ROWCOUNT
To check for existance, do a SELECT COUNT like so:
CREATE PROCEDURE CommitPost
#UrlSlug VARCHAR(100),
#Heading VARCHAR(100),
#SubHeading VARCHAR(300),
#Body VARCHAR(MAX)
AS
DECLARE #count INT
SELECT #count = COUNT(*) FROM Posts WHERE UrlSlug = #UrlSlug
IF #count = 0 THEN
BEGIN
INSERT INTO Posts(UrlSlug, Heading, SubHeading, Body, Timestamp)
VALUES(#UrlSlug, #Heading, #SubHeading, #Body, GETDATE())
END
You may set an unique index on UrlSlug to make the database reject insertions of urls already in the database, but nonetheless you should check before inserting.
If your caller wants to know if the row was inserted, return the #count value. If it's 0 then the line was inserted, else not. I'm not aware of a "bad practice" regarding to return values from a SP. As a SP does not have a result, though, you need to use an out parameter.
If you do just one SQL statement like this insert you could just use paratemerized query i.e. I assume that you are using .NET.
If you want to return values I would suggest that you use a FUNCTION instead of a STORED PROCEDURE. You can return either tables or whatever you want from a function.
There are some limitations though. You can dig a little deeper into the differences to see what is used when. Here's a link that can help you out get started:
Function vs. Stored Procedure in SQL Server
If you want to use stored procedure anyway, you can either return a single row, single column result set, using SELECT, or just use an output parameter.
If you want to do actions depending of whether the column exists or not I would suggest that you look into MERGE statement.That way you would perform only one query to the database instead of two or more(doing SELECT and then INSERT).
There are also other ways to use database access like various ORMs on top of the database in the code that will make your life easier, like LINQ-to-SQL etc. There are a lot of possibilities out there. You need to determine what's best in a given situation.
I have a select query that joins 3 tables to display information. If I want to update a field is do
ADOQuery.Edit.
<Set fields Values>
ADOQuery.Post.
I get the following error howwever 'Insufficient key column information for updating or refresing' and I'm not sure how to proceed.
Thank you.
Pieter
Try this OnCreate:
YourADODataSet.Properties['Unique Table'].Value := 'YourTable';
This will let the Engine send only one (instead of two) query from joined tables.