How to migrate complex Rails database to use UUID primary keys Postgresql - ruby-on-rails

I have a database I would like to convert to use UUID's as the primary key in postgresql.
I have roughly 30 tables with deep multi-level associations. Is there an 'easy' way to convert all current ID's to UUID?
From this: https://coderwall.com/p/n_0awq, I can see that I could alter the table in migration. I was thinking something like this:
for client in Client.all
# Retrieve children
underwritings = client.underwritings
# Change primary key
execute 'ALTER TABLE clients ALTER COLUMN id TYPE uuid;'
execute 'ALTER TABLE clients ALTER COLUMN id SET DEFAULT uuid_generate_v1();'
# Get new id - is this already generated?
client_id = client.id
for underwriting in underwritings
locations = underwriting.locations
other_record = underwriting.other_records...
execute 'ALTER TABLE underwritings ALTER COLUMN id TYPE uuid;'
execute 'ALTER TABLE underwritings ALTER COLUMN id SET DEFAULT uuid_generate_v1();'
underwriting.client_id = client_id
underwriting.saved
underwriting_id = underwriting.id
for location in locations
buildings = location.buildings
execute 'ALTER TABLE locations ALTER COLUMN id TYPE uuid;'
execute 'ALTER TABLE locations ALTER COLUMN id SET DEFAULT uuid_generate_v1();'
location.undewriting_id = underwriting_id
location.save
location_id = location.id
for building in buildings
...
end
end
for other_record in other_records
...
end
...
...
end
end
Questions:
Will this work?
Is there an easier way to do this?
Will child records be retrieved properly as long as they are retrieved before the primary key is changed?
Will the new primary key be already generated as soon as the alter table is called?
Thanks very much for any help or tips in doing this.

I found these to be quite tedious. It is possible to use direct queries to PostgreSQL to convert table with existing data.
For primary key:
ALTER TABLE students
ALTER COLUMN id DROP DEFAULT,
ALTER COLUMN id SET DATA TYPE UUID USING (uuid(lpad(replace(text(id),'-',''), 32, '0'))),
ALTER COLUMN id SET DEFAULT uuid_generate_v4()
For other references:
ALTER TABLE students
ALTER COLUMN city_id SET DATA TYPE UUID USING (uuid(lpad(replace(text(city_id),'-',''), 32, '0')))
The above left pads the integer value with zeros and converts to a UUID. This approach does not require id mapping and if needed old id could be retrieved.
As there is no data copying, this approach works quite fast.
To handle these and more complicated case of polymorphic associations please use https://github.com/kreatio-sw/webdack-uuid_migration. This gem adds additional helpers to ActiveRecord::Migration to ease these migrations.

I think trying to do something like this through Rails would just complicate matters. I'd ignore the Rails side of things completely and just do it in SQL.
Your first step is grab a complete backup of your database. Then restore that backup into another database to:
Make sure that your backup works.
Give you a realistic playpen where you can make mistakes without consequence.
First you'd want to clean up your data by adding real foreign keys to match all your Rails associations. There's a good chance that some of your FKs will fail, if they do you'll have to clean up your broken references.
Now that you have clean data, rename all your tables to make room for the new UUID versions. For a table t, we'll refer to the renamed table as t_tmp. For each t_tmp, create another table to hold the mapping from the old integer ids to the new UUID ids, something like this:
create table t_id_map (
old_id integer not null,
new_id uuid not null default uuid_generate_v1()
)
and then populate it:
insert into t_id_map (old_id)
select id from t_tmp
And you'll probably want to index t_id_map.old_id while you're here.
This gives us the old tables with integer ids and a lookup table for each t_tmp that maps the old id to the new one.
Now create the new tables with UUIDs replacing all the old integer and serial columns that held ids; I'd add real foreign keys at this point as well; you should be paranoid about your data: broken code is temporary, broken data is usually forever.
Populating the new tables is pretty easy at this point: simply use insert into ... select ... from constructs and JOIN to the appropriate t_id_map tables to map the old ids to the new ones. Once the data has been mapped and copied, you'll want to do some sanity checking to make sure everything still makes sense. Then you can drop your t_tmp and t_id_map tables and get on with your life.
Practice that process on a copy of your database, script it up, and away you go.
You would of course want to shut down any applications that access your database while you're doing this work.

Didn't want to add foreign keys, and wanted to to use a rails migration. Anyways, here is what I did if others are looking to do this (example for 2 tables, I did 32 total):
def change
execute 'CREATE EXTENSION "uuid-ossp";'
execute <<-SQL
ALTER TABLE buildings ADD COLUMN guid uuid DEFAULT uuid_generate_v1() NOT NULL;
ALTER TABLE buildings ALTER COLUMN guid SET DEFAULT uuid_generate_v1();
ALTER TABLE buildings ADD COLUMN location_guid uuid;
ALTER TABLE clients ADD COLUMN guid uuid DEFAULT uuid_generate_v1() NOT NULL;
ALTER TABLE clients ALTER COLUMN guid SET DEFAULT uuid_generate_v1();
ALTER TABLE clients ADD COLUMN agency_guid uuid;
ALTER TABLE clients ADD COLUMN account_executive_guid uuid;
ALTER TABLE clients ADD COLUMN account_representative_guid uuid;
SQL
for record in Building.all
location = record.location
record.location_guid = location.guid
record.save
end
for record in Client.all
agency = record.agency
record.agency_guid = agency.guid
account_executive = record.account_executive
record.account_executive_guid = account_executive.guid unless account_executive.blank?
account_representative = record.account_representative
record.account_representative_guid = account_representative.guid unless account_representative.blank?
record.save
end
execute <<-SQL
ALTER TABLE buildings DROP CONSTRAINT buildings_pkey;
ALTER TABLE buildings DROP COLUMN id;
ALTER TABLE buildings RENAME COLUMN guid TO id;
ALTER TABLE buildings ADD PRIMARY KEY (id);
ALTER TABLE buildings DROP COLUMN location_id;
ALTER TABLE buildings RENAME COLUMN location_guid TO location_id;
ALTER TABLE clients DROP CONSTRAINT clients_pkey;
ALTER TABLE clients DROP COLUMN id;
ALTER TABLE clients RENAME COLUMN guid TO id;
ALTER TABLE clients ADD PRIMARY KEY (id);
ALTER TABLE clients DROP COLUMN agency_id;
ALTER TABLE clients RENAME COLUMN agency_guid TO agency_id;
ALTER TABLE clients DROP COLUMN account_executive_id;
ALTER TABLE clients RENAME COLUMN account_executive_guid TO account_executive_id;
ALTER TABLE clients DROP COLUMN account_representative_id;
ALTER TABLE clients RENAME COLUMN account_representative_guid TO account_representative_id;
SQL
end

Related

Sequelize : how to join on foreign keys

I'm using Sequelize and Node.js, and I need to join 2 tables on 2 foreign keys,
tableA.hasOne(tableB, { foreignKey: 'fk_tableB' });
tableB.belongsTo(tableA, {foreignKey: 'fk_tableB' });
(by the way, I don't understand the functionality of "targetKey")
but I can only obtain a join on tableA.primaryKey = tableB.fk_tableB.
How can I replace the tableA.primaryKey by tableA.fk_tableA?
I also tried to define twice tableA : in 2 different structures (one with the real primary key and the other with fk_tableA as primary key), but it's also not working (because I need the real tableA mode in another place).
Has someone an idea? Is it a bug from Sequelize?
How can I replace the tableA.primaryKey by tableA.fk_tableA?
There is no tableA.fk_tableA. But if there were, we would expect you to have named it that because column tableA.fk_tableA is a FK to a key column in tableA. Because that's the convention for naming a column fk_tableA. Similarly we would expect a belongTo like yours that adds a column that is a FK to the tableA PK to call it fk_tableA, not fk_tableB. Just like your hasOne gives tableA a column fk_tableB to the tableB PK. (If you want a FK to be to some other column than the PK then you say so via targetKey.)
If you so named FKs after their target table, you seem to want tableA.fk_tableB = tableB.fk_tableA. The way you have named them now, you seem to want tableA.fk_tableB = tableB.fk_tableB.
I need to join 2 tables on 2 foreign keys
It is extremely unlikely that you need the join above. Declaring a column to be a FK says that a value of the source/referencing column is always a value of the target/referenced column. Here targets are PKs. Such a join on a FK to one table and a FK to another table will only return rows that share the same PK value, even though the PKs are from different tables.

Regenerate ids of existing records

I have a Rails app with a custom algorithm for id generation for one table. Now I decided to use default incremental ids generated by Postgres as primary keys and move my custom values from id to another column like uid. And I want to regenerate values in id column for all records - as normal from 1 to n.
What is the best way to do this? I have about 1000 records. Records from other tables are associated with these records.
You can keep whatever value is in ID column but create a new column named UID and set it as a primary key and auto increment
def self.up
execute "ALTER TABLE mytable modify COLUMN uid int(8) AUTO_INCREMENT"
end
You can tell your model to use UID as primary key as
self.primary_key = 'uid'
You can simply do it by iterating on your records, and updating them (and their associated objects). 1000 records is not that much to process.
Let's say that you have a table named "my_objects", with its model named "MyObject". Let's also say that you have another table, named "related_objects" and its model "RelatedObject"
You can iterate on all your MyObjects records, and update their related objects and the record itself at the same time.
records = MyObject.all #Whatever "MyObject" you have.
i = 0
records.each do |record|
#Updating whatever associated objects the record have
record.related_objects.each do |related_object|
related_object.update_column("my_object_id", i)
end
#Updating the record itself
record.update_column("id", i)
i++
end

How to change the sql column data type in iOS?

In my iOS app,I want to change the column data type in database.
ALTER TABLE XXX ALTER COLUMN myColumn INT.
I always get 'near ALTER Syntax error'
How to resolve the problem?
Thanks for your help.
You cannot change the column type. You can create a new table, using the correct data type for the column this time, and then select data from the old table and insert it into the new table. The full procedure is outlined in the ALTER TABLE documentation:
Remember the format of all indexes and triggers associated with table X. This information will be needed in step 7 below. One way to do this is to run a query like the following: SELECT type, sql FROM sqlite_master WHERE tbl_name='X'.
Use CREATE TABLE to construct a new table "new_X" that is in the desired revised format of table X. Make sure that the name "new_X" does not collide with any existing table name, of course.
Transfer content from X into new_X using a statement like: INSERT INTO new_X SELECT ... FROM X.
If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF.
Drop the old table X: DROP TABLE X.
Change the name of new_X to X using: ALTER TABLE new_X RENAME TO X.
Use CREATE INDEX and CREATE TRIGGER to reconstruct indexes and triggers associated with table X. Perhaps use the old format of the triggers and indexes saved from step 1 above as a guide, making changes as appropriate for the alteration.
If foreign key constraints were originally enabled (prior to step 4) then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints, and run PRAGMA foreign_keys=ON to re-enable foreign key constraints.
If any views refer to table X in a way that is affected by the schema change, then drop those views using DROP VIEW and recreate them with whatever changes are necessary to accommodate the schema change using CREATE VIEW.
Note, SQLite uses type affinity (the column definition doesn't alter what type of data you insert into the table). So if you change the data type, you'll want to change the data, too.
ALTER TABLE table ADD newColumn INTEGER;
UPDATE table SET newColumn = oldColumn;

DROP Syntax error in SQLiteManager

I have downloaded sqlite manager app in Firefox browser and open sqlite manager app.
1) Created database named DBSQLTEST.
2) Created table named SQLTEST contain has threes fields are SLNO, NAME and AGE
3) Inserted new records
But i want remove a ‘AGE’ column in sqltest table
i using sql command like below
ALTER TABLE SQLTEST DROP COLUMN AGE
SQLiteManager message says
SQLiteManager: Likely SQL syntax error: ALTER TABLE SQLTEST DROP COLUMN AGE [ near "DROP": syntax error ]
Exception Name: NS_ERROR_FAILURE
Exception Message: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [mozIStorageConnection.createStatement]
What is error that?
sqlite does not support DROP COLUMN in ALTER TABLE. You can only rename tables and add columns.
Reference: http://www.sqlite.org/lang_altertable.html
If you need to remove columns, create a new table, copy the data there, drop the old table and rename the table to its intented name.
SQLite does not fully support ALTER TABLE statements. You can only rename table, or add columns.
If you want to drop a column, your best option is to create a new table without the column, and to drop the old table in order to rename the new one.
For example, suppose you have a table named "t1" with columns names "a", "b", and "c" and that you want to delete column "c" from this table. The following steps illustrate how this could be done:
BEGIN TRANSACTION;
CREATE TEMPORARY TABLE t1_backup(a,b);
INSERT INTO t1_backup SELECT a,b FROM t1;
DROP TABLE t1;
CREATE TABLE t1(a,b);
INSERT INTO t1 SELECT a,b FROM t1_backup;
DROP TABLE t1_backup;
COMMIT;
Credits to Benoit.
Create a table
CREATE TABLE SQLTEST ("SLNO" INTEGER PRIMARY KEY ,"NAME" TEXT,AGE INTEGER)
If want to remove a column
ALTER TABLE SQLTEST RENAME TO SQLTESTCOPY
CREATE TABLE SQLTEST ("SLNO" INTEGER PRIMARY KEY ,"NAME" TEXT)
INSERT INTO SQLTEST"SELECT "SLNO","NAME" FROM SQLTESTCOPY
DROP TABLE SQLTESTCOPY
easy.

Create missing auto increment attribute with rails migration

I'm writing a migration to convert a non-rails app into the right format for rails - one of the tables for some reason does not have auto increment set on the id column. Is there a quick way to turn it on while in a migration, maybe with change_column or something?
You need to execute an SQL statement.
statement = "ALTER TABLE `users` CHANGE `id` `id` SMALLINT( 5 ) UNSIGNED NOT NULL AUTO_INCREMENT"
ActiveRecord::Base.connection.execute(statement)
Note this is just an example. The final SQL statement syntax depends on the database.
If you're on postgesql, a single request won't make it. You'll need to create a new sequence in the database.
create sequence users_id_seq;
Then add the id column to your table
alter table users
add id INT UNIQUE;
Then set the default value for the id
alter table users
alter column id
set default nextval('users_id_seq');
Then populate the id column. This may be quite long if the table has many rows
update users
set id = nextval('users_id_seq');
Hope this helps postgresql users...
The Postgres answer by #jlfenaux misses out on the serial type, which does all of it for you automatically:
ALTER TABLE tbl add tbl_id serial;
More details in this related answer.

Resources