Case-insensitive unique index in Rails/ActiveRecord? - ruby-on-rails

I need to create a case-insensitive index on a column in rails. I did this via SQL:
execute(
"CREATE UNIQUE INDEX index_users_on_lower_email_index
ON users (lower(email))"
)
This works great, but in my schema.rb file I have:
add_index "users", [nil],
:name => "index_users_on_lower_email_index",
:unique => true
Notice the "nil". So when I try to clone the database to run a test, I get an obvious error. Am I doing something wrong here? Is there some other convention that I should be using inside rails?
Thanks for the help.

Since MySQL indexes are already case-insensitive, I'm guessing you're dealing with PostgreSQL, which creates case-sensitive indexes by default. I'm answering here based on Rails 3.2.3 and PostgreSQL 8.4.
It seems functional indexes are one more example of things that ActiveRecord can't generate. Foreign keys and UUID columns are two more that come to mind. So there is no choice (other than monkey-patching ActiveRecord) but to use execute statements.
This means for an accurate dump of your database, you'll need to abandon the DB-agnostic schema.rb in favor of DB-specific structure.sql. See the Rails Guide on Migrations, section 6.2 Types of Schema Dumps. This is set as follows:
config/application.rb
config.active_record.schema_format = :sql
db/structure.sql should be updated automatically when you run a migration. You can generate it manually with this command:
rake db:structure:dump
The file is pure Postgres SQL. Although not documented when you use rake -T to list rake tasks, it seems that you can use this command to load the database from the structure.sql dump:
rake db:structure:load
There's nothing magic here: the source code (shown here from Rails 3.2.16) just calls psql on structure.sql.
Finally, here is my migration to drop an old, case-sensitive email constraint and add the case-sensitive functional index:
class FixEmailUniqueIndexOnUsers < ActiveRecord::Migration
def up
remove_index :users, :email
execute "CREATE UNIQUE INDEX index_users_on_lowercase_email
ON users USING btree (lower(email));"
end
def down
execute "DROP INDEX index_users_on_lowercase_email;"
add_index :users, :email, :unique => true
end
end

If you are using PostgreSQL you can change your column type to citext - case-insensitive string. It also makes search independent from the register.
def change
enable_extension :citext
change_column :users, :email, :citext
add_index :users, :email, unique: true
end

I would simplify this...
In your model:
before_validation :downcase_email
def downcase_email
self.email = email.downcase
end
That way, the index is database agnostic, and your emails are all lowercase in the database.

Have you considered using schema_plus (https://github.com/lomba/schema_plus)? Among other things (support for enforcing foreign keys in the database and for views), it supports setting case-insensitive indexes for PostgreSQL databases and handles dumping them in the schema. From the Readme, "If you’re using Postgresql, SchemaPlus provides support for conditions, expressions, index methods, and case-insensitive indexes."

The documentation is unclear on how to do this but the source looks like this:
def add_index(table_name, column_name, options = {})
index_name, index_type, index_columns = add_index_options(table_name, column_name, options)
execute "CREATE #{index_type} INDEX #{quote_column_name(index_name)} ON #{quote_table_name(table_name)} (#{index_columns})"
end
So, if your database's quote_column_name is the default implementation (which does nothing at all), then this might work:
add_index "users", ['lower(email)'], :name => "index_users_on_lower_email_index", :unique => true
You note that you tried that one but it didn't work (adding that to your question might be a good idea). Looks like ActiveRecord simply doesn't understand indexes on a computed value. I can think of an ugly hack that will get it done but it is ugly:
Add an email_lc column.
Add a before_validation or before_save hook to put a lower case version of email into email_lc.
Put your unique index on email_lc.
That's pretty ugly and you might feel dirty for doing it but that's the best I can think of right now.

I would suggest (just as an opportunity to consider among others) to use two separate fields:
email
email_original
The first one is always down cased and thus can be uniquely indexed in a database agnostic manner, while the second field keeps verbatim to what user enetered and can have upper characters.
Obviously in User model one then needs to set email based on email_original on each save and prohibit direct mangling with email field.

For Rails 4.2, create case-insensitive unique index in users table on name column.
Create new migration file with empty change method:
$ rails generate migration add_index_in_users_on_name
Add call of add_index method to empty change method:
add_index :users, 'lower(name)', name: 'index_users_on_lower_name', unique: true
Run Rake db:migrate task:
$ rake db:migrate
In result, index will be added correctly and file db/schema.rb contains correct add_index:
add_index "users", ["LOWER(\"NAME\")"], name: "index_users_on_lower_name", unique: true
This tested only with RDB Oracle.

I think you need column name(s) as below
add_index "users", [email], {:name => "index_users_on_lower_email_index", :unique => true }
And you have to make email field in database with proper Case Insensitive collation, that way your index will be also case insensitive.
depending of db engine which you are using it syntax may be different but
alter table [users] alter column [email] varchar(250) collate utf8_general_ci ...
and when you add index to this column it will be case insensitive.

Related

Active Record adding plain sql index fails with case sensitivity [duplicate]

I need to create a case-insensitive index on a column in rails. I did this via SQL:
execute(
"CREATE UNIQUE INDEX index_users_on_lower_email_index
ON users (lower(email))"
)
This works great, but in my schema.rb file I have:
add_index "users", [nil],
:name => "index_users_on_lower_email_index",
:unique => true
Notice the "nil". So when I try to clone the database to run a test, I get an obvious error. Am I doing something wrong here? Is there some other convention that I should be using inside rails?
Thanks for the help.
Since MySQL indexes are already case-insensitive, I'm guessing you're dealing with PostgreSQL, which creates case-sensitive indexes by default. I'm answering here based on Rails 3.2.3 and PostgreSQL 8.4.
It seems functional indexes are one more example of things that ActiveRecord can't generate. Foreign keys and UUID columns are two more that come to mind. So there is no choice (other than monkey-patching ActiveRecord) but to use execute statements.
This means for an accurate dump of your database, you'll need to abandon the DB-agnostic schema.rb in favor of DB-specific structure.sql. See the Rails Guide on Migrations, section 6.2 Types of Schema Dumps. This is set as follows:
config/application.rb
config.active_record.schema_format = :sql
db/structure.sql should be updated automatically when you run a migration. You can generate it manually with this command:
rake db:structure:dump
The file is pure Postgres SQL. Although not documented when you use rake -T to list rake tasks, it seems that you can use this command to load the database from the structure.sql dump:
rake db:structure:load
There's nothing magic here: the source code (shown here from Rails 3.2.16) just calls psql on structure.sql.
Finally, here is my migration to drop an old, case-sensitive email constraint and add the case-sensitive functional index:
class FixEmailUniqueIndexOnUsers < ActiveRecord::Migration
def up
remove_index :users, :email
execute "CREATE UNIQUE INDEX index_users_on_lowercase_email
ON users USING btree (lower(email));"
end
def down
execute "DROP INDEX index_users_on_lowercase_email;"
add_index :users, :email, :unique => true
end
end
If you are using PostgreSQL you can change your column type to citext - case-insensitive string. It also makes search independent from the register.
def change
enable_extension :citext
change_column :users, :email, :citext
add_index :users, :email, unique: true
end
I would simplify this...
In your model:
before_validation :downcase_email
def downcase_email
self.email = email.downcase
end
That way, the index is database agnostic, and your emails are all lowercase in the database.
Have you considered using schema_plus (https://github.com/lomba/schema_plus)? Among other things (support for enforcing foreign keys in the database and for views), it supports setting case-insensitive indexes for PostgreSQL databases and handles dumping them in the schema. From the Readme, "If you’re using Postgresql, SchemaPlus provides support for conditions, expressions, index methods, and case-insensitive indexes."
The documentation is unclear on how to do this but the source looks like this:
def add_index(table_name, column_name, options = {})
index_name, index_type, index_columns = add_index_options(table_name, column_name, options)
execute "CREATE #{index_type} INDEX #{quote_column_name(index_name)} ON #{quote_table_name(table_name)} (#{index_columns})"
end
So, if your database's quote_column_name is the default implementation (which does nothing at all), then this might work:
add_index "users", ['lower(email)'], :name => "index_users_on_lower_email_index", :unique => true
You note that you tried that one but it didn't work (adding that to your question might be a good idea). Looks like ActiveRecord simply doesn't understand indexes on a computed value. I can think of an ugly hack that will get it done but it is ugly:
Add an email_lc column.
Add a before_validation or before_save hook to put a lower case version of email into email_lc.
Put your unique index on email_lc.
That's pretty ugly and you might feel dirty for doing it but that's the best I can think of right now.
I would suggest (just as an opportunity to consider among others) to use two separate fields:
email
email_original
The first one is always down cased and thus can be uniquely indexed in a database agnostic manner, while the second field keeps verbatim to what user enetered and can have upper characters.
Obviously in User model one then needs to set email based on email_original on each save and prohibit direct mangling with email field.
For Rails 4.2, create case-insensitive unique index in users table on name column.
Create new migration file with empty change method:
$ rails generate migration add_index_in_users_on_name
Add call of add_index method to empty change method:
add_index :users, 'lower(name)', name: 'index_users_on_lower_name', unique: true
Run Rake db:migrate task:
$ rake db:migrate
In result, index will be added correctly and file db/schema.rb contains correct add_index:
add_index "users", ["LOWER(\"NAME\")"], name: "index_users_on_lower_name", unique: true
This tested only with RDB Oracle.
I think you need column name(s) as below
add_index "users", [email], {:name => "index_users_on_lower_email_index", :unique => true }
And you have to make email field in database with proper Case Insensitive collation, that way your index will be also case insensitive.
depending of db engine which you are using it syntax may be different but
alter table [users] alter column [email] varchar(250) collate utf8_general_ci ...
and when you add index to this column it will be case insensitive.

Can I add comments to a table or column using ActiveRecord Migrations?

In MySQL (and other SQL databases), it can be helpful to add comments to a table or column whose purpose may be unclear. (Search MySQL's create table syntax for "comment" for examples.)
Is there a way to do this in an ActiveRecord Migration? I have tried this with no results.
create_table :stuff do |t|
t.integer :obscure_column, :comment => "Explanatory comment"
end
I'm using Rails 3.1.
The migration_comments gem mentioned in a comment to the original question appears to be the best cross-database solution for this need. In addition to providing migrations support for adding table and column comments, the gem also annotates the schema.rb file to include all of the comments. Perfect for my company's needs (large legacy rails app where the database model is ambiguous and also shared with a team of analysts writing native SQL reports).
In Rails 5 you can use the change_column:
class AddCommentsToReferences < ActiveRecord::Migration[5.2]
def up
change_column :references, :achievement_id, :integer, comment: 'Achievement'
change_column :references, :object_id, :integer, comment: 'Achievement object id'
end
end
don't forget write correct column_type as third parameter.
There is a gem called pg_comment that will add this functionality if you are using postgresql.
The gem adds extra commands to add the comments. Note that the syntax in postgresql is different than in mysql, and I guess that is why there is no general ActiveRecord implementation.
For example:
create_table :stuff do |t|
t.integer :some_value
end
set_table_comment :stuff, 'This table stores stuff.'
set_column_comment :stuff, :some_value, 'Stores some value'
This could get pretty verbose, but I know there are some nice tools that make use of this.
Secondly, Rails indeed allows you to manage your schema from within rails (and that is awesome), it may seem sufficient to document your migrations, but after a while nobody looks at the migrations anymore. And you are stuck with an undocumented schema.
In the oracle-enhanced adapter this feature is available from the start, and has just the same syntax as you proposed.
Unfortunately I have not found a similar gem or solution for MySQL.
I don't know when this method is introduced, but in the latest rails (6.0) you can use change_column_comment method.
def change
change_column_comment(:posts, :state, from: "old_comment", to: "new_comment")
end
refer: https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/SchemaStatements.html#method-i-change_column_comment

Can I remove or alter a PostgreSQL table's geometry dimension constraint via a Rails migration?

The question
I am using Rails 2.3 with the Spatial Adapter plugin and a Postgres 8.4 database with PostGIS extension.
A while ago, a geometry field has been added to a table via a migration like this:
add_column :my_table, :the_geom, :geometry, :srid => 4326
Along with the field, a table dimension constraint is created. In this case it makes sure the geometry entered into the the_geom field does have exactly two dimensions.
Now, I'd like to add geometries with more than 2 dimensions. What Rails migration code could I use to either change the constraint to allow more than two dimensions or remove it altogether?
What didn't work
When I create a new geometry field via the add_column method, I know I can use the :with_z and :with_m options to increase the number of dimensions allowed (the constraint would allow up to four dimension when using both options).
However, when using these options with the change_column command, it doesn't seem to have any effect on the constraint.
I don't want to remove the column and re-add it, because then I lose all data.
I know, I can use raw SQL to remove the constraint (I could also use a migration or rake task as a wrapper). The problem with this is, that it doesn't have an effect on the test database, because it is always created from scratch when I run all tests.
I'm not familiar with the spatial adapter fields/options so this code might be a little off, but how about something like this?
class UpgradeGeometryMigration < ActiveRecord::Migration
def self.up
rename_column :my_table, :the_geom, :the_geom_old
add_column :my_table, :the_geom, :geometry, :srid => 4326, :with_z => true, :with_m => true
# Add some code here to iterate through all of your records in the
# :the_geom_old table and copy them into the new :the_geom field
remove_column :my_table, :the_geom_old
# Add any indices required for :the_geom
end
end
There is no way to alter a geometry column without turning to raw SQL via ActiveRecord::Base.connection.execute.
Unfortunately raw SQL statements do not make it into the Rails database schema and so the constraints remain in the DB when it is recreated (for tests for example).
Some raw SQL statements end up in the database schema. The following modifies the column type from geometry to geography:
class ChangePostGisGeometryToGeography < ActiveRecord::Migration
def up
execute <<-SQL
ALTER TABLE mytable ALTER COLUMN geom TYPE geography(MULTIPOLYGON,4326);
SQL
end
end
The migration modifies db/schema.rb correctly :
- t.spatial "geom", :limit => {:srid=>4326, :type=>"multi_polygon"}
+ t.spatial "geom", :limit => {:srid=>4326, :type=>"multi_polygon", :geographic=>true}
(Tested under Rails 3.2.13 and activerecord-postgis-adapter 0.6.6)

Can Rails Migrations be used to convert data?

I'm trying to convert a column in my Rails app, for arguments sake let's pretend I'm trying to change the age column in my users table to a string representation rather than an int.
In my migration I have this;
def.self up
add_column :users, :age_text, :string
users = User.find(:all)
users.each do |u|
u.age_text = convert_to_text(u.age)
u.save
end
end
def self.convert_to_text(number)
#code here to convert 1 to 'one' etc
end
But it doesn't seem to be working, is what I'm attempting here even possible with migrations?
What you're trying to do is possible, and I would say the correct thing to do.
You need, though, to reload the column info for the model classes you're updating in the migration, so that Rails knows about the new columns. Try this:
def.self up
add_column :users, :age_text, :string
User.reset_column_information
users = User.find(:all)
users.each do |u|
u.age_text = convert_to_text(u.age)
u.save
end
end
On a separate note, please note that if your table is large, doing updates one by one will take a looong time.. Be careful with that.
Since I'm new here I can't comment on the above so I'll add my own answer.
GENERALLY manipulating data in migrations is a BAD idea. Migrations with direct model access can get stuck if the model logic changes.
Imagine in your second migration you've added a new column. You want to seed that column with new data.
Let's also say a few weeks later you add a new validation to the model - a validation that operates on a field that does not yet exist in your second migration. if you ever were to construct the database from migration 0, you'd have some problems.
I strongly suggest using migrations to alter columns and other means to manage database data, especially when moving to production.
Here is an example migration I ran to convert data. You can easily convert it to use integers instead of strings. Making the conversion in SQL is much faster than loading each row in Rails.
class ConvertCommentTextToText < ActiveRecord::Migration
def up
add_column :comments, :text_tmp, :text
# copy text column
execute <<-SQL
update comments set text_tmp = text
SQL
remove_column :comments, :text
rename_column :comments, :text_tmp, :text
end
def down
add_column :comments, :text_tmp, :string
# copy text column
execute <<-SQL
update comments set text_tmp = text
SQL
remove_column :comments, :text
rename_column :comments, :text_tmp, :text
end
end
And to test it:
rake db:migrate
rake db:rollback
rake db:migrate
I would say that if you can "undo" the imported data when rolling back the migration version, then it's appropriate to put imports into the migration.
For example, I have a migration which sets up a lot of lookup tables and other meta-data. The data for these tables are populated during this phase. As the data for these lookup tables changes, I create new YAML files storing the meta-data and load those files in subsequent migrations (and un-do those YAMLS, re-loading the previous YAML file when backing out of a migration version). This is pretty clean. I have files (in different well-defined folders in my case) with these files:
002_setup_meta_data.rb
002_meta_data.yaml
007_change_meta_data.rb
007_meta_data.yaml
If you're importing "production" data from another system into transactional (non-static) tables, then I would say using migrations is not appropriate. Then I would follow Brian Hogan's advice of using rake tasks.

Use of Migrations in Ruby on Rails

I would like to confirm that the following analysis is correct:
I am building a web app in RoR. I have a data structure for my postgres db designed (around 70 tables; this design may need changes and additions during development to reflect Rails ways of doing things. EG, I designed some user and role tables - but if it makes sense to use Restful Authentication, I will scrub them and replace with whatever RA requires. ).
I have a shellscript which calls a series of .sql files to populate the empty database with tables and initial data (eg, Towns gets pre-filled with post towns) as well as test data (eg, Companies gets a few dummy companies so I have data to play with).
for example:
CREATE TABLE towns (
id integer PRIMARY KEY DEFAULT nextval ('towns_seq'),
county_id integer REFERENCES counties ON DELETE RESTRICT ON UPDATE CASCADE,
country_id integer REFERENCES countries ON DELETE RESTRICT ON UPDATE CASCADE NOT NULL,
name text NOT NULL UNIQUE
);
Proposition 0: Data lasts longer than apps, so I am convinced that I want referential integrity enforced at the DB level as well as validations in my RoR models, despite the lack of DRYNESS.
Proposition 1: If I replace the script and sql files with Migrations, it is currently impossible to tell my Postgres database about the Foreign Key and other constraints I currently set in SQL DDL files within the migration code.
Proposition 2: The touted benefit of migrations is that changes to the schema are versioned along with the RoR model code. But if I keep my scripts and .sql files in railsapp/db, I can version them just as easily.
Proposition 3: Given that migrations lack functionality I want, and provide benefits I can replicate, there is little reason for me to consider using them. So I should --skipmigrations at script/generate model time.
My question: If Proposition 0 is accepted, are Propositions 1,2,3 true or false, and why?
Thanks!
Proposition 1 is false in at least two situations - you can use plugins like foreign_key_migrations to do the following:
def self.up
create_table :users do |t|
t.column :department_id, :integer, :references => :departments
end
end
which creates the appropriate foreign key constraint in your DB.
Of course, you might have other things that you want to do in your DDL, in which case the second situation becomes more compelling: you're not forced to use the Ruby DSL in migrations. Try the execute method, instead:
def self.up
execute 'YOUR SQL HERE'
end
With that, you can keep the contents of your SQL scripts in migrations, gaining the benefits of the latter (most prominently the down methods, which you didn't address in your original question) and retaining the lower-level control you prefer.
Proposition 1 is mistaken : you can definitely define referential integrity using migrations if only by using direct SQL inside the migration, see this post for more details.
Proposition 2: The touted interest of migrations is to be able to define your database model incrementally while keeping track of what each change added and be able to easily rollback any such change at a later time.
You have to be careful with the order you create/modify things in but you can do it.
One thing to keep in mind : rails is better suited for application-centri design. in the Rails Way(tm) the database is only ever accessed through the application active record layer and exposes data to the outside using webservices
1: You may want to try out this plugin. I didn't try it myself though, but it seems to be able to add foreign key constraints through migrations.
2: The real benefit of migration is the ability to go back and forth in the history of your database. That's not as easy with your .sql files.
3: See if the above-mentioned plugin works for you, then decide :) At any rate, it's not a capital sin if you don't use them!
Since you are using Postgres and may not want to install the foreign_key_migrations plugin, here is what I do when I want to use both migrations and foreign key constraints.
I add a SchemaStatements method to ActiveRecord::SchemaStatements called "add_fk_constraint".
This could go in some centralized file, but in the example migration file below, I have just put it inline.
module ActiveRecord
module ConnectionAdapters # :nodoc:
module SchemaStatements
# Example call:
# add_fk_constraint 'orders','advertiser_id','advertisers','id'
# "If you want add/alter a 'orders' record, then its 'advertiser_id' had
# better point to an existing 'advertisers' record with corresponsding 'id'"
def add_fk_constraint(table_name, referencing_col, referenced_table, referenced_col)
fk_name = "#{table_name}_#{referencing_col}"
sql = <<-ENDSQL
ALTER TABLE #{table_name}
ADD CONSTRAINT #{fk_name}
FOREIGN KEY (#{referencing_col}) REFERENCES #{referenced_table} (#{referenced_col})
ON UPDATE NO ACTION ON DELETE CASCADE;
CREATE INDEX fki_#{fk_name} ON #{table_name}(#{referencing_col});
ENDSQL
execute sql
end
end
end
end
class AdvertisersOrders < ActiveRecord::Migration
def self.up
create_table :advertisers do |t|
t.column :name, :string, :null => false
t.column :net_id, :integer, :null => false
t.column :source_service_id, :integer, :null => false, :default => 1
t.column :source_id, :integer, :null => false
end
create_table :orders do |t|
t.column :name, :string, :null => false
t.column :advertiser_id, :integer, :null => false
t.column :source_id, :integer, :null => false
end
add_fk_constraint 'orders','advertiser_id','advertisers','id'
end
def self.down
drop_table :orders
drop_table :advertisers
end
end
I hopes this helps someone. It has been very useful to me since I need to load a lot of externally supplied data with SQL "COPY" calls, yet I find the migrations system very convenient.

Resources