In Rails 7:
Migration:
class AddSavedSearchAPIKeyIndex < ActiveRecord::Migration[7.0]
def change
execute <<~SQL
ALTER TABLE saved_searches
ADD INDEX ecomm_analyze_api_key ((
CAST(configuration->>"$.ecomm_analyze_api_key" as CHAR(255)) COLLATE utf8mb4_bin
)) USING BTREE;
SQL
end
end
My schema ends up looking like this:
t.index "(cast(json_unquote(json_extract(`configuration`,_utf8mb4\\'$.ecomm_analyze_api_key\\')) as char(255) charset utf8mb4) collate utf8mb4_bin)", name: "ecomm_analyze_api_key"
I can't load this schema.rb without getting an error when I use:
RAILS_ENV=test rake db:test:prepare
Error message
ActiveRecord::StatementInvalid: Mysql2::Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use
I am able to fix this by manually editing the schema.rb but this will just break again on the next migration.
The Ruby schema dumper only actually understands a limited subset of SQL that can be created by the migrations DSL and is only actually useful until your app reaches the degree of complexity where you want to use database specific features.
Anything else that the Ruby schema dumper doesn't properly understand like this index will either be mangled or just lost in translation when it tries to parse the resulting SQL from dumping the database schema.
The solution is to use SQL schema dumps instead:
module YourApp
class Application < Rails::Application
# Add this line:
config.active_record.schema_format = :sql
end
end
This will dump the file structure.sql which should serve as the source of truth for the database schema. Check this file into version control and delete schema.rb to avoid confusion.
See:
Rails Guides : Schema Dumping and You
Pros and Cons of Using structure.sql in Your Ruby on Rails Application
Related
I have this migration in a Rails 7 app using Postgresql 13:
class AddContractsDateRangeConstraint < ActiveRecord::Migration[7.0]
def up
execute "CREATE EXTENSION IF NOT EXISTS btree_gist"
execute <<~SQL
ALTER TABLE contracts
ADD CONSTRAINT date_overlap_exclude
EXCLUDE USING GIST (
employee_id WITH =,
daterange(starts_on, ends_on) WITH &&
)
SQL
end
def down
execute "ALTER TABLE contracts DROP CONSTRAINT date_overlap_exclude"
end
end
When executing via rails db:migrate the constraint gets added to the DB like this and everything works:
app_development=# \d+ contracts
...
Indexes:
...
"date_overlap_exclude" EXCLUDE USING gist (employee_id WITH =, daterange(starts_on, ends_on) WITH &&)
...
The generated schema.rb from Rails looks like this:
create_table "contracts", force: :cascade do |t|
# ...
t.index "employee_id, daterange(starts_on, ends_on)", name: "date_overlap_exclude", using: :gist
# ...
end
This looks suspicious, as it lacks the whole EXCLUDE part from my constraint. And indeed, when creating the database from the generated schema using rails db:schema:load the constraint is broken and is generated like this:
app_development=# \d+ contracts
...
Indexes:
"contracts_pkey" PRIMARY KEY, btree (id)
"date_overlap_exclude" gist (employee_id, daterange(starts_on, ends_on))
And of course, the whole constraint doesn't work anymore. Any idea how to solve this?
The Rails Ruby schema dumper works very well for simple applications and use cases but it only actually understands a very limited subset of SQL. As you have experienced here any parts of the database schema that it doesn't understand are simply lost in translation.
Once you get the point where you need to use functions, constraints, materialized views or any other database specific features you need to use SQL schema dumps which are much more high fidelity since its just using the databases own dumping tools to output the schema as SQL. While you're losing the simplicity and database-agnostic characteristics of schema.rb this is an inevitable change as the complexity of the application grows.
You can switch the schema format with a simple line in your configuration:
module YourApp
class Application < Rails::Application
# Add this line:
config.active_record.schema_format = :sql
end
end
When you run rails db:schema:dump it will now output a structure.sql file instead. At this point you should delete your schema.rb and check structure.sql into version control.
See:
Pros and Cons of Using structure.sql in Your Ruby on Rails Application
Rails Guides : Schema Dumping and You
I have created a migration with custom postgres function creation:
class CreatePopularityPgFunctions < ActiveRecord::Migration[5.2]
def up
execute %{
CREATE OR REPLACE FUNCTION popularity(count integer, weight integer default 3) RETURNS integer AS $$
SELECT count * weight
$$ LANGUAGE SQL IMMUTABLE;
}
end
def down
execute 'drop function popularity(integer, integer) cascade'
end
end
and running rake db:migrate properly adds it to the schema. However, running rake db:reset seems to not create this function in PG for some reason. The function is not in the schema and trying to use it in SQL query results in error about missing function.
db:reset runs a db:drop db:setup. In db:setup creates a database schema. But schema.rb doesn't handle a custom functions or views.
From documentation:
db/schema.rb cannot express database specific items such as foreign key constraints, triggers, or stored procedures. While in a migration you can execute custom SQL statements, the schema dumper cannot reconstitute those statements from the database. If you are using features like this, then you should set the schema format to :sql.
You can use structure.sql instead. To use structure.sql:
This is set in config/application.rb by the config.active_record.schema_format setting, which may be either :sql or :ruby.
More details here.
How could I create an index like this
CREATE INDEX index_companies_on_addresses_on_zipcode ON companies USING gin ((addresses -> 'zipcode'));
using the syntax of migrations?
I've created using the syntax below and it was created on database
execute "CREATE INDEX index_companies_on_addresses_on_zipcode ON companies USING gin ((addresses -> 'zipcode'));"
but when I saw schema.rb index wasn't there.
I've tried
add_index :companies, :addresses, using: :gin
but it create an index only on column addresses(is a jsonb) not on key zipcode.
AFAIK there's no way to make add_index understand that sort of indexing. Fear not, there are ways around this. You're using jsonb so there's no need to worry about database portability so you can switch from db/schema.rb to db/structure.sql for managing your schema. The db/structure.sql file is pretty much a native SQL dump of your database structure so it will contain everything that the database understands (CHECK constraints, advanced indexes, ...).
The process is quite simple:
Edit config/application.rb to set the schema format:
class Application < Rails::Application
#...
config.active_record.schema_format = :sql
#...
end
Run your migration that uses execute as normal. You'll want to use separate up and down methods in your migrations or reversible inside the usual change method because ActiveRecord won't know how to reverse execute some_raw_sql on its own.
This should leave you with a db/structure.sql file so add that to your revision control and delete db/schema.rb.
Use different rake tasks for working with structure.sql instead of schema.rb:
db:structure:dump instead of db:schema:dump
db:structure:load instead of db:schema:load
I have an application that requires a sequence to be present in the database. I have a migration that does the following:
class CreateSequence < ActiveRecord::Migration
def self.up
execute "CREATE SEQUENCE sequence"
end
def self.down
execute "DROP SEQUENCE sequence"
end
end
This does not modify the schema.rb and thus breaks rake db:setup. How can I force the schema to include the sequence?
Note: The sequence exists after running rake db:migrate.
Rails Migrations because they aim toward a schema of tables and fields, instead of a complete database representation including stored procedures, functions, seed data.
When you run rake db:setup, this will create the db, load the schema and then load the seed data.
A few solutions for you to consider:
Choice 1: create your own rake task that does these migrations independent of the Rails Migration up/down. Rails Migrations are just normal classes, and you can make use of them however you like. For example:
rake db:create_sequence
Choice 2: run your specific migration after you load the schema like this:
rake db:setup
rake db:migrate:up VERSION=20080906120000
Choice 3: create your sequence as seed data, because it's essentially providing data (rather than altering the schema).
db/seeds.rb
Choice 4 and my personal preference: run the migrations up to a known good point, including your sequence, and save that blank database. Change rake db:setup to clone that blank database. This is a bit trickier and it sacrifices some capabilities - having all migrations be reversible, having migrations work on top of multiple database vendors, etc. In my experience these are fine tradeoffs. For example:
rake db:fresh #=> clones the blank database, which you store in version control
All the above suggestions are good. however, I think I found a better solution. basically in your development.rb put
config.active_record.schema_format = :sql
For more info see my answer to this issue -
rake test not copying development postgres db with sequences
Check out the pg_sequencer gem. It manages Pg sequences for you as you wish. The one flaw that I can see right now is that it doesn't play nicely with db/schema.rb -- Rails will generate a CREATE SEQUENCE for your tables with a serial field, and pg_sequencer will also generate a sequence itself. (Working to fix that.)
I'm an idiot...screwed up a migration in Rails:
thinking migrations would work like model generators (using references:modelname) I did the following:
$ rails g migration add_event_to_photos references:event
which created the migration
class AddEventToPhotos < ActiveRecord::Migration
def change
add_column :photos, :references, :event
end
end
And now my development database (SQLite3) has a references column of type event in the photos table.
And my schema.rb has a line in the middle saying:
# Could not dump table "photos" because of following StandardError
# Unknown type 'event' for column 'references'
rake db:rollback is powerless against this:
$ rake db:rollback
== AddEventToPhotos: reverting ===============================================
-- remove_column("photos", :references)
rake aborted!
An error has occurred, this and all later migrations canceled:
undefined method `to_sym' for nil:NilClass
So, how to roll back and maintain my development data in the database? I'd even be happy trashing the photos table if that's my only choice..but don't want to have to rebuild the whole thing. What to do?
btw- for anyone reading this about to make same stupid mistake...don't! Use the correct migration generator:
$ rails g migration add_event_to_photos event_id:integer
The easiest way I found to do this was to recreate the table in the schema.rb file in /db/. Afterwards I ran a rake db:reset (if it says you have pending migrations, just delete them and try again).
This took care of the problem.
Go into the database by ./script/rails dbconsole. Then type these commands:
.output dump.sql
.dump
In the file dump.sql you will have the SQL commands used to recreate and populate your database. Just edit it with your favourite editor (like vim ;-) removing or fixing the column type. You may also remove the invalid migration identifier from the schema_migrations table. Drop your database (I suggest just rename the db/development.sqlite file), create new database and read the dump file into it (using command .read dump.sql).
Now you just need to fix and run your migrations.
add an empty down method and run rake db:rollback
edit ahh that's the new migration syntax, you can replace the body with simply:
def self.down; end
which is the old syntax, or perhaps delete the body altogether (haven't tried this) and then run rake db:rollback
Just an idea, I know it's not SQLite specific you can revert to an older version schema perhaps, load it up. And try again from there? You can revert (checkout) specific files in GIT. And then do def self.down; end, as was suggested by another poster.
The problem arises because while SQLite will create the schema with whatever type you give it (in this case event it can't dump that type back to ActiveRecord.
You need to edit the sqlite_master file and change create table string (sql) to be the right thing.
You probably want to back up your table first since messing up that string will wreck your table if you do it wrong.
Here is a related rails issue