My Rails 5 schema.rb file has an options: section for each create_table that I do not want. What I have is:
create_table "accounts", id: false, force: :cascade, options: "ENGINE=InnoDB DEFAULT CHARSET=utf8" do |t|
but what I want is:
create_table "accounts", id: false, force: :cascade do |t|
Since this is a generate file, I don't want to edit it manually.
The reason I don't want the options section is because for a fast TDD loop, I want to use SQLite in memory when RAILS_ENV=test but MySQL in development and production.
How do I prevent DB adapter specific options from being generated and put into the schema?
Rails 5 broke the Schema dump format
In Rails 5 the core team decided to change the schema dump format and there were basically two issues with the new schema format:
1) the create_table statements now have options which are specific for the adapter, e.g. MySQL as in the OP's example.
2) the dumped columns do not show :limit statements if the value is the default value for the column type.
Why is that bad?
1) ActiveRecord is supposed to be DB agnostic, and some people use the schema files to load them into a different kind of database, e.g. load the schema file into a SQLite DB for faster testing.
SQLite does not understand the MySQL specific options, and breaks miserably.
2) Limits between different versions of the same kind of database could change over time, and limits could definitely change when you go from one DB to another DB.
Therefore it is not a good idea to just not show the actual value of the "default" limit.
Work Around
It is a terrible idea to monkey patch ActiveRecord. I hate having to do this, but that was the easiest way to get our schema files back to a DB-agnostic state, so we could do testing with SQLite.
If you are dumping your schema FROM MySQL on Rails 5.0,
then you can fix it by adding these two files to your Rails 5.0 project:
config/initializers/active_record/schema_dumper.rb
config/initializers/active_record/connection_adapters/abstract/schema_dumper.rb
These two files contain the original code form the 5-0-stable branch of Rails, slightly modified to not do (1) and (2).
It's not pretty, but it will keep you generating the same dump format as before.
Hopefully the core team will fix the issue soon.
If you have a better solution, which allows cross-DB usage of schema files, please comment or post a better solution.
I'm really not a big fan of monkey-patching :-(
The Rails core team is aware of this and locked the thread down because they were tired of hearing about it. I wouldn't look for a fix any time soon. While I agree that testing in the same db as you're prod environment as a rule of thumb, this is not always a good idea.
Forcing it is a really, really bad idea.
https://github.com/rails/rails/issues/26209
Related
After upgrading to Rails 5 my schema file keeps getting altered when running db:migrate. Rails is changing:
create_table "flightlessons", force: :cascade do |t|
to:
create_table "flightlessons", id: :integer, default: -> { "nextval('lessons_id_seq'::regclass)" }, force: :cascade do |t|
It only occurs on this one model. Why is rails implementing nextval on this particular model? And, why is it getting the model name wrong (lessons_id_seq should be flightlessons_id_seq). Manually changing it to flightlessons_id_seq, however, results in the same no relation error.
PG::UndefinedTable: ERROR: relation "lessons_id_seq" does not exist
To proceed, I simply alter the schema.rb file back to what that line 'should' be. Then, I can migrate or test:prepare or whatever until the next time rails alters it back to using the nextval method.
Thank you for any insight into this.
This is a bit long of an answer, so I've broken it into sections. Buckle up!
My theory
My guess is that your development database does contain the lessons_id_seq sequence, and that its definition of flightlessons.id is set to depend on it (i.e., exactly what Rails is putting into your schema file).
How and why? You likely renamed the lessons table to flightlessons at some point in the past, but that rename didn't change the sequence that the table depended on -- and since schema.rb does not record sequences, the lessons_id_seq sequence does not get copied to your test database, and thus you get this error.
To verify my theory, run rails db and try the following commands:
\d lessons_id_seq
This should return the definition of that sequence. Then, try:
\d flightlessons
And look at the definition of the id column. I expect it to include DEFAULT nextval('lessons_id_seq').
Fixes
The easiest way to fix this is to switch to using structure.sql instead of schema.rb (see the docs). This will carry over the exact state of your database and avoid any interference or interpretation by Rails, which is what's causing your current issue. I always recommend structure.sql for production systems.
However, you can also go into your development database and change the sequence name:
ALTER SEQUENCE lessons_id_seq RENAME TO flightlessons_id_seq;
ALTER TABLE flightlessons ALTER COLUMN id SET DEFAULT nextval('flightlessons_id_seq');
This would be a terrible idea on a production system, but if your issue is just local, it should rectify your current database state with your schema.rb and thus address your current problem. You may wish to encode that into a migration, if you want rails db:drop db:create db:migrate to work on a fresh app.
Why now?
The behavior where Rails is dumping out the default value for your table's primary key may very well be new in Rails 5. Previously, Rails may have just trusted that your ID column had a sane default, and ignored whatever value it actually saw. But I haven't done the research to see if that's true or not.
The simplest fix is to just to rename the sequence in production to match the current table name. E.g. in a production Rails console:
ActiveRecord::Base.connection.execute("ALTER SEQUENCE lessons_id_seq RENAME TO flightlessons_id_seq;")
Turns out it's fine to just rename it, if you haven't done anything fancy with the sequence (like implementing your own Postgres function that references it by name).
Apparently the table points to the sequence by ID, not by name, so the rename is instant and with no ill effects that we could see. More details here: https://dba.stackexchange.com/questions/265569/how-can-i-safely-rename-a-sequence-in-postgresql-ideally-without-downtime
We tried it on staging first, and verified that the ID sequence kept on ticking after making the change in staging and production. Everything just worked.
(Also see Robert Nubel's fantastic answer for more details on what's going on.)
Like some others, I am having trouble with an error
"ActiveRecord::StatementInvalid: SQLite3::SQLException: object name
reserved for internal use: sqlite_sp_functions: CREATE TABLE
"sqlite_sp_functions" ("name" text, "text" text)"
when running rake test on a Rails project.
The offending lines in schema.rb are:
create_table "sqlite_sp_functions", id: false, force: true do |t|
t.text "name"
t.text "text"
end
The suggestions on the previous query about this involved editing schema.rb or deleting that file & regenerating it, but schema.rb (and the offending code) are regenerated on each migration (plus, I don't want to delete Rails-generated code without knowing the implications), so that's not really a satisfactory solution.
So what is the sqlite_sp_functions table for, and how can I get Rails to generate a schema.rb file that doesn't break rake test for the project?
as per https://stackoverflow.com/a/25077833/601782:
Add the following config line to config/application.rb or in config/environments/development.rb:
ActiveRecord::SchemaDumper.ignore_tables = /^sqlite_*/
This will ignore all tables starting with "sqlite_" in your database during the schema dump.
Try removing the offending lines from your schema.rb and then issue:
RAILS_ENV=test rake db:reset
which will completely nuke your testing database but you shouldn't care anyway. You are not supposed to run migrations for test environments. Migrations are meant to be small (reversible) steps for environments that hold data (such as production and sometimes staging/dev).
The preferred way to handle your testing DB is to use db:schema:load as part of your testing routine which will of course delete all your DB data.
Again: Please do not try this in development mode (if you have data that you set up manually) and definitely not in production.
Other than that, you could probably delete the whole sqlite_sp_functions table from your SQLite testing DB and get rid of the issue altogether. I don't think that this has anything to do with Rails (nor is Rails generating that). I believe that SQLite created this table and your schema just picks it up (as it should). Apparently the table is used to hold stored procedures of some kind.
Let's say I have a migration that creates a table 'pages' , this is my migration:
class CreatePages < ActiveRecord::Migration
def up
create_table :pages do |t|
t.string "name" , :limit => 50
t.integer "permalink"
t.integer "position"
t.timestamps
end
end
def down
drop_table :pages
end
end
and that I've created the appropriate migration file X_create_pages.rb and ran it(the table is created in the database).
Now after a few days I realize the structure isn't complete and I need to add another column to my pages table .
What is the best practice , do I create a new migration file with add_column method or do I just change the current migration file's up method -e.g just add my columns to the up method (and then move down a version and then up again - so to run the up method?)
The practice should be you create a new migration to add the new column. This becomes the simplest and risk free path. Of course, we can talk about unseen circumstances!
Updating original migration files should not be used after application is released to production; at least this is what the practice should be. But, this also depends on applications, if you don't have a table that is referenced elsewhere then backing up the data, adding the column and restoring the data is also a possibility which is enabled by this approach of directly modifying the original migration file and rolling down that version and migrating up that version again.
While you are on development, the choice I guess is yours. You could choose either!
Once a migration has been checked into source control it shouldn't be changed.
Modifying a migration on your development environment providing it hasn't been checked into source control is perfectly fine.
Once a migration has been checked in it may have been run by your team mates on their development environment or even been deployed and run in production.
Changing previously committed migrations is going to make you very unpopular in your team very quickly as team mates struggle to understand why their database schema is different from yours but all the migrations have been run.
I have a pretty old migration on a legacy app by a friend that contains this snippet:
class MakeChangesToProductionDbToMatchWhatItShouldBe < ActiveRecord::Migration
def self.up
# For some reason ActiveRecord::Base.connection.tables.sort().each blows up
['adjustments',
'accounts',
...## more rows of classes here ###...
'product_types'].each do |table|
t = table.singularize.camelize.constantize.new
if t.attributes.include?('created_at')
change_column( table.to_sym, :created_at, :datetime, :null => false ) rescue puts "#{table} doesnt have created_at"
end
if t.attributes.include?('updated_at')
change_column( table.to_sym, :updated_at, :datetime, :null => false ) rescue puts "#{table} doesnt have updated_at"
end
end
This old migration is now conflicting with a new migration I wrote to remove two of the tables mentioned in this long list, which is now causing any deployment to error upon rake db:migrate.
What's the correct kind of migration or down action to write to address this migration and get db:migrate working again?
There are a few different best practices that can help, but at the end of the day there's no good way to always upgrade a database from an arbitrary point without stepping through the codebase along as you run migrations (speaking of which, why is there not already a rake task to do this?).
Always include a migration-namespaced copy of the models you're working on. Example below.
When building a database from scratch, do not run migrations…use db:schema:load which will re-create the last snapshot of the database.
Don't give your migrations ridiculous and aggression fueled titles like MakeChangesToProductionDbToMatchWhatItShouldBe.
Avoid making assumptions, when writing migrations, about the environment they will be run in. This includes specifying table names, database drivers, environment variables, etc.
Write down actions when you write up actions whenever a down action is feasible. It's usually much easier (especially on esoteric or complex migrations) when the series of transformations is fresh in your head.
For this specific case, there's an argument to be made for declaring “Migration Bankruptcy” — clearing out some or all existing migrations (or refactoring and coalescing into a new one) to achieve the desired database state. When you do this you are no longer backwards compatible so it is not to be taken lightly, but there are times it is the appropriate move.
I'm currently running a Rails migration where I am adding a datatype specific to Postgres, the tsvector. It holds search information in the form that Postgres expects for its built-in text searching capabilities.
This is the line from my migration:
t.column "search_vectors", :tsvector
Everything seems to be working fine, and the search works with it. However, when I opened up schema.rb, this is what I got:
Could not dump table "users" because of following StandardError
Unknown type 'tsvector' for column 'search_vectors'
This is preventing me from running unit tests on the user table, and also strikes me as really dangerous looking given that the schema.rb is supposed to be the authoritative definition of my database.
I notice there are a number of Rails plugins that seem to use the same approach of storing the tsvector like I would expect, such as tsearchable. Am I really just stuck without testing and without an authoritative definition of my database?
FYI for anyone who happens across this page, I fixed this by adding this (actually uncommenting it) to my Rails config:
config.active_record.schema_format = :sql
Have you tried specifying the type as a string instead of a symbol?
t.column "search_vectors", "tsvector"
If that doesn't work then you might need to drop down to database-specific SQL:
def self.up
execute "--Put your PostgreSQL specific SQL statements here"
end