Like some others, I am having trouble with an error
"ActiveRecord::StatementInvalid: SQLite3::SQLException: object name
reserved for internal use: sqlite_sp_functions: CREATE TABLE
"sqlite_sp_functions" ("name" text, "text" text)"
when running rake test on a Rails project.
The offending lines in schema.rb are:
create_table "sqlite_sp_functions", id: false, force: true do |t|
t.text "name"
t.text "text"
end
The suggestions on the previous query about this involved editing schema.rb or deleting that file & regenerating it, but schema.rb (and the offending code) are regenerated on each migration (plus, I don't want to delete Rails-generated code without knowing the implications), so that's not really a satisfactory solution.
So what is the sqlite_sp_functions table for, and how can I get Rails to generate a schema.rb file that doesn't break rake test for the project?
as per https://stackoverflow.com/a/25077833/601782:
Add the following config line to config/application.rb or in config/environments/development.rb:
ActiveRecord::SchemaDumper.ignore_tables = /^sqlite_*/
This will ignore all tables starting with "sqlite_" in your database during the schema dump.
Try removing the offending lines from your schema.rb and then issue:
RAILS_ENV=test rake db:reset
which will completely nuke your testing database but you shouldn't care anyway. You are not supposed to run migrations for test environments. Migrations are meant to be small (reversible) steps for environments that hold data (such as production and sometimes staging/dev).
The preferred way to handle your testing DB is to use db:schema:load as part of your testing routine which will of course delete all your DB data.
Again: Please do not try this in development mode (if you have data that you set up manually) and definitely not in production.
Other than that, you could probably delete the whole sqlite_sp_functions table from your SQLite testing DB and get rid of the issue altogether. I don't think that this has anything to do with Rails (nor is Rails generating that). I believe that SQLite created this table and your schema just picks it up (as it should). Apparently the table is used to hold stored procedures of some kind.
Related
The app in question was originally created as a Rails 4 app, and later upgraded to Rails 5.
I will create a rails migration that might look like this:
class AddPubliclyVisibleToGcodeMacros < ActiveRecord::Migration[5.0]
def change
add_column :gcode_macros, :publicly_visible, :boolean, default: false
end
end
And when I run it, I expect the schema to have a few lines updated, specifically adding t.boolean "publicly_visible", default: false
to the gcode_macros table.
However, running the migration creates a LOT of changes to my schema, mostly just moving indexes from outside a create_table block, into it.
Im quite confused over whats going on here. This isn't something that happened all of a sudden, I've just been working around it for a while now.
Any help would be greatly appreciated!
The answer is that this is just how the schema dumper works in Rails. It takes the schema from the database, absolutely irrelevant from how you created the structure in the first place, whether with migrations or direct sql statements.
So when you create a new migration or change anything in the db a new schema is dumped based on the database.
Edit
I should add that schema.rb is not updated if the db is changed directly with sql statements, that is not through a migration. Only when either
rake db:migrate
or ...
rake db:schema:dump
are run is the schema.rb file updated.
After upgrading to Rails 5 my schema file keeps getting altered when running db:migrate. Rails is changing:
create_table "flightlessons", force: :cascade do |t|
to:
create_table "flightlessons", id: :integer, default: -> { "nextval('lessons_id_seq'::regclass)" }, force: :cascade do |t|
It only occurs on this one model. Why is rails implementing nextval on this particular model? And, why is it getting the model name wrong (lessons_id_seq should be flightlessons_id_seq). Manually changing it to flightlessons_id_seq, however, results in the same no relation error.
PG::UndefinedTable: ERROR: relation "lessons_id_seq" does not exist
To proceed, I simply alter the schema.rb file back to what that line 'should' be. Then, I can migrate or test:prepare or whatever until the next time rails alters it back to using the nextval method.
Thank you for any insight into this.
This is a bit long of an answer, so I've broken it into sections. Buckle up!
My theory
My guess is that your development database does contain the lessons_id_seq sequence, and that its definition of flightlessons.id is set to depend on it (i.e., exactly what Rails is putting into your schema file).
How and why? You likely renamed the lessons table to flightlessons at some point in the past, but that rename didn't change the sequence that the table depended on -- and since schema.rb does not record sequences, the lessons_id_seq sequence does not get copied to your test database, and thus you get this error.
To verify my theory, run rails db and try the following commands:
\d lessons_id_seq
This should return the definition of that sequence. Then, try:
\d flightlessons
And look at the definition of the id column. I expect it to include DEFAULT nextval('lessons_id_seq').
Fixes
The easiest way to fix this is to switch to using structure.sql instead of schema.rb (see the docs). This will carry over the exact state of your database and avoid any interference or interpretation by Rails, which is what's causing your current issue. I always recommend structure.sql for production systems.
However, you can also go into your development database and change the sequence name:
ALTER SEQUENCE lessons_id_seq RENAME TO flightlessons_id_seq;
ALTER TABLE flightlessons ALTER COLUMN id SET DEFAULT nextval('flightlessons_id_seq');
This would be a terrible idea on a production system, but if your issue is just local, it should rectify your current database state with your schema.rb and thus address your current problem. You may wish to encode that into a migration, if you want rails db:drop db:create db:migrate to work on a fresh app.
Why now?
The behavior where Rails is dumping out the default value for your table's primary key may very well be new in Rails 5. Previously, Rails may have just trusted that your ID column had a sane default, and ignored whatever value it actually saw. But I haven't done the research to see if that's true or not.
The simplest fix is to just to rename the sequence in production to match the current table name. E.g. in a production Rails console:
ActiveRecord::Base.connection.execute("ALTER SEQUENCE lessons_id_seq RENAME TO flightlessons_id_seq;")
Turns out it's fine to just rename it, if you haven't done anything fancy with the sequence (like implementing your own Postgres function that references it by name).
Apparently the table points to the sequence by ID, not by name, so the rename is instant and with no ill effects that we could see. More details here: https://dba.stackexchange.com/questions/265569/how-can-i-safely-rename-a-sequence-in-postgresql-ideally-without-downtime
We tried it on staging first, and verified that the ID sequence kept on ticking after making the change in staging and production. Everything just worked.
(Also see Robert Nubel's fantastic answer for more details on what's going on.)
I have a question regarding my migrations in rails.
Normally when i want to add a colum to a model for i dont make extra migrations but instead i perform this steps:
rake db:rollback
next i change the migration file in db/migrations and rerune:
rake db:migrate
The biggest problem is that when i do this i loose my data.
Previous i wrote migrations from the command line with for example
rake g migration Add_Column_House_to_Users house:string
The problem with this approach is that my db/migrations folder afterwards get very large and not very clear! I mean at the end i dont know wich variables the object has! Im not an expert in rails and would like to ask you how to keep the overview over the migrations!Thanks
Just a minor thought - I just use the file db/migrate/schema.rb to determine whats in the database as opposed to tracking through the migrations
You definitely shouldn't use db:rollback with a table with existing data.
I have a few production RonR apps with a ton of data and there are 100+ entries in the migrations table and adding new migrations to tweak tables is the rails way to do things. Not sure what you mean by lucid, but your schema and data model are going to change over time and that is ok and expected.
One tip. The migrations are great, but they are just the beginning, you can include complex logic as needed to fix your existing data (like so)
Changing data in existing table:
def up
add_column :rsvps, :column_name_id, :integer
update_data
end
def update_data
rsvps = Rsvp.where("other_column is not null")
rsvps.each do |rsvp|
invite = Blah.find(rsvp.example_id)
...
rsvp.save
end
end
Another tip: backup your production database often (should do this anyway), but use it to test all of your migrations before deploying. I run scripts like this all the time for local testing:
mysql -u root -ppassword
drop database mydatabase_dev;
create database mydatabase_dev;
use mydatabase_dev;
source /var/www/bak/mydatabase_backup_2013-10-04-16.28.06.sql
exit
rake db:migrate
I had used default generator to create some tables and they all had that t.timestamp in their definition so the schema that was created also has created_at and updated_at fields.
Now I am told that I don't need those two fields in my schema so I went to the original create_table* files and took out t.timestamp line from them and ran the db:migrate and schema:load commands
But still when I go to my schema.rb file I can see they are still there.
Is there anything wrong I am doing here?
Run
rails g migration remove_timestamps_from_table created_at updated_at
with table being your model's name. Since this is following the pattern remove x from y, rails is smart enough to generate the appropriate migration for you.
Then run
rake db:migrate
to update your development database and
rake db:test:prepare
to prepare the test database, and you're all set!
Read more on migrations here. If you are still having trouble, consider restarting your rails server or database server.
If I edit my shema, and run db:migrate the database doesn't changed, but if I clear to version 0, and recall migration it's works, but I lost all database data.
What's wrong?
That's how db:migrate works. It maintains a table in the database called schema_migrations that keeps track of the migration timestamps (i.e. if your file is called 20090807152224_create_widgets.rb, the 20090807152224 part is the timestamp -- and the line that will get added to your schema_migrations table).
You're not supposed to modify the schema.rb file by hand -- that file gets autogenerated as a result of db:migrate.
The thinking in rails is that if you want to make a change to your schema, you're going to generate a new migration with those changes and then run db:migrate (which, as a result, will update the schema.rb file appropriately).
When you say you are updating your schema, does that mean you're updating the db/schema.rb file, or actual migrations?
If you're updating the schema.rb file, you should note that it will have no effect because the file is auto-generated.
See the comment at the top of the file:
# This file is auto-generated from the current state of the database. Instead of editing this file,
# please use the migrations feature of Active Record to incrementally modify your database, and
# then regenerate this schema definition.
It sounds like you were changing a migration file.
Don't change migration files. Add new ones. You can have migrations that change say, the type of a column. There are times when changing old migrations may be helpful, but don't do it unless you know the consequences. As others have mentioned, don't change the schema either, but I don't think you were doing that.