Trying to add Timestamps to existing table.
According to Api documenation add_timestamps
Here is my code in migration:
def change
add_timestamps(:products, null: false)
end
Getting error:
*-- add_timestamps(:products, {:null=>false})
rails aborted!
StandardError: An error has occurred, this and all later migrations canceled:
SQLite3::SQLException: Cannot add a NOT NULL column with default value NULL: ALTER TABLE "products" ADD "created_at" datetime NOT NULL*
I've also tried all solution in this thread
Same error...
Rails 5.1.4
Ruby 2.4.0
You cannot add columns with not-null constraint to a non-empty table because the existing lines in the table would have empty values right away and therefore the condition fails.
Instead, introduce the columns in three steps:
def change
# add new column but allow null values
add_timestamps :products, null: true
# backfill existing records with created_at and updated_at
# values that make clear that the records are faked
long_ago = DateTime.new(2000, 1, 1)
Product.update_all(created_at: long_ago, updated_at: long_ago)
# change to not null constraints
change_column_null :products, :created_at, false
change_column_null :products, :updated_at, false
end
In my opinion, it is wrong to manipulate existing data with activerecord queries or even SQL in migrations.
The correct rails 5.2+ way to do this is :
class AddTimestampsToCars < ActiveRecord::Migration[5.2]
def change
add_timestamps :cars, null: false, default: -> { 'NOW()' }
end
end
It's a proc so you should be able to set a date in the past if you want to.
Source: https://github.com/rails/rails/pull/20005
I like #spickermann's approach since it takes into account the existing records and probably your migration already went all the way to production, his method ensures data perseverance.
Nevertheless, many of you guys might find yourselves in that situation, but still in development, meaning that there's no real sensitive data you might be afraid of losing... That gives you a bit more freedom on how you can perform the change in the table.
If your code and records only exist locally (if you still have no records created, just skip step 1.) and that table was created in the last migration , my suggestion is:
1.- Delete all the records from that table.
2.- Go to your migration file and edit it by adding t.timestamps so that it looks something like this:
class CreateInstitutionalLegals < ActiveRecord::Migration[5.0]
def change
create_table :institutional_legals do |t|
# Your original migration content goes here
.
.
t.timestamps # This is your addition
end
end
end
3.- Then go to your console and enter rails:db:redo. As explained here, that command is a shortcut for doing a rollback and then migrating back up again.
Now you will see that your schema is updated with the corresponding created_atand updated_at columns.
The concrete benefit of this is that it is super easy to do, you don't create an extra migration file and you learn to use a very handy command ;)
I'm on rails 5.0 and none of these options worked. The rails:db:redo will work but isn't a feasible solution for most.
The only thing that worked was
def change
add_column :products, :created_at, :timestamp
add_column :products, :updated_at, :timestamp
end
I had the same issue. I wanted the end result to be strictly equivalent to add_timestamps :products on an fresh database.
Instead of running a query to backfill, I ended up doing a 3-steps process.
add column with null allowed and default to current time to backfill
change constraint to not null
remove default
And it is reversible.
add_column :products, :created_at, :datetime, precision: 6, null: true, default: -> { "CURRENT_TIMESTAMP" }
add_column :products, :updated_at, :datetime, precision: 6, null: true, default: -> { "CURRENT_TIMESTAMP" }
change_column_null :products, :created_at, false
change_column_null :products, :updated_at, false
change_column_default :products, :created_at, from: -> { "CURRENT_TIMESTAMP" }, to: nil
change_column_default :products, :updated_at, from: -> { "CURRENT_TIMESTAMP" }, to: nil
NB: This is with Rails 6.1 and PostgreSQL
Related
I'm currently in the process of writing code to convert our databases default ID columns to a UUID format.
One example of the code that I have for our migration is
def up
remove_reference :examples, :user, null: false, foreign_key: true
add_column :users, :uuid, :uuid, default: "gen_random_uuid()", null: false
change_table :users do |t|
t.remove :id, type: :id
t.rename :uuid, :id
end
execute "ALTER TABLE users ADD PRIMARY KEY (id);"
add_reference :examples, :user, null: false, foreign_key: true, type: :uuid
end
Essentially this allowed me to convert my ID column to a UUID format.
I created a down function so I would be able to rollback but it fails due to this error ERROR: column "user_id" of relation "examples" contains null values
I realize that there would be an issue because once there is data in the database it would be unable to rollback and create the correct references again. Does anyone have any ideas on how I should work on my down function?
def down
remove_reference :examples, :user, null: false, foreign_key: true, type: :uuid
execute 'ALTER TABLE users DROP CONSTRAINT users_pkey'
add_column :users, :new_id, :primary_key
change_table :users do |t|
t.remove :id, type: :uuid
t.rename :new_id, :id
end
add_reference :examples, :user, null: false, foreign_key: true
end
Does anyone have any suggestions on how I should proceed with this? The original migration was in one change function, but it would be unable to rollback due to the execute block.
Be careful doing this kind of database change in one-shot. I would suggest you to break into steps.
First step (new column uuid)
Create the new column for the uuid
def up
add_column :users, :uuid, :uuid, default: "gen_random_uuid()", null: false
add_column :examples, :user_uuid, :uuid
end
Adapt your code to populate the examples.user_uuid column with the recent created column; You can easily achieve this by creating a model callback, feeling the user_uuid automatically.
If your database has a GB of data, consider adding the uuid as nullable, and populate the column using queues or in batches. The new records will be already filled.
We have now two new columns, with new data comming and all synced
Second step (the renaming)
Once populated and working with new columns, is time to rename the columns and associate the new keys.
def up
change_table :examples do |t|
t.rename :user_id, type: :old_user_id
t.rename :user_uuid, :user_id
end
change_table :users do |t|
t.rename :id, type: :old_id
t.rename :uuid, :id
end
execute "ALTER TABLE users ADD PRIMARY KEY (id);"
add_reference :examples, :user, null: false, foreign_key: true, type: :uuid
end
def down
# ...
end
Rember to remove, or review the model code changed before, to support this new columns or ignore.
Now we have changed the columns, without loosing the old reference.
Be careful here, if your database is big, you may lock your operation. Perhaps you may need a maintenance window here.
Third step (removing the old columns)
Now we can remove the old columns and everything should work fine
note: Always be careful when making this kind of change into your database. It is very risk to perform something like this. If you want to go on, simulate several times the step of renaming. Make snapshot of your database before performing and inform your clients that might have a downtime in your service.
I don't know why you want to change your primary keys to be an uuid, this costs a lot to the database to query and join data. It's more complicated to compare an UUID than an integer. Consider just create a new indexed column uuid into your tables and let the database to join and based on this field.
I've already created a model in Rails to collect some user information
I created the columns as :string initially but I've since changed the way this data is looked up and entered by using separate populated models.
Now instead of entering into these fields as string - i want these columns to be "references" instead.
Is there an easy way to change from the string to reference without having to create a new model entirely?
*do not need to save the existing data
Is there any data in the strings you would like to save?
Or is it just because it has the same name?
You don't have to create a new model.
You could create a simple migration
remove_column :table, :your_column_name, :string
add_column :table, :your_column_name, :integer, references: :your_parent_model
You can add a temporary string column to save the string column first:
rails g migration add_temporary_string_column_to_model temporary_string_column:string
And run rails console:
SomeModel.all.each do |some_model|
some_mode.temporary_string_column = some_mode.string_column
some_mode.save
end
And now you can change your original string column's type to references which is an int(4) column in MySQL, migration like this:
class ChangeFormatInSomeTable < ActiveRecord::Migration
def change
change_column :some_table, :string_column, :references
end
end
Finally, you can run rails console again to convert the string data to integer like this:
SomeModel.all.each do |some_model|
some_mode.string_column = some_mode.temporary_string_column.to_i
some_mode.save
end
And at last, remove the temporary string column:
rails g migration remove_temporary_string_column_from_model temporary_string_column
Here is another solution, without dropping the column itself (not exactly in my case). I'm not sure though if this is the best solution.
In my case, I have a tickets table that holds purchase_uid in itself. I decided to keep purchases in another table after making the necessary improvements in our backend. Purchases table has uuid as the primary key. Given this background, here is my migration to change my column into a reference.
class AddPurchaseRelationToTickets < ActiveRecord::Migration[5.2]
def up
change_column :tickets, :purchase_uid, :uuid, references: :purchase, foreign_key: true, using: 'purchase_uid::uuid'
end
def down
change_column :tickets, :purchase_uid, :string
end
end
In my case, since string doesn't automatically cast into uuid, purchase_uid were dropped and recreated as well. However, if you decide to keep the column type same, I don't think it will be a problem.
You can create migrations to serve the exact purpose.
rails generate migration AddAddressToUsers address:references
This will create a migration file in db/migrate directory.
Then run: rails db:migrate to run migration and make changes in your database.
Don't forget to create associations in your models (belongs_to, has_many, etc.) depending on your system structure.
Wanted to add a simpler alternative to the accepted answer that preserves data:
class ChangeStringToInt < ActiveRecord::Migration[5.1]
def up
change_column :table_name, :field_name, :integer, null: false, references: :table_referenced, using: 'field_name::integer'
add_index :chapter_actions, :field_name
end
def down
change_column :table_name, :field_name, :string, null: false, using: 'field_name::character varying'
remove_index :table_name, :field_name
end
end
I have a rails model that has a non-defaulted boolean field that is nullable and am trying to set a default. I found a blog post about avoiding a 3-stat boolean issue, so am trying to cater for this. This is the migration I have:
def change
change_column :table, is_foo, :boolean, null: false, default: false
end
Running the migration fails because of existing null values in the database. What is the correct way to update existing entries to allow the schema change? Or should the not null control be added to the model:
validates :is_foo, presence: true
Not sure if adding this to the migration is the "right" way:
Table.update_all(:is_foo => false)
Similarly, this field was added by a migration without extra not null / default parameters. Would the migration to add column also require this, or would the default set the value? Here's the migration I ran:
add_column :table, is_foo, :boolean
If I had added ,null: false, default: false on the add_column, would all the values have been set correctly?
You can actually combine the change_column_null and change_column_default methods to accomplish this in a migration.
The change_column_null method lets you add a NOT NULL constraint, with the last argument specifying what to replace any existing NULL values with.
The change_column_default then sets the default for any new records.
class UpdateTable < ActiveRecord::Migration
def change
change_column_null :table, :is_foo, false, false
change_column_default :table, :is_foo, false
end
end
You could do it this way:
class UpdateFoo < ActiveRecord::Migration
def change
reversible do |direction|
direction.up {
Table.where(is_foo: nil).update_all(is_foo: false)
}
end
change_column :table, :is_foo, :boolean, null: false, default: false
end
end
When migrating up it will ensure first that all nulls are converted to false and then change the column to add your restrictions.
And yes, you could have avoided this if the first migration contained the restrictions.
I think you are also right adding the model validation.
I am getting the following error while trying to add a NOT NULL column to an existing table. Why is it happening ?. I tried rake db:reset thinking that the existing records are the problem, but even after resetting the DB, the problem persists. Can you please help me figure this out.
Migration File
class AddDivisionIdToProfile < ActiveRecord::Migration
def self.up
add_column :profiles, :division_id, :integer, :null => false
end
def self.down
remove_column :profiles, :division_id
end
end
Error Message
SQLite3::SQLException: Cannot add a NOT NULL column with default value NULL: ALTER TABLE "profiles" ADD "division_id" integer NOT NULL
This is (what I would consider) a glitch with SQLite. This error occurs whether there are any records in the table or not.
When adding a table from scratch, you can specify NOT NULL, which is what you're doing with the ":null => false" notation. However, you can't do this when adding a column. SQLite's specification says you have to have a default for this, which is a poor choice. Adding a default value is not an option because it defeats the purpose of having a NOT NULL foreign key - namely, data integrity.
Here's a way to get around this glitch, and you can do it all in the same migration. NOTE: this is for the case where you don't already have records in the database.
class AddDivisionIdToProfile < ActiveRecord::Migration
def self.up
add_column :profiles, :division_id, :integer
change_column :profiles, :division_id, :integer, :null => false
end
def self.down
remove_column :profiles, :division_id
end
end
We're adding the column without the NOT NULL constraint, then immediately altering the column to add the constraint. We can do this because while SQLite is apparently very concerned during a column add, it's not so picky with column changes. This is a clear design smell in my book.
It's definitely a hack, but it's shorter than multiple migrations and it will still work with more robust SQL databases in your production environment.
You already have rows in the table, and you're adding a new column division_id. It needs something in that new column in each of the existing rows.
SQLite would typically choose NULL, but you've specified it can't be NULL, so what should it be? It has no way of knowing.
See:
Adding a Non-null Column with no Default Value in a Rails Migration (2009, no longer available, so this is a snapshot at archive.org)
Adding a NOT NULL Column to an Existing Table (2014)
That blog's recommendation is to add the column without the not null constraint, and it'll be added with NULL in every row. Then you can fill in values in the division_id and then use change_column to add the not null constraint.
See the blogs I linked to for an description of a migration script that does this three-step process.
If you have a table with existing rows then you will need to update the existing rows before adding your null constraint. The Guide on migrations recommends using a local model, like so:
Rails 4 and up:
class AddDivisionIdToProfile < ActiveRecord::Migration
class Profile < ActiveRecord::Base
end
def change
add_column :profiles, :division_id, :integer
Profile.reset_column_information
reversible do |dir|
dir.up { Profile.update_all division_id: Division.first.id }
end
change_column :profiles, :division_id, :integer, :null => false
end
end
Rails 3
class AddDivisionIdToProfile < ActiveRecord::Migration
class Profile < ActiveRecord::Base
end
def change
add_column :profiles, :division_id, :integer
Profile.reset_column_information
Profile.all.each do |profile|
profile.update_attributes!(:division_id => Division.first.id)
end
change_column :profiles, :division_id, :integer, :null => false
end
end
You can add a column with a default value:
ALTER TABLE table1 ADD COLUMN userId INTEGER NOT NULL DEFAULT 1
The following migration worked for me in Rails 6:
class AddDivisionToProfile < ActiveRecord::Migration[6.0]
def change
add_reference :profiles, :division, foreign_key: true
change_column_null :profiles, :division_id, false
end
end
Note :division in the first line and :division_id in the second
API Doc for change_column_null
Not to forget that there is also something positive in requiring the default value with ALTER TABLE ADD COLUMN NOT NULL, at least when adding a column into a table with existing data. As documented in https://www.sqlite.org/lang_altertable.html#altertabaddcol:
The ALTER TABLE command works by modifying the SQL text of the schema
stored in the sqlite_schema table. No changes are made to table
content for renames or column addition. Because of this, the execution
time of such ALTER TABLE commands is independent of the amount of data
in the table. They run as quickly on a table with 10 million rows as
on a table with 1 row.
The file format itself has support for this https://www.sqlite.org/fileformat.html
A record might have fewer values than the number of columns in the
corresponding table. This can happen, for example, after an ALTER
TABLE ... ADD COLUMN SQL statement has increased the number of columns
in the table schema without modifying preexisting rows in the table.
Missing values at the end of the record are filled in using the
default value for the corresponding columns defined in the table
schema.
With this trick it is possible to add a new column by updating just the schema, operation that took 387 milliseconds with a test table having 6.7 million rows. The existing records in the data area are not touched at all and the time saving is huge. The missing values for the added column come on-the-fly from the schema and the default value is NULL if not otherwise stated. If the new column is NOT NULL then the default value must be set to something else.
I do not know why there is not a special path for ALTER TABLE ADD COLUMN NOT NULL when the table is empty. A good workaround is perhaps to create the table right from the beginning.
I have the problem, that I have an migration in Rails that sets up a default setting for a column, like this example:
def self.up
add_column :column_name, :bought_at, :datetime, :default => Time.now
end
Suppose, I like to drop that default settings in a later migration, how do I do that with using rails migrations?
My current workaround is the execution of a custom sql command in the rails migration, like this:
def self.up
execute 'alter table column_name alter bought_at drop default'
end
But I don't like this approach, because I am now dependent on how the underlying database is interpreting this command. In case of a change of the database this query perhaps might not work anymore and the migration would be broken. So, is there a way to express the undo of a default setting for a column in rails?
Rails 5+
def change
change_column_default( :table_name, :column_name, from: nil, to: false )
end
Rails 3 and Rails 4
def up
change_column_default( :table_name, :column_name, nil )
end
def down
change_column_default( :table_name, :column_name, false )
end
Sounds like you're doing the right thing with your 'execute', as the docs point out:
change_column_default(table_name, column_name, default)
Sets a new default value for a column.
If you want to set the default value
to NULL, you are out of luck. You need
to DatabaseStatements#execute the
appropriate SQL statement yourself.
Examples
change_column_default(:suppliers, :qualification, 'new')
change_column_default(:accounts, :authorized, 1)
The following snippet I use to make NULL columns NOT NULL, but skip DEFAULT at schema level:
def self.up
change_column :table, :column, :string, :null => false, :default => ""
change_column_default(:table, :column, nil)
end
Rails 4
change_column :courses, :name, :string, limit: 100, null: false