I am working with a database that uses a lot of constraints defined within the schema. This is necessary, to ensure that other services and clients that use the database do not break the data model (please don't reply that this level of DB definition is inappropriate for a Rails application). Unfortunately this seems to take Rails beyond its ability to define, dump and subsequently recreate schemas, unless somebody knows something that I have missed.
The specific issue that I have encountered is with add_foreign_key statements in schema.rb, and I am looking to see if anybody knows a workaround that will save me embedding SQL directly into the schema.rb definition.
The Postgres DDL that I need to represent is:
ALTER TABLE ONLY trackers
ADD CONSTRAINT valid_protocol_sub_process
FOREIGN KEY (protocol_id, sub_process_id)
REFERENCES sub_processes(protocol_id, id) MATCH FULL;
Unfortunately, when I rake db:schema:dump the existing database to schema.rb this results in the following:
add_foreign_key "trackers", "sub_processes",
column: "protocol_id",
primary_key: "protocol_id",
name: "valid_protocol_sub_process"
This results in an invalid specification, when recreating the database, that only includes a single field and (fortunately) fails to run, since the resulting schema constraints would be incorrect.
I have attempted to change the primary_key and column option strings to include both fields to match the required SQL, but ActiveRecord puts quotes around the whole lot, making the SQL statement invalid. I also attempted to use an array of columns too, but it appears to just #to_s the array.
Is this just beyond the ability of add_foreign_key, or is there a way to use multiple fields in a foreign key specification?
It appears that there is no checking that schema.rb can validly represent the full database if you use database specific DDL. Although I understand that schema.rb may not be able to represent every possibility, it is unfortunately that there is no error produced to indicate that the schema.rb generated by rake is invalid.
In order to get a full SQL dump of the database, performed by the database's own schema dumping tool, I added:
config.active_record.schema_format = :sql
to application.rb. This ensures that in future I get a valid, usable database schema to rebuild an environment.
Related
I'm working on porting a very large Rails project from DataMapper to ActiveRecord. Among the models that has to be ported is a set of User models that used Single Table Inheritance (STI) to distinguish one type from another. Here is a simplified version of what it looks like:
class User < ActiveRecord::Base
...
end
class AdminUser < User
...
end
Generally, the 'type' field is used to tell the difference between Users and AdminUsers, by storing the class name of the object being saved (i.e. 'AdminUser'). And it works fine in development, but when I try User.create in the test environment, I get:
ActiveRecord::StatementInvalid: Mysql::Error: Column 'type' cannot be null
ActiveRecord tries to insert a new row, setting the type column to NULL... what could cause this to be happening in the test environment, but not in development?
Turns out it was a slight difference in the database table itself that was causing a change in behavior for ActiveRecord. The test database had no default value for the type column, whereas in development, the default value was 'User'. Apparently ActiveRecord uses the default value when inserting data for an object of the primary class type (the class that inherits from ActiveRecord::Base - in this case, User). Why it doesn't just use the class name is beyond my understanding!
My real confusion came when I updated my dev database to have a default for the type column, which I actually knew it needed, because somehow the production database already had one, so clearly my dev database was just out of sync. So I did this:
mysql> ALTER TABLE users MODIFY COLUMN type varchar(50) NOT NULL DEFAULT 'User';
...[ok]
mysql> exit
Bye
$> bundle exec rake db:test:prepare # <-- My Mistake
...[ok]
I thought this was all I had to do, but it turns out running db:test:prepare just matches your test database to your schema.rb file, and my schema.rb file hadn't been updated, so then User.create worked in development, but broke in testing :D
Eventually, I came to understand all of the above, in addition to the fact that I needed to run db:migrate in order to update my schema.rb file BEFORE running db:test:prepare. Once I did that: voila! User.create actually used the default value of the type column to insert new User objects.
Moral of the story:
NEVER let your development database get out of sync with production. If it is: blow it away with a db:schema:load and start over with new dev data! (Or get a production dump or something)
Choose your ORM wisely. R.I.P. DataMapper - I'll miss your elegant abstractions... but not your bugs.
I'm a junior Rails developer and at work we faced the following problem:
Needed to update the value of a column only for one record.
What we did is creating a migration like this:
class DisableAccessForUser < ActiveRecord::Migration
def change
User.where(name: "User").first.update_column(:access, false)
end
end
Are migrations only for schema changes?
What other solutions do you suggest?
PS: I can only change it with code. No access to console.
The short version is, since migrations are only for schema changes, you wouldn't want to use them to change actual data in the database.
The main issue is that your data-manipulating migration(s) might be ignored by other developers if they load the DB structuring using either rake db:schema:load or rake db:reset. Both of which merely load the latest version of the structure using the schema.rb file and do not touch the migrations.
As Nikita Singh also noted in the comments, I too would say the best method of changing row data is to implement a simple rake task that can be run as needed, independent of the migration structure. Or, for a first time installation, the seed.rb file is perfect to load initial system data.
Hope that rambling helps.
Update
Found some documentation in some "official" sources:
Rails Guide for Migrations - Using Models in your Migrations. This section gives a description of a scenario in which data-manipulation in the migration files can cause problems for other developers.
Rails Guide for Migrations - Migrations and Seed Data. Same document as above, doesn't really explain why it is bad to put seed or data manipulation in the migration, merely says to put all that in the seed.rd file.
This SO answer. This person basically says the same thing I wrote above, except they provide a quote from the book Agile Web Development with Rails (3rd edition), partially written by David Heinemeier Hansson, creator of Rails. I won't copy the quote, as you can read it in that post, but I believe it gives you a better idea of why seed or data manipulation in migrations might be considered a bad practice.
Migrations are fine for schema changes. But when you work on much collaborated projects like pulling code everyday from lot of developers.
Chances are you might miss some migrations(Value update migrations..No problem for schema changes) Because migrations depends on the timestamps.
So what we do is create a rake task in a single namespace to update some table values( Be careful it does not overwrites)
And invoke all the rake task in that NameSpace whenever we update the code from Git.
Making data changes using classes in migrations is dangerous because it's not terribly future proof. Changes to the class can easily break the migration in the future.
For example, let's imagine you were to add a new column to user (sample_group) and access that column in a Rails lifecycle callback that executes on object load (e.g. after_initialize). That would break this migration. If you weren't skipping callbacks and validations on save (by using update_column) there'd be even more ways to break this migration going forward.
When I want to make data changes in migrations I typically fall back to SQL. One can execute any SQL statement in a migration by using the execute() method. The exact SQL to use depends on the database in use, but you should be able to come up with a db appropriate query. For example in MySQL I believe the following should work:
execute("UPDATE users SET access = 0 WHERE id IN (select id from users order by id limit 1);")
This is far more future proof.
There is nothing wrong with using a migration to migrate the data in your database, in the right situation, if you do it right.
There are two related things you should avoid in your migrations (as many have mentioned), neither of which preclude migrating data:
It's not safe to use your models in your migrations. The code in the User model might change, and nobody is going to update your migration when that happens, so if some co-worker takes a vacation for 3 months, comes back, and tries to run all the migrations that happened while she was gone, but somebody renamed the User model in the mean time, your migration will be broken, and prevent her from catching up. This just means you have to use SQL, or (if you are determined to keep even your migrations implementation-agnostic) include an independent copy of an ActiveRecord model directly in your migration file (nested under the migration class).
It also doesn't make sense to use migrations for seed data, which is, specifically, data that is to be used to populate a new database when someone sets up the app for the first time so the app will run (or will have the data one would expect in a brand new instance of the app). You can't use migrations for this because you don't run migrations when setting up your database for the first time, you run db:schema:load. Hence the special file for maintaining seed data: seeds.rb. This just means that if you do need to add data in a migration (in order to get production and everyone's dev data up to speed), and it qualifies as seed data (necessary for the app to run), you need to add it to seeds.rb too!
Neither of these, however, mean that you shouldn't use migrations to migrate the data in existing databases. That is what they are for. You should use them!
A migrations is simply a structured way to make database changes, both schema and data.
In my opinion there are situations in which using migrations for data changes is legitimate.
For example:
If you are holding data which is mostly constant in your database but changes annually, it is fine to make a migration each year to update it. For example, if you list the teams in a soccer league a migration would be a good way to update the current teams in each year.
If you want to mass-alter an attribute of a large table. For example if you had a slug column in your user and the name "some user" would be translated to the slug "some_user" and now you want to change it to "some.user". This is something I'd do with a migration.
Having said that, I wouldn't use a migration to change a single user attribute. If this is something which happens occasionally you should make a dashboard which will allow you to edit this data in the future. Otherwise a rake task may be a good option.
This question is old and I think rails approach changed over time here. Based on https://edgeguides.rubyonrails.org/active_record_migrations.html#migrations-and-seed-data it's OK to feed new columns with data here. To be more precise your migration code should contain also "down" block:
class DisableAccessForUser < ActiveRecord::Migration
def up
User.where(name: "User").first.update_column(:access, false)
end
def down
User.where(name: "User").first.update_column(:access, true)
end
end
If you use seeds.rb to pre-fill data, don't forget to include new column value there, too:
User.find_or_create_by(id: 0, name: 'User', access: false)
If I remember correctly, changing particular records may work, but I'm not sure about that.
In any case, it isn't a good practice, migrations should be user for schema changes only.
For updating one record I would use console. Just type 'rails console' in terminal and input code to change attributes.
For instance, when I generate an Event model, the table automatically sets to the public schema. How do I specify it to get set to a different schema?
Furthermore, how do you alter the schema of an existing table? Perhaps move it to a different schema?
Thank you!
Disclaimer: I don't know rails, so I'm going to give very postgresql-oriented answers here. For the first part of your question, there is quite possibly a much better way to do this, by making rails specify the schema when creating tables.
In PostgreSQL, tables are searched for in schemas according to the search_path setting. This is set by default to "$user",public. Tables are created in the first schema found in the search path that exists. So if you connect as "my_user", it will try to create tables in "my_user", and fall back to creating them in "public" if "my_user" doesn't exist.
So one approach is to update the "search_path" setting used for the user you connect to the database to make schema changes. For example you can say ALTER USER my_user SET search_path = my_app, public. If you then create a "my_app" schema then subsequent CREATE TABLE foo(...) commands executed by "my_user" will put the new table into "my_app".
You can change the schema of a table using ALTER TABLE foo SET SCHEMA my_app.
Create a migration to generate your new schema. ActiveRecord can't update you schema to you it's the pattern system. You can try sequel or DataMapper if you want update you schema from your code.
I'm doing maintenance work on a Rails site that I inherited; it's driven by an Oracle database, and I've got access to both development and production installations of the site (each with its own Oracle DB). I'm running into an Oracle error when trying to insert data on the production site, but not the dev site:
ActiveRecord::StatementInvalid (OCIError: ORA-00001: unique constraint (DATABASE_NAME.PK_REGISTRATION_OWNERSHIP) violated: INSERT INTO registration_ownerships (updated_at, company_ownership_id, created_by, updated_by, registration_id, created_at) VALUES ('2006-05-04 16:30:47', 3, NULL, NULL, 2920, '2006-05-04 16:30:47')):
/usr/local/lib/ruby/gems/1.8/gems/activerecord-oracle-adapter-1.0.0.9250/lib/active_record/connection_adapters/oracle_adapter.rb:221:in `execute'
app/controllers/vendors_controller.rb:94:in `create'
As far as I can tell (I'm using Navicat as an Oracle client), the DB schema for the dev site is identical to that of the live site. I'm not an Oracle expert; can anyone shed light on why I'd be getting the error in one installation and not the other?
Incidentally, both dev and production registration_ownerships tables are populated with lots of data, including duplicate entries for country_ownership_id (driven by index PK_REGISTRATION_OWNERSHIP). Please let me know if you need more information to troubleshoot. I'm sorry I haven't given more already, but I just wasn't sure which details would be helpful.
UPDATE: I've tried dropping the constraint on the production server but it had no effect; I didn't want to drop the index as well because I'm not sure what the consequences might be and I don't want to make production less stable than it already is.
Curiously, I tried executing by hand the SQL that was throwing an error, and Oracle accepted the insert statement (though I had to wrap the dates in to_date() calls with string literals to get around an "ORA-01861: literal does not match format string" error). What might be going on here?
Based on the name of the constraint, PK_REGISTRATION_OWNERSHIP, you have a primary key violation. If these databases aren't maintaining this data in lockstep, something/someone has already inserted a record into the registration_ownerships table in your production database with company_ownership_id=2 & registration_id=2920. (I'm guessing at the specifics based on the names)
If this particular set of values needs to exist in the production database,
1) check that what's already there isn't what you're trying to insert. if it is, you're done.
2) If you need to insert your sample data as-is, you need to modify the existing data & re-insert it (and all the dependent/refering records), then you can insert your values.
If you query the table and find no matching rows, then one of the following may be the cause:
The session is trying to insert the row twice.
Another session has inserted the row, but hasn't committed yet.
Also, check that the state of the unique constraint is the same between dev and prod. Perhaps the one on dev is marked as not validated - check that the index exists on dev and is a unique index (note: in Oracle it is possible to have a unique constraint validated by a non-unique index).
Take a hard look at the underlying unique index for the constraint. The reason dropping the constraint doesn't change anything is because the index remains, and it's a unique index. What does the following tell you about the indexes in both environments? Are both indexes valid? Are both defined the same? Are they both actually unique?
SELECT ai.table_name, ai.index_name, ai.uniqueness, aic.column_name, ai.status
FROM all_constraints ac JOIN all_indexes ai ON (ac.index_name = ai.index_name)
JOIN all_ind_columns aic ON (ai.index_name = aic.index_name)
WHERE ac.owner = 'YOUR_USER'
AND ac.constraint_name = 'PK_REGISTRATION_OWNERSHIP'
ORDER BY ai.index_name, column_position;
As it happens, there was a spare copy of the "registrations" model lying around the directory; even though it had a different name ("registrations_2349871.rb" or something like that) Rails was running all model functionality (saving, validating, etc) twice, hence the key constraint violation! I've never seen behavior like this before. Deleting the rogue file solved the problem.
I need to make changes to an in-use production database. Just adding a few columns. I've made the changes to the dev database with migrations. What is the best way to update the production database while preserving the existing data and not disrupting operation too much?
It's MYSQL and I will be needing to add data to the columns as well for already existing records. One column can have a default value (it's boolean) but the other is a timestamp and should have an arbitrary backdated value. The row counts are not huge.
So if I use migrations how do I add data and how do I get it to just do the two (or three - I add data -latest migrations on the production db when it wasn't initially built via migrations (I believe they used the schema instead)?
I always follow this procedure:
Dump prod database with mysqldump command
Populate dev / test database with dump using mysql command
Run migrations in dev / test
Check migration worked
Dump prod database with mysqldump command (as it may have changed) keeping backup on server
Run migrations on prod (using capristano)
Test migration has worked on prod
Drink beer (while watching error logs)
It sounds like you're in a state where the production db schema doesn't exactly match what you're using in dev (although it's not totally clear). I would draw a line in the sand, and get that prod db in a better state. Essentially what you want to do is make sure that the prod db has a "schema_info" table that lists any migrations that you >don't< ever want to run in production. Then you can add migrations to your hearts content and they'll work against the production db.
Once you've done that you can write migrations that add schema changes or add data, but one thing you need to be really careful about is that if you add data using a migration, you must define the model within the migration itself, like this:
class AddSomeColumnsToUserTable < ActiveRecord::Migration
class User < ActiveRecord::Base; end
def self.up
add_column :users, :super_cool, :boolean, :default => :false
u = User.find_by_login('cameron')
u.super_cool = true
u.save
end
def self.down
remove_column :users, :super_cool
end
end
The reason for this is that in the future, you might remove the model altogether, during some refactoring or other. If you don't define the user class on line "User.find_by_login..." the migration will throw an exception which is a big pain.
Is there a reason you are not using the same migrations you used in your dev environment?
Adding a column with add_column in a migration should be non-destructive: it will generate a "ALTER TABLE" statement. If you know what you're going to put into the columns once created, you can fill in the values within the migration (you may choose a less time-consuming alternative if the row counts are large).
Removing or altering the definition of a column is, I think, platform-dependent: some will allow deletion of a column in place, others will perform a rename-create-select-drop sequence of commands.
To get more specific, we need more information: what kind of migration are you looking at, what platform are you running on, do you need to set values as part of the migration? Stuff like that would help a lot - just edit the question, which will push it back up the list.